There seems to be some confusion about decibels here. "Decibel" stands for deci⋅Bel, meaning 10x of a Bel. A Bel is the log10 ratio of two powers, named for Alexander Graham Bell. Since a Bel is a large jump in human terms, the world has converged to largely use decibels, abbreviated "dB".
For example, if sound A is 20 times more powerful than sound B, then you can say that A has log10(20) = 1.30 Bels, or 13.0 dB, more power.
Decibels is a useful unit to compare sound volumes with, since human hearing perceives sound volume mostly logarithmically, not linearly.
Since dB always indicates a ratio, there are no such things as absolute dB values. However, having a universal logarithmic scale for measuring sound volume is useful. As a result, the dBA scale has emerged, with specifies a particular sound power for the 0 dBA reference. That sound power was chosen to be about the limit of human hearing.
To bring this back to your question, despite common sloppiness with units, including the image you show, those values are not actually in dB, but in dBA. Note that 0 is even marked with "hearing threshold".
Now to speculate about why your sound meter shows 50 to 80 dBA when you think everything is quiet. This is probably due to the inherent noise floor of your device.
Consider the very large dynamic range represented by the chart you show. There is a 130 dB range between limit of hearing and the loudest noise. That's 1013 range of power values, and the square root of that for the voltage signals produced by an ideal microphone. 1013/2 = 3.16M = 221.6. I show the value as a power of two, because that's the number of bits it would take to represent the full volume range in a binary number.
It seems that you are using a phone with some app to do the sound level measuring. Do you really think this phone has a 22 bit A/D digitizing the microphone signal? Even if it did, you can't expect the analog circuitry in a phone to have that kind of signal to noise ratio, let alone the microphone that was optimized for small size and cost.
Maybe a 16 bit A/D is used, but due to limitations mentioned above the result only has 15 or 14 bits of actual signal to noise ratio. 15 bits can only represent a 90 db range, and 14 bits 85 dB. You say the lowest you have observed is 50 dBA. That would be the inherent noise floor if the microphone system is good to 15 bits and tops out at 140 dBA, or 14 bits and tops out at 135 dBA. Both these seem quite plausible at first glance, so there is nothing unexpected going on here.
If you want to measure sound levels down to 0 dBA, you either have to settle for a lower maximum measurable level, or pay a lot for a very sensitive and delicate scientific instrument.