I was astonished when someone told me they thought I suffered from low self esteem. Apparently I've been too successful masking my scorn for most of humanity behind a façade of modesty and self-abasement. I am usually quite chuffed about myself, merely trying to hide it.
The original poster makes a good point, though, avoiding social interaction is subject to the perception of those around us and people can, literally, get the wrong idea. I know I had problems with that after a pretty much non-correctable hearing loss in 1990. I didn't realize how serious it was at first; turns out I was hearing only 60 percent of the words spoken to me correctly, and in the hubbub of a crowd, a party, or a noisy office, it was less. Ditto less for people with soft or high voices, absolute disaster with people who had accents that required a lot of attention, on my part, to sibilants and consonants. So if I'm your boss and you tell me something important and I hear something else and am unresponsive, your perception of me changes. And if I can't understand you, my tendency is to avoid you - which also feeds my personality preferences.
Sometimes, perhaps, low self esteem, like a hearing loss or blindness or a missing leg, is neither the cause nor the result of a personality preference, but surely it can be a second mule pulling your wagon in the same direction.
That's all I got.