One of the main things I hate about telephones is the uncertainty of the environment in which the conversation is taking place. A lot of people will call you from and/or answer their phones in places in earshot of others. People eavesdrop on telephone conversations. I do it and know others do as well. It can't be helped. A lot of the time the phones volume is so loud that both participants can be heard by someone within earshot of either. There's a staggering amount of scenarios where one might say the wrong thing because of ignorance of the others location. Also it just seems a little creepy having others listen in on you talking to someone and not knowing if others are doing the same on the other end.
A hypothetical example of where this would be very very awkward: You are at home, your mother calls and wants to have lunch together. During lunch, your girlfriend (who thinks you are at home) calls and very obviously/explicitly wants sex. Now imagine your mother didn't know about her and she probably overheard this. This would be an extremely awkward situation for a lot of people. On top of this, you now have to figure out how to defuse the situation. You could just say If you say "I'm having lunch, I'll call you later" this might be interpreted as a little rude/suspicious (if she's the jealous type) by your girlfriend and might be followed by the question "with who?". Your mother will probably also ask "Who was that?" and you are forced into the situation of either telling her about your girlfriend (which would lead to "why haven't you introduced us?" or something like that) or lie to your mothers face.
Aside from that, people tend to put on different faces in front of different people. You can't predict how people will act if they answer their phone in front of group A vs group B. An example of this is if the person was on the job and they answered in front of a client or if someone was with their kids when they answered the phone, they might respond much different than if they were alone.
Trying to write down information as it's spoken can also be painful (especially when it comes to things where individual characters are important).
If I know the topic of conversation before entering, I might be more willing to enter it. This is often not the case on phones. People call you, but you are not sure what they want to discuss and in order to find out you have to agree to discuss it (by answering the phone) regardless. It would be very rude to just say "I don't want to talk about that, that's boring and/or unimportant." and just hang up. Also, if people call just to talk, you have to either pretend like you're really busy, pretend like you are interested in the mundane events of their life, or risk offending them.
As for me, I have a cell phone, but only charge it once a month (at max) so it's usually dead. When it is powered on, I rarely answer, and only do so if I know the person calling (even then, I usually don't).
