Generally, I think most of the annoyances of any type are the result of inadequacies regarding the inferior function. When forced to use an inferior function, people tend to behave in often unpredictable ways. Dominant functions can be annoying too, or at least boring, but there is usually a mastery of them that is somewhat impressive, even if the person ultimately pisses you off. I mean, you have to admit there is something kind of cool about how Fe-doms can control a group.
(when you're not a part of that group, of course) 
So, for INTPs this is of couse Fe. I think one misconception of the MBTI is that F types are the ones running around crying, screaming and generally behaving irrationally. While they may not value logic as much, and are more prone to using emotional manipulation, I don't think they're the ones most prone to emotional outbursts and losing control of emotions. They're actually likely the ones most
in control of their feelings, since they generally have a greater tolerance for such things. It's the inferior Feelers, imo, that can often have the least control over their emotions. They try to compensate by burying them and staying detached, but when they do come out, watch out.
So yeah, I think that's what annoys me about INTPs. Emotions are kind of all or nothing, and this can be difficult to deal with. That's not to say that F = emotions; I know it doesn't, but generally inferior Feelers (esp. Fe) will be the ones who have less control over emotional displays.
EDIT: Obviously, this all applies for me as well. I sometimes have sudden bursts of anger that seemingly come from nowhere and then mysteriously end seconds later.
.
.
Its a people trait.
