You're probably not clinically schizoid, although it's hard to make such judgements definitively as everyone is different and "schizoid" is only a classification for people exhibiting certain behavioural traits/problems that interfere with daily life to a sufficient degree that it could be called a disorder (just as disease is technically any measurable loss of the body's normal functioning capacity).
Well anyway, personal explorations of the subject have led me to believe that this touch aversion/phobia is primarily the result of either not being touched enough as a child and/or traumatic experiences that involved being touched (a mishandled infant receiving a dislocated arm for example). Now you see the problem with such "disorders" is that you may feel inclined to defend it, stating that it is merely an aspect of your personal behavioural identity and should be accepted as such, which is entirely valid, there's nothing actually wrong with you. Indeed if the majority of society had encountered the stimuli that led you to develop this manner of behavioural bias it would probably be accepted as the norm, making overly "touchy" people seem weird, and they would likely be ostracised as a result.
However if this really is a disorder for you (it's affecting your life in some negative way) then of course it may be beneficial for you to change, not for the sake of society wishing you to conform to some standard of normality, instead for your own sake, so that you may live a healthier, happier life.
To do this we must understand the nature of your behavioural bias, which more than likely is based upon a perceptual bias, that bias being that you perceive being touched as an abnormal event, an event that may trigger strong memories of negative past experiences with being touched (in which case you’ll probably need professional therapy). Or as in most cases the relative absence of physical human interaction has made you highly touch sensitive, as I’ve said before (ages ago) being hugged is for some people akin to being raped, because the same sort of personal space boundaries are being broken.
The problem is (I’m guessing, but it’s a reasonable guess) that the rarity of human contact in your life has given it powerful connotative meaning to you, that for you being touched on the arm is as much a violation of privacy as being touched somewhere generally accepted as private. Now the only way to change this is to re-evaluate your personal space, try to imagine everywhere except where you wish to be explicitly private as “public space”, better yet play some contact games with your brother (like holding one leg up with both hands and hopping around, trying to push each other over, whoever remains standing, well hopping, wins) as the crux of your bias is simply how little you’ve been touched.
Better still, tell people they’re allowed to hug you, that indeed the only way you’re going to get over your adversity to being hugged is if people hug you until you get over it.