Of course you are going to be sensitive as you are on constantly alert as you can never truly relax because life is so unpredictable unlike the rational, logical minds we possess. But, experience helps and over time, you will learn how things work and what can happen at what times. Our sensitivity can be useful in this way because we are more likely to pick these things up because we have a desire and NEED to understand. There is so much negativity involved with Aspergers, but I think that's societies problem, it teaches us that life is unpredictable and we HAVE to conform to these sets of rules which are almost alien and non nonsensical a lot of the time.
But, we have the abilities to turn the negatives around, its just about accepting how you are and not believing you should live and think like the majority of NT's, use you special interests and abilities in a way which suits you. Society has to be a little more accepting, but we must make an effort to understand ourselves. Your sensitivity on the telephone comes from a fear that can be turned into understanding with patience and time, our openness to learning helps us to overcome this sensitivity and engage with the world more effectively. In time, the telephone will not be scary any more, it may take many years, but it will lessen significantly. If they get annoyed, its because, THEY are the ones who cannot adapt to you, I assume you do not shout back at them rudely or whatever, NT's have as much trouble adapting to us as we do to them!
I hope this helps in SOME way Nitty

, I know how negative life can be and what you can feel (I understand sadness and hopelessness more than anything else thanks to depression), but I am starting to become more positive again thanks to various "interventions" in my life.