Hey. I know how you feel.
For about 10 years now I have been terrified of looking other people in the eye.
Although I cannot pinpoint the cause of your fear, I can tell you a little bit about the cause of mine.
I hope that this may help you a little bit in understanding your problem.
I believe my fear sparks from an experience when I was younger. In this experience I was terribly shown up over and over by a very intimidating person in public. Ever since then I have been scared of people and of looking directly at people, in case they want to start a commotion.
In addition, I find that sometimes I manage to forget about it and I look at someone. But then when the fear does kick in it's even worse because I know I look anxious, so I therefore avoid watching other people seeing me look anxious in their eyes.
There's nothing worse than the 'sympathy' look in someones eyes when you can feel your face burning up.