I've had this problem for a while. My AvPD manifests by making me feel like a nuisance a lot of the time. I was making good progress overcoming it a few years ago, but then ended up confirming my fears with one particular woman (which I posted about here). Since then, I've been very hesitant about showing any interest in women.
I'm not saying that I would have approached anyone back then. Just that I was starting to show some attention at a distance, like making eye-contact and smiling, as well as getting some attention back. It was really helping my self-esteem.
I've tried to do the same recently, but I look away as soon as they notice. I can't seem to maintain that eye-contact anymore.
I also find myself worrying about showing interest if I'm getting to know someone. I try not to seem too keen when talking to them, and I don't maintain eye-contact during pauses in conversation, like I've noticed other people do.
(It's funny that I'd actually flirted a little and asked someone out, not long after the incident I mentioned above. The woman I asked out didn't give a bad reaction, and we still got on well aftwerwards. I even contacted her a few months ago to see how she was getting along. I've been finding it hard getting a job and she suggested I should try to get a job where she works. She even offered to help me find one.)
Any words of wisdom? Or does anyone else have the same problem for the same, or similar, reason?