I get this for a few reasons, often not just due to AvPD. One is that I go into a very anxious state around people, but this is shielded from me by my dissociation (google it

). When they go away, the anxiety doesn't completely go away, but it does become more obvious because the dissociation has become less severe. I know that when something has bothered me I won't be able to let it go or feel right again until I've slept either.
A true avoidant reason is because I'm afraid. Afraid that I said the wrong thing or said something the wrong way, or forgot to say something, or came across as weird.... I get scared that all this stuff might have happened, that the person I spoke to might think I'm weird and laugh about me or hate me, or both and I just can't bear it for some reason. If it went ok, I even get scared thinking how I could have got things wrong and how lucky I was that time. I worry that this person is ok with me now but it's only a matter of time before s/he figures out the truth of what I'm really like. It goes on and on for no logical reason.
But ultimately you need to try and figure out what it is that bothers you about being around people. There will be a reason in there somewhere. For example, if it's only around people you dislike it might be your imense relief at being around someone who could have hurt you and escaping unscathed, without being yelled at, criticised or embarrassed in some way (whatever you're scared of). It could also have put your body in overdrive the rest of the day, on the lookout for 'danger'. It may be avoidant or it may not be, depends on what exactly you're scared of and why.