My family growing up always viewed any non-aggressive physical contact that wasn't a handshake as something sexual in nature (and therefore to be avoided), and I just can't seem to get over that (I mean, the beautiful people are fine, but somehow it's always the ugly ones that want hugs, ugh). So I understand the awkwardness.
As for affection in general, I have a hard time with responding to complements (my family didn't do that growing up either). "Thanks, you too" only works in certain situations, "Thanks" is a bit too short and leaves an awkward pause, "It's kind of you to say" sounds self-depricating (plus, they could be not kind at all), "Mhmm" or silent acceptance present an unfavorable image of a know-it-all (which, I am, but I don't want them to know, obviously), and disputing their statement is considered ungracious. And saying a complement in return is just brown-nosing if they didn't do anything to deserve it.
It's less of a self-loathing thing and more of a "Great, you just handed me a social puzzle during a perfectly good day" sort of thing.