I have a similar thing, in my case though I feel as though if out n about I should be on my way somewhere or doing something, not just meandering about. Wouldn't say it's agoraphobic though, that to me is the Jody Foster movie where she's a profiler and literally collapses just trying to get out of her doorway.
I began being teased in junior high and it got so bad I tested out of school by high school. Of course, this was long before anyone knew bringing a gun to school was an option...Sorry, my gallows sense of humour.

So I think my self-esteem, though consciously all right isn't subconsciously. I've gotten better about it over the years though. When I first moved in here I almost never opened the drapes if I'd be visible, especially at night (where people can see in, but I can't see out.) Now I manage that fine. Been here 14 years so maybe familiarity with surroundings helps? Or some sort of stranger anxiety compounds things. Moved around a lot growing up Mom havng a marked case of wanderlust. Since being out in the world though I've discovered I'm very change-adverse and like things stable and predictable and unchanging and joke I'm quite happy with the thought of dying right here in a few more decades.
Think a lot of what's going on for many not paralyzingly agoraphobic is simply we're stimulated much more when at a computer and.or tv, and being out represents a seperation from that stimulation. ...Plus of course UV light is bad and there's radiation out there from the Sun.
"I'd rather be hated for who I am, than loved for who I pretended to be." - Me.