What do you think it was about wearing a bra that triggered you? Was it uncomfortable?
Sometimes I can't wait to get mine off, but like you, I wouldn't want my nipples showing in public. Is there a more comfortable and relaxed style you could try. You can get soft ones without underwiring.
Maybe you're wearing the wrong size? That's really common.
Maybe you could set yourself the goal of getting a more comfortable bra and break yourself in gradually? Getting one would be a good place to start and then build up the amount of time.
However, you
need to see a therapist for anxiety treatment.
PS - I am a woman, don't let the avatar fool you.