Help with childhood "scar"?

This is my first post on this forum, I didn't really know where to post this so I just decided to put it here. My main reason for joining this site is for social anxiety problems, but I wanted to get this off my chest first.. I am not sure if anyone can relate to me at all, but I hope so. I am a 21 year old guy, and I grew up as an only child in a home where my parents did nothing but drink and argue nearly all day everyday. There was no physical abuse or anything, but it took its tole on me mentally. Anyway when I was about 5, my mom was pregnant and I was going to have a little brother. And I was more excited then both of them combined, because I wanted a little sibling so badly. Long story short, she had a miscarriage and they had to tell me that the baby died etc. It absolutely devastated me, and nobody knows just how much it upset me even to this day. So from that day on, I was always lonely and wanted a little brother. And anytime I would see an older and younger brother together it would sort of upset and depress me, even at a young age. And my mom has a big family, and all of her siblings have multiple kids, and whenever we'd hang around them, I would always try to be the big brother to my younger cousins, and I loved it. But then we'd have to leave, and I'd be upset and lonely again. And now fast forwarding to more recent years. I have a cousin on my dad's side of the family who is 8. and I never met him till 2 years ago because they live across the country. And I remember before I met him, when I was like 15 my grandparents were trying to get me to visit them (cousin, uncle and aunt) on a vacation. And I would always say no, because I was afraid I would become attached to him. But then a few years ago they came to visit us, and exactly what I thought would happen happened. He liked all the same sports as me etc. I had a great time hanging around with him, and he is an only child too, so he kinda reminded me of myself. So long story short again, he feels like a little brother to me, and I only see him one or two times a year. And every time I see him it's one of the greatest feelings in the world for me, because I get to feel like a big brother. But then each time I have to leave I feel a stab in the heart every day afterwards, cause it feels like reliving the scar of losing a brother. Which sends me into a depressed state. I don't really know what to do about this, can anyone relate to me at all?? Sorry this is so long btw..