It would be advisable to join whatever group that you are interested in, because then you will most likely be surrounded by like-minded people. That is why - I feel - people who didn't enjoy school become happier in college (because they choose a course that they are interested in). Don't join what everybody else joins. Also, if the scouts is going well for you, then I'm sure you can become more active in that. I doubt people bother you there, right?
Kevin