Its a perv, a fetish called cuckolding. Probably comes out of inner conflicts you will not even know are happening.
I know, I have had it since I was 13. (works for girlfriends too).
I have lots of fantasies and fetishes, this one is powerful as it works alongside jealousy and fear of losing.
Embracing it is not a good idea as it will just get ever more pervasive. Like any other perversion you will have a fight to get rid of it, and will probably be fighting it forever.
Buy a porno mag (with women only) and think straight thoughts while you relieve the pressure.