Letting go.

Okay so I haven't been on here in a long time but I'm really unsure of what to do so I thought I'd ask here as a lot of you may be able to relate.

I have bipolar & BPD. I have been with my partner for 6 years (since I was 14) & we have 4 beautiful children.
When I am on a 'high' I often sleep with other men. In a normal state I wouldn't dare do this, it's just not me at all. Every time this happens my partner forgives me as he knows how I am when I'm in a manic state. Of course this isn't without arguments and constantly bringing it up whenever we have a disagreement. I understand why he can't let it go - I've hurt him time and time again & I feel $#%^ and disgusting, knowing I've done that to him. But I can't live with myself or move on from it myself when it's constantly being brought up. I just want to put it behind me.
Anyway, I recently had a manic episode and slept with someone I had been friends with and been speaking with for a while. Usually I would come down from my manic state and regret everything but this time I still really like the guy & really would like to be with him.
My problem is is that I want the best of both and I can't have that. I can't continue to hurt the father of my children. But I love him so much. He is the love of my life, he's all I've known really. I'm 100% comfortable around him and honestly don't know what I would do without him around. But I question myself a lot lately, 'could I really love him if I want someone else too?'
It's literally tearing me apart. I don't want to let him go but I know that it's the right thing to do. For his sake. He loves me - despite everything - he does.. Obviously we've had a lot of ups and downs. He's smashed my things up, and said hateful things which I can't get off my mind but the thought of him with anyone else makes me want to rip my face off.
I feel so selfish but I don't think I can let go.
Basically I'm here because I need to know what you all think. I love him more than anything but I don't want to keep hurting him & I can't have both of them. I sound like such a horrible person - I need to let him go but I don't know how

Hmmm...
Your note was posted a week ago and nobody replied.
I'll jump in... and go perhaps where angels fear to tread.

Promiscuity or sexual inhibition is a well documented symptom of bipolar illness.
You got it... you are not alone.

Now I'll go where angels fear to tread.
You say you have been with your partner for 6 years since you were 14 and have 4 children.
Does that make you age 20?
In your signature block you say you have 3 children.
Do you have 3 or 4 children? *mod edit*

I need more information. *mod edit*