Accidentally Terminated Therapy

I had been in therapy for almost a year. I recently accidentally terminated therapy via email. I was frustrated with my inability to talk about important topics in sessions. So, I emailed him about them. I had said in that weeks session that I thought it might be easier to talk with a female, but in my email I said that I thought I probably wouldn't make a difference.
He replied to my email by saying that he thought I was important that I talk about these things and that he would "completely understand" if I felt more comfortable with a female. I took the wording in his email to mean that he felt ill equipped or simply didn't want to deal with these topics. I guess I was kind of bother by this (?) so I asked him for names of female counselors in the area (something he'd offered to do when I quit for about a month in the summer).
Now I'm devastated! I really didn't mean to terminate therapy. But at the same time I think I have an unhealthy attachment to him, one of the topics he thought we needed to talk about (my fascination with older men), so maybe this is a good thing.
I don't want to see a different therapist. I don't want to go to a female therapist. I feel pathetic for being so upset over this. It keeps creeping into my mind and I start bawling my eyes out through the day.