Especially when you're just getting back into it, it is very exhausting. Your body isn't used to that kind of exertion, so it does take a lot out of you. For some people (like JKPAS and myself), it doesn't matter how long you've been exercising, you may still get tired afterwards. When I used to swim religiously, I was in fantastic shape, but I'd also come home and take a nap, because I was just too exhausted to do anything else. Of course, in my heyday, I'd also swim until failure (i.e. I'd often go into the washroom after and throw up), so that may have added to it a little.
Hey, I'm a Borderline - we go overboard on everything!