As a women who is visually stimulated (men aren't the only ones), I have also found myself gawking at attractive looking men. So yes, it works both ways. He has probably worried about losing you to "better'" man at some point in your relationship too, though it's not something people like to admit so you probably won't hear it from him. It's quite rare for anyone to have that truly unshakable confidence.
Yes, he is looking at her, but that necessarily mean that he wants her? Perhaps. But probably not at as badly as you think he does. Every time a man sees a woman, he thinks of sex. What it would be like with her, what she might look like naked etc etc. What you are killing yourself over is that you think she will then become stuck in his mind when in truth,
it all ended the moment she totally passed him. It was a short meaningless moment that he probably won't even remember.
The female form is divine, so of course they are going to be admired. It's not only women who are around the malls and the streets, but as the competitive female, they just so happen to be the only ones you
notice. Same with insecure males who sees other males as potential threats. You were so caught up in your own insecurities and watching other women that you didn't notice the men that were checking YOU out too.

If any woman caught her man gawk at other women, it's only natural for her to become upset to some extent. However, if the the anger/jealousy exceeds the day of occurrence, trust issues can begin manifest (relationship killer). I'm no shrink, but this is an issue that can only be solved if you
believe in your spouse. He probably not as bad as you think he is.
The truth is simply that you will
never know if/when he is going to leave or, if he does, you
can not stop him. Your fears will only be in vain. Just as death is inevitable. We may cry or laugh our way to the grave, but it's the same outcome either way. The psychological stress is quite unnecessary and bad for you both. As hard as it is, just learn it let go and let things be. Your spouse will thank you for it and your lives will be much easier.
Knowing is one thing, but truly believing it can prove to be a difficult task. The truth is useless if it's not true to YOU. There will never be shortage of beautiful women/men, that is a fact. But another fact is that your man is still with
you. Smart men won't risk losing a good woman to scratch a temporary itch, but he will if your persistent anger/jealous drives him to.
One more thing, while men look at nudity/porn and may think it's "sexy", it's just a fantasy. Sex in real life is not perfect. Besides I highly doubt that most men would be comfortable with his woman's "lady bits" being on public display for all to see. They are natural egotists.
(English is not my mother language so please disregard any minor grammatical errors)