I used to be so jealous of my husband when I was home with the kids and he went to work, and had the freedom to take time off and go to lunch with his friends
just be sure to clock in and out for lunch as every minute must be accounted for and made up some way. By friends you mean coworkers right? Most of us work miles and miles away from any real friends and their jobs, making seeing them at lunch entirely nonviable (this is NOT EUROPE with it's dense cities where you hang out for a 2 hour leisurely lunch with friends or something afterall - this is the U.S. where the pressures of work and distance alienates us from our friends further). And coworkers are really a poor substitution for friends, you can't talk to them about things the way you might friends, so it's always the same boring safe topics that bore me to death, I'd rather eat alone and do, I much prefer my own company to that inane chatter (powerball is the topic du jour for coworkers now, I am so sick of hearing about powerball I can't even say. Listen delusional idiots: you are not going to win the fricken powerball! enough about powerball already!).

go to the gym after work, and pretty much call his time his own
by the end of the day I'm far too tired to think about gyms, that's the reality for most it seems. But yes you have a tiny amount of time to call your own and it can be true me time. It seems to me the stay at home mom has far more time to call their own if they enjoy what they do because isn't time enjoyed with kids their own in a way work never can be? Isn't playing with the kids their own time in a way a job can never be?

and reap recognition from his boss and peers while he was at it.
or not, most people aren't reaping a heck of a lot of recognition these days. this reaping recognition stuff is more a mythical conception of work than the reality much of the time. Near invisibility seems far more often the reality.

I had no free time (as in zero), no recognition, no lunch, no gym.
isn't playing with the kids free time? Doing jigsaws with them? Baking cookies with the kids? Teaching the kids stuff like sewing etc.? This very much seems like free time to me, almost the very definition of play, even though it is true it's not "me time", which people also need (hire a babysitter occasionally or I guess there is no money for that because one paycheck barely covers bills - poor suffering husband who has the entire responsibility for whether a whole bunch of other humans eat on him). Of course you can't do this playing with the kids with a screaming infant I guess, but that stage is temporary.

I've known a lot of corporate colleagues who cut their their maternity leave short because they'd rather be at work than at home with their kids.
They probably shouldn't have had kids. I have no problem with them working (in fact if we insist that everyone not working lacks worth ethic then ...), but if they actively hate being with their kids - nope probably wasn't the best choice.