I've never had a passport, either. We did quite a bit if traveling around the US in the motorhome- I've been to 44 states- and all across Canada and well into Mexico, in the late 1980s and early 1990s, when you could do that without a passport.
Thinking of cruising to Alaska after I retire next yesr- Yikes!- do I need passport?