Interestingly, I have had my wallet and/or pocketbook stolen about 4 times (in my 20s, working in NYC), and EVERY time, my wallet was on me, or at arm's length. It was never when I had left it alone. I'm not saying that you should just leave your purse unattended, but having it on you is no guarantee it's safe, either.
I think Golde Hawn said something like, "I would rather trust people and risk losing something than live my life in fear and suspicion."