The old Victorian homes have been kept intact and in good shape pretty much for 150 years. It was developed by the Methodists who had a very clear, communal, holistic vision of the town. Therefore, they consciously designed the blocks so that everyone could enjoy the ocean to the extent possible. Also, one of the unique features of the town is the tent community. The church is surrounded by platforms and in the summer generations of families have come and built their semi-permanent tents there. I cannot image that "rich people" from NYC are going to look kindly on the "riff-raff"---I also don't think they'll give a rats ass about blocking a neighbor's view.
Plus, they might say they like good design, but does that mean they really know Victorian architecture, or that their modern touches won't completely overshadow the vintage aura of the town?
This reminds me Madison, where I grew up in the summer--all the "rich people" have made similar changes to that town. Of course, the older generation hates it. The only solace I can take from this, now that I live, not on the ocean, but at least on a lake, is that maybe someday my kids will say "Wow, Mom and Dad only paid $164k for the house and NOW it's worth 1M!!" And that will be nice, as long as higher taxes don't force them to sell, which is what happened to my old Madison friends and neighbors.