What was with all the spambot activity?

[rosarugosa]

I'm sorry you had to deal with a bunch of spammy crap, Alan. Do you have any idea what causes this uptick in bot activity?

[Alan]

I've come to believe that these disruptions are often planned by major tech players for their own purposes, although I don't think their efforts are meant to be disruptive. On a normal day we usually have several hundred non-member visitors on the site at any given time, those may be people or bots/web crawlers (probably a combination of both) and pose no disruptive threat to our operation. However, once in a while we'll experience multiple thousands of visitors arriving in just a few minutes time and sticking around for several hours before leaving just as quickly as they arrived.

Our forum analytics software keeps track of visitors and we spent the vast majority of our 15 year existence with a simultaneous visitor record of 6000 or so visitors. About 18 months ago we set a new record of just over 10,000 simultaneous visitors then a few months ago that record was raised to 14,000, then just the other day it jumped to 24,000 with subsequent daily highs of 15,000 to 16,000. For small sites such as ours which cannot afford dedicated servers/high memory allocations, etc., and must depend on virtual servers tweaked for smaller audiences, these large quantities of visitors stretch our resources significantly during their visits.

I have the ability to get into our virtual server's control panel and see individual IP addresses and activity in real time so what I've seen these past few days has been very large quantities of IP addresses which can be traced back to Microsoft, Google, Meta, etc. I have to admit that in my frustration I've placed large swaths of those IP addresses on a 'banned IP' list to prevent them from being able to access the site ever again, although I've found that those efforts haven't been very productive even though I found them personally satisfying at the time.

Subsequent internet searches show that this is becoming a real problem for smaller sites such as ours and the blame is almost universally placed on AI and the major players efforts to 'scrape' data from all sites large or small to feed their language models. Many sites are having success using custom commands in robots.txt or llms.txt files embedded in their operating systems to allow or disallow specific traffic such as this. If ours continues I'll be looking into that as well.

[KayLR]

Again I missed all the excitement. Thank you, Alan. It's interesting but a giant pain in the neck for you.

[Alan]

Another aspect of this is that we don't even have to be the object of a massive visitor influx and still be affected. In a virtual server environment such as we have there can be many sites such as ours sharing physical space on one physical server. It's entirely possible that if one of those many sites is experiencing huge payloads, the physical server itself slows down which affects all the virtual servers it houses. I think that may be happening this morning as I see short periods of diminished capacity on our site while our simultaneous visitor count remains low.

[Rogar]

I've noticed an increase in sites I visit that have the "I am not a robot" check box to enter, or in the case where a password might be required will also need verification with a code sent no email or phone. Maybe AI is he cause of all the new precautions. I suspect the mis-uses of AI col be the end of a few things or new scam opportunities and it's just starting.

At any rate, thanks.

[iris lilies]

A tiresome business. Thanks Alan for devoting time to it

[rosarugosa]

Thanks, Alan.

[nswef]

Thank you Alan!