Some of you may have had issues getting onto the forum this morning, from around 10:00 until around now. At best pages were probably slow, or more likely not available at all.
An issue that we have long faced with our website has been bot traffic. Over the years we built up a fairly lengthy blacklist of IPs and browser headers that we blocked. Some of these were legitimate but not relevant, but most appeared to be nefarious. Unfortunately, this list is likely lost in the ruins of our old server.
What we saw this morning was ~>500 requests a minute for forum threads (but never any other resources like images or javascript) coming from a wide range of IPs that seemed to come from Chinese networks. Requests for this type of content which is largely un-cachable and requires database lookups is pretty heavy going, and is the equivalent of many thousands of normal users browsing the site at the same time.
Ever cloud has a silver lining though, and it was very interesting to see how this new server performed under such heavy load. The results were extremely positive in terms of the volume that was handled, and very insightful in terms of showing where the bottlenecks lie. So it isn’t all bad.
Needless to say, this IP range is now blocked. There will be more of these to come, but we’ll keep fighting them off.