Public Service Announcement for anyone using the Domino Blog template

April 7 2010

Many of you who read this blog or any of the other IBM blogs that PSC hosts noticed that since Sunday evening, the blog server was very slow. Sometimes not reachable at all. Since fixing it last night, I figured I would pass along a few thoughts on how others can prevent this problem.

1. If you are using the Domino Blog template from IBM, please make sure you have the Anti Spam Check? Option turned to YES

Image:Public Service Announcement for anyone using the Domino Blog template

The mix-and-mash blog did not have this turned on ... and had gotten over 250,000 comments in about 36 hours. They were from different IP addresses, but this was causing the HTTP server task to pretty much die

2. I figured out that the mix-and-mash blog was the issue specifically because Luis found out that a blog entry from 2008 on that blog was open in HTTP threads a lot ... like over 10,000 active connections. This must have been how the botnet software was keeping the connection to the server and passing the information along to the different machines that were auto-generating comments. I blocked the root IP address that was keeping the connection. There was no way to block all the IP addresses that were making comments. I am not going to share the IP address(es) publically, but if this happens to you - contact me and we can compare data. I will say this ... every address that either created a comment or kept a http connection alive was from Eastern Europe or Russia. Nothing personal to folks in those regions, but I was not surprised at all.

3. Try not to host blogs that make multiple negative comments against Microsoft in a single day. Thanks Ed :-) (folks, this is a joke)

Lessons learned, server back to full steam. Sorry to all the hosted blogs for the performance issues.