UPTIME - Who’s WRONG about the PSC Blog server?

December 16 2008

There has been a bunch of chatter over at Ed Brill 's blog about Google Apps SLA and some comments I wanted to pick out.

First, we have Volker Weber 's comment

vowe.net does not get over three nines every month of the year, edbrill.com definitely not.

and then we have Ed following up to that
Vowe.net and edbrill.com are not "mission-critical" mail servers, and in my case I certainly wouldn't ask PSC to run it on a UPS, cluster the server to a remote site, etc. The interjection of their uptime is irrelevant.

There seems to be some big misconceptions about the blog server PSC hosts and I wanted to correct those. Now I know PSC was not the focus of this discussion (the general uptime of the Domino server seems to be), but I wanted to clarify what is reality for edbrill.com, the other blogs we host (including this one), and the IdeaJam.net server.

PSC has a dedicated server for the blogs that we host. It is a nice Dell PowerEdge server with dual quad-core processors and 4 gigs of ram. Nothing huge but a decent machine. On that machine, we run a Domino server. This server is not connected to the PSC domain, but is a stand-alone domain just for the purpose of the blogs. We host the following blogs on that server:
  • Edbrill.com
  • johndavidhead.com
  • lotusblog.ru (Lotus Russian blog)
  • NotesDesignBlog.com (Mary Beth and her team's blog)
  • alanlepofsky.com
  • rodrigoborges.net
  • chrispepin.com
  • blog.lotisverige.com and blog.lotisverige.se (Lotus Sweden blog)
  • adamgartenberg.com
  • synch.rono.us blog (Lotus Connections team)
  • www.mix-and-mash.com (Lotus Mashups team)
  • www.dominoblog.com/net (Domino server team)
  • jeffeisen.com but that is a bit dead at the moment :-)

We also have also hosted MikeRhodin.com in the past and have a new blog coming and two that seem to be in limbo that were created by never launched. And for the record, we only host IBM employee blog's on this server. We do not host any community member's blogs and that is by choice.

If we look at the planetlotus.org Total Hits numbers, we get this:
Name Blog Title Description Hits Total
Ed Brill Ed Brill Collaboration, technology, travel, and more
Alan Lepofsky LepoLand - A Blog by Alan Lepofsky Alan’s blog about software, technology, travel, and the occasional golf post.
John D Head
Adam Gartenberg Adam Gartenberg’s Blog IBM Data Management and Social Marketing
Christopher Pepin Chris Pepin My perspectives on collaboration, social software, technology and IBM
Notes Design Blog Notes Design Blog Fighting for truth, justice, and a kick-butt user experience.
Alex Debian Lotuscube Все о программных продуктах IBM
synch.rono.us Synch.rono.us
Domino Blog Domino Blog News, views and directions on IBM Lotus Domino
The Sametime Blog The Sametime Blog The inside scoop on Lotus Sametime from the product management team
Mix and Mash Blog Mix and Mash Blog: Musings on Enterprise Mashups

That is a heck of a lot of traffic that goes to that server. And that is just from PlanetLotus. Ed's site gets the most traffic by far and much of that is from links from other blogs, twitter, and IBMers who read the site regularly (oh and his competition). On top of that, we also host the Ideajam.net server (Bruce's second piece of server hardware for OpenNTF.org before that moved off of the PSC network) and I believe the Ideajam and LinkJam blogs are hosted on that box along with the Ideajam and LinkJam applications.

So for the record, both servers are in a server room that is climate controlled and controlled by multiple UPS power devices. The blog server has nearline and tape backups. The ideajam server has nearline backups. Neither server is clustered, but PSC has an internal environment that is top of the line.

When I look back at 2008, I can easily claim 99.7% uptime. We have not been down over 26.25 hours this year. I bet it would turn out to be higher, but we do not keep track of that for these two machines. We have had some issues with all of our network providers in 2008 (we have multiple internet connections as all sites should have) and are doing infrastructure upgrades to attempt to solve that problem going forward.

One of the reasons the blog server is not four or five nine covered is that I upgrade that server to the latest beta's and design partner code drops all the time. Why? It provides one of the highest traffic public Domino servers and I let the folks from IBM see the HTTP and server logs when we have issues. I know they have super testing facilities in Westford to put Domino beta's thru the ringer when it comes to traffic, but this is real world. I know of two bugs that were fixed in just the 8.0.0 timeframe before the product was released because of the blog server.

PSC provides the hosting of the IBM blogs and the IdeaJam.net server because it is one way we can give back to the community. We do not charge anyone for this service. All we ask is for a hosted by logo on the front page. We provide open Notes client access for the IBMers ... they cross certify their IBM Notes ids and get to have a local replica. Is there some quid pro quo that goes on? for sure ... but that is what being part of a community is. We do this because it's what we can do, as does Yancy with PlanetLotus.org and Vince (and Bruce and Nathan and Anil in the past) with OpenNTF.org and Bruce and Julian for the Taking Notes podcast and Nathan and Chris with BleedYellow.com and Chris Miller with the non-IBMer blogs he hosts.. It helps the community and it helps ourselves. It is 2008 - just participating is no longer acceptable. Everyone needs to add value.

To wrap this up, our server has awesome uptime and is in an environment I would put customer production applications on. Nobody bothered to ask or research.