UPTIME - Who’s WRONG about the PSC Blog server?

December 16 2008

There has been a bunch of chatter over at Ed Brill 's blog about Google Apps SLA and some comments I wanted to pick out.

First, we have Volker Weber 's comment

vowe.net does not get over three nines every month of the year, edbrill.com definitely not.

and then we have Ed following up to that
Vowe.net and edbrill.com are not "mission-critical" mail servers, and in my case I certainly wouldn't ask PSC to run it on a UPS, cluster the server to a remote site, etc. The interjection of their uptime is irrelevant.

There seems to be some big misconceptions about the blog server PSC hosts and I wanted to correct those. Now I know PSC was not the focus of this discussion (the general uptime of the Domino server seems to be), but I wanted to clarify what is reality for edbrill.com, the other blogs we host (including this one), and the IdeaJam.net server.

PSC has a dedicated server for the blogs that we host. It is a nice Dell PowerEdge server with dual quad-core processors and 4 gigs of ram. Nothing huge but a decent machine. On that machine, we run a Domino server. This server is not connected to the PSC domain, but is a stand-alone domain just for the purpose of the blogs. We host the following blogs on that server:
  • Edbrill.com
  • johndavidhead.com
  • lotusblog.ru (Lotus Russian blog)
  • NotesDesignBlog.com (Mary Beth and her team's blog)
  • alanlepofsky.com
  • rodrigoborges.net
  • chrispepin.com
  • blog.lotisverige.com and blog.lotisverige.se (Lotus Sweden blog)
  • adamgartenberg.com
  • synch.rono.us blog (Lotus Connections team)
  • www.mix-and-mash.com (Lotus Mashups team)
  • www.dominoblog.com/net (Domino server team)
  • jeffeisen.com but that is a bit dead at the moment :-)

We also have also hosted MikeRhodin.com in the past and have a new blog coming and two that seem to be in limbo that were created by never launched. And for the record, we only host IBM employee blog's on this server. We do not host any community member's blogs and that is by choice.

If we look at the planetlotus.org Total Hits numbers, we get this:
#
Name Blog Title Description Hits Total
2
Ed Brill Ed Brill Collaboration, technology, travel, and more
43,818
7
Alan Lepofsky LepoLand - A Blog by Alan Lepofsky Alan’s blog about software, technology, travel, and the occasional golf post.
22,215
24
John D Head
10,803
25
Adam Gartenberg Adam Gartenberg’s Blog IBM Data Management and Social Marketing
10,385
57
Christopher Pepin Chris Pepin My perspectives on collaboration, social software, technology and IBM
5,202
65
Notes Design Blog Notes Design Blog Fighting for truth, justice, and a kick-butt user experience.
4,691
133
Alex Debian Lotuscube Все о программных продуктах IBM
2,543
167
synch.rono.us Synch.rono.us
1,808
168
Domino Blog Domino Blog News, views and directions on IBM Lotus Domino
1,803
181
The Sametime Blog The Sametime Blog The inside scoop on Lotus Sametime from the product management team
1,551
267
Mix and Mash Blog Mix and Mash Blog: Musings on Enterprise Mashups
516


That is a heck of a lot of traffic that goes to that server. And that is just from PlanetLotus. Ed's site gets the most traffic by far and much of that is from links from other blogs, twitter, and IBMers who read the site regularly (oh and his competition). On top of that, we also host the Ideajam.net server (Bruce's second piece of server hardware for OpenNTF.org before that moved off of the PSC network) and I believe the Ideajam and LinkJam blogs are hosted on that box along with the Ideajam and LinkJam applications.

So for the record, both servers are in a server room that is climate controlled and controlled by multiple UPS power devices. The blog server has nearline and tape backups. The ideajam server has nearline backups. Neither server is clustered, but PSC has an internal environment that is top of the line.

When I look back at 2008, I can easily claim 99.7% uptime. We have not been down over 26.25 hours this year. I bet it would turn out to be higher, but we do not keep track of that for these two machines. We have had some issues with all of our network providers in 2008 (we have multiple internet connections as all sites should have) and are doing infrastructure upgrades to attempt to solve that problem going forward.

One of the reasons the blog server is not four or five nine covered is that I upgrade that server to the latest beta's and design partner code drops all the time. Why? It provides one of the highest traffic public Domino servers and I let the folks from IBM see the HTTP and server logs when we have issues. I know they have super testing facilities in Westford to put Domino beta's thru the ringer when it comes to traffic, but this is real world. I know of two bugs that were fixed in just the 8.0.0 timeframe before the product was released because of the blog server.

PSC provides the hosting of the IBM blogs and the IdeaJam.net server because it is one way we can give back to the community. We do not charge anyone for this service. All we ask is for a hosted by logo on the front page. We provide open Notes client access for the IBMers ... they cross certify their IBM Notes ids and get to have a local replica. Is there some quid pro quo that goes on? for sure ... but that is what being part of a community is. We do this because it's what we can do, as does Yancy with PlanetLotus.org and Vince (and Bruce and Nathan and Anil in the past) with OpenNTF.org and Bruce and Julian for the Taking Notes podcast and Nathan and Chris with BleedYellow.com and Chris Miller with the non-IBMer blogs he hosts.. It helps the community and it helps ourselves. It is 2008 - just participating is no longer acceptable. Everyone needs to add value.

To wrap this up, our server has awesome uptime and is in an environment I would put customer production applications on. Nobody bothered to ask or research.

13 Responses to “UPTIME - Who’s WRONG about the PSC Blog server?”

  1. 1) john head says:

    test

  2. 2) Nathan T. Freeman says:

    heh... what's anyone going to say, John? "PSC rocks!?"

    You do. We just don't want to admit it. ;-)

  3. 3) John Head says:

    Thanks Nathan .. and the test post was because I changed the comment security on the site .. not cause I was looking for comments

  4. 4) Ben Poole says:

    OK, all good. So where's the answer implied by the title of this post?

    Who's wrong about the PSC blog server? And what did this mysterious person say that was wrong in the first place?

    Volker said that neither edbrill.com nor vowe.net achieved 99.9% uptime, and Ed pointed out that neither site is hosted on a mission-critical mail server. Both assertions are correct (and I see nothing contentious in either statement to boot), so I'm somewhat confused; your last sentence implies some kind of hurt on PSC's behalf, but I don't get where anything has been said that is vexed or argumentative.

  5. 5) Chris Pepin says:

    A great post. Thanks for sharing and for continuing to host my blog.

  6. 6) Nathan T. Freeman says:

    @Ben, I'll spell it out for you.

    edbrill.com is on the PSC blog server. That server *IS* mission critical as far as PSC is concerned, whether it's a mail server or not. And they treat it as such.

    That server has some uptime figure that may or may not be 99.9%. Volker doesn't know what it is. Neither does Ed. It's definitely at least 99.7%, so there's no anecdotal way to know the difference.

    So two people made an assertion about PSC's capabilities as a hosting provider that was probably wrong. At the very least, it was unfounded. As a fellow hosting provider, I can understand why John would be taken aback by that. Having people publicly claim you don't meet SLA targets is harmful to one's business.

  7. 7) Charles Robinson says:

    Just two quick things. First: "We do not host any community member's blogs and that is by choice."

    I would consider both you and Alan Lepfosky community members. :-)

    Second, I take offense at two things that were inserted in what was otherwise a very good rebuttal of a misconception:

    "Is there some quid pro quo that goes on? for sure ... but that is what being part of a community is. ... It helps the community and it helps ourselves. It is 2008 - just participating is no longer acceptable. Everyone needs to add value."

    I'm not so naive that I believe PSC hosts blogs for free without some type of reciprocity. PSC is a company in business to make a profit and I have no issue whatsoever with that.

    What bothers me is your statement that being a part of a community is about mutual back-scratching and "just participating is no longer acceptable." For some people value isn't measured in dollars and cents, reciprocity, public recognition, or awards. Some people participate in communities simply because they can. The returns do not have to be tangible to make the effort worthwhile.

    I apologize for intruding with a philosophical discussion. I was really bothered by what you said and felt compelled to respond.

  8. 8) Andrew Pollack says:

    I host my blog on my server at a massive datacenter, and I've got a secondary server at another center in another city. I get at least 99.7% that way. To be frank, Domino hasn't crashed on me for my blog sites in months and months. The last time I did have an outage, is when one of the data centers had an explosion in their power distribution room.

    Now for the real point -- SO WHAT?

    Its a blog. Ed's is a blog. They get some traffic, but they're not CNN or YouTube. If its down for an hour, nobody will die.

    What's the big deal here?

  9. 9) John Head says:

    Charles - I understand what you mean. I think in hindsight, I would word it as "It is 2008 - just lurking is no longer acceptable. Everyone needs to add value." I am not trying to tell people how to be part of the community. I am saying that in the new world of a recession and one of the worst job markets in my lifetime (and most likely my parents), you can't just sit on the sidelines. I am not asking people to spend money ... blogging, twitter, particpating in the LDD forums - that all counts as adding value. I just think we are beyond the time of lurking. People have to add value if they want their career to grow. If they want to be the ones in charge of their career instead of having it dictated to. This isn't about me telling any specific person that what they are doing is wrong. Not singling anyone out. Just me expressing my fustration with people who ask me why things aren't happening for them and a google search for them comes up empty.

  10. 10) Tom Franks says:

    I'm sorry. I gather that there are two servers being discussed here. But any server, regardless of what nice, pristine environment it sits behind that .."I upgrade that server to the latest beta's and design partner code drops all the time"... would, I believe, by definition be a test server, not a production server. I have certainly lived through some miserable domino releases in the past, some that were production, and not beta, but really? Beta code does not belong on "a production server".

  11. 11) John Head says:

    And that is what makes this server different Tom and why our infrastructure around it so important. But it does help with the testing of the product for sure :-)

  12. 12) John Head says:

    Andrew, we treat the blog server as an email server when it comes to uptime. Why? because people read the blogs around the clock (not mine, but Ed's, MBR, etc). I did not like the classification and I corrected the information. When part of the solution you offer companies is hosting of their production solution, perception is very important.

  13. 13) Alan Lepofsky says:

    John and everyone else at PSC, thank you for hosting my blog, it is appreciated.

Leave a Reply