Geekularity

I spent the better part of the day and night yesterday recovering from a power failure at one of my client’s data centers; a failure that we did not have a contingency plan for. A transformer inside of the building, one that we thought was solid-state and not a concern, decided to self-destruct in the middle of a busy day, bringing down half of the office and all of the IT infrastructure.

We had battery backups and a diesel generator in the event of power loss, but the diesel generator connects to the building on the other side of the dead transformer, so once the batteries died, we were down. The only thing that could have saved our bacon would have been keeping about 1,000 feet of industrial extension cords on hand to run between the generator’s auxiliary ports and our most critical systems. I think the IT director is putting in for a purchase order this morning.

So, with this little nightmare behind me, I thought I’d try to open a thread of top lessons learned while implementing and supporting server infrastructure. These can be hard-learned lessons, or near misses that you’d like to see IT professionals think about and revisit periodically. I’ll start it off with my top 10, which I’ve picked up in my decade of IT industry experience:

1. BACK IT UP!.. AND THEN CHECK YOUR BACKUP. I’ve sent way too many hard drives to a clean-room laboratory to try to resurrect the data off of the failed media, all because the victim either didn’t back up their data, or never bothered to check that their backups were working. This is a sophomoric mistake, but it may take a $20,000 invoice from the data restoration company for a systems administrator to finally get religious about backups.
2. Get servers with redundant power supplies. Plug each power supply into a separate UPS, and each UPS into it’s own breaker, preferably on different legs of the building’s 2 or 3 phase circuit. This will allow you to swap out UPS batteries without bringing down the system and minimizes your exposure to building infrastructure problems (like I experienced last night).
3. Label every outlet in your data center. At minimum, each outlet should have the breaker number and panel location clearly labeled. You don’t want to be searching franticly for the correct breaker when your UPS units are beeping at you.
4. All electrical circuits used by your core IT infrastructure should be dedicated circuits. Do not use shared circuits, especially in locations where office tenants could plug in appliances like coffee makers and space heaters, or housekeeping could plug in a vacuum cleaner.
5. If you have more than one server, label it on the server. If the servers share the keyboard, monitor, and mouse through a KVM switch, make sure that the switch is also clearly labeled, and that the labels are correct.
6. Have a startup and shutdown procedure for bringing all of the systems down and bringing them back up again. Your data center is an organism, and there are critical services on some machines that need to be up and running before other systems can function. Make sure you know which servers or appliances are hosting DHCP, DNS, SysLog, Active Directory, etc., and make sure those devices are high on the boot order.
7. For the love of all that is good in the world, use velcro or zip ties to clean up the cabling around your servers. Keep the wires as short as possible and try to prevent any sort of rats nest wherever you can. It will pay huge dividends later when you are trying to isolate essential from non-essential power cords. Plus, clean wiring will promote air flow and will prolong the life of your equipment. If a cord hangs down below where it’s plugged in, it’s too long. If it touches the floor, it’s too long. Any loose cord that’s near the floor or at about hip level where most geeks keep their blackberry holstered, will undoubtedly get pulled out accidentally if not properly secured.
8. Disks will fail. It’s not a question of IF, it’s a matter of WHEN. So, for every RAID array you maintain, keep one or more spare drives on hand and readily available. If you use your spare, it is imperative that you order a new spare on the same day. Do not put this off.
9. Have an emergency resource guide inside your data center with phone numbers and reference information. Check and update this information regularly. Phone numbers should include electricians, HVAC, plumbing, fire sprinkler contractors and your building’s facility services hotline. Also include the cell and/or home phone numbers of any company executives that you may need to get emergency purchase approval from. Reference materials must include at the very least, all telco and ISP account and circuit IDs. If the resource guide is securable, you may want to include root and administrator login information for your critical systems.
10. I don’t drink coffee any more unless it’s in a traveller mug with a close-able lid. My son calls it “Daddy’s sippy-cup.” Even still, coffee, soda, water, or whatever should never enter the server room. I can still recall, in vivid detail, the day the CEO of the company I was working with, dropped his mug three feet in front of an open server cabinet. In slow motion, I watched as a splash of coffee arced gracefully from the shattered mug and into the front of a server’s hot swap drive bay. The result… refer to the invoice mentioned in tip #1.

So this list is just a starter. Please add your tips and tricks to the comment section below!

OpenID is an up and coming authentication standard that allows you to log into participating web applications using the same credentials for each. No more having to remember different usernames and passwords. You can authenticate with an OpenID compatible web service using a unique URL instead of a username and password. In my case the unique URL is http://seanosteen.com. Since this URL is unique to me, I do not need to worry about someone else registering the same username and blocking me. My authentication URL will always be http://seanosteen.com for as long as I own my own domain name.

Of course, right now, this standard is still in its infancy. Many more sites need to support it, and many *many* more users need to start using it in order for the technology to gain the necessary critical mass. Give it 3-4 more years, and I think most commercial websites will support OpenID authentication.

OpenID will thrive in an arena where other federated or single sign-on services have not, and here’s why:

1. OpenID is just as its name implies, an open standard. Anyone can implement an OpenID provider service, and there is no way for a big corporation to force you to use their service over another provider. They can only attract customers with better service and value-added solutions.
2. It’s easy to get started using it. You can sign up with any number of OpenID providers and receive an OpenID based on one of their accounts. With a little bit more work, you can setup a delegation to make your website or blog your own OpenID. I strongly recommend the delegating to your own custom URL as this decouples your OpenID from your service provider which will allow you to change providers at any time with a minimum of hassle.
3. It’s easy to implement on existing web applications. There are code libraries and samples for implementing OpenID on just about every web publishing platform. Some of the popular content management systems like Drupal, Joomla, and Wordpress already have plugins available to use. Most of the blog publishing services like LiveJournal & Wordpress allow you to easily setup your blog’s URL as your OpenID.
4. It’s easy to switch between providers! The OpenID standard provides for a delegation model. This means that you can make an OpenID out of any URL which you have control over and set it up using the OpenID provider of your choice! This is how I made http://seanosteen.com my OpenID. Just recently in fact, I switched between my old provider MyOpenID and my new one Personal Identifcation Provider (PIP), by Verisign; and I did so in about 30 minutes. All I had to do on my end was to copy and paste two lines of code into the HTML markup on my website. I didn’t even need to visit any of the websites, on which I use OpenID, to make changes. They automatically picked up my new delegation and authenticated me using my new provider. It’s that easy! This is of course using my custom URL as the OpenID. If you use the OpenID provider’s OpenID URL, a little more work will be involved to associate your new OpenID with an existing account.

By the way, Verisign’s Personal Identification Provider (PIP) is still in a beta testing phase. But one of their cool value added services, and the reason for my switch, is the availability of multi-factor credentials, specifically their SecureID key faubs. My only difficulty in implementing the PIP OpenID service was that they have not published how to setup delegation to their service. So, I contacted support on Saturday afternoon, of the long Labor Day Weekend here in the United States. To my surprise, I got a very prompt and helpful reply within an hour from Gary Krall, the Technical Director for the PIP project. Kudos to Gary and crew for the amazing response time on a holiday weekend.

Ignoring that faint voice in the back of my head that tells me that Google may indeed be Skynet, and that our robot overlords are just a few years away from their planned invasion, I decided to hand Google the keys to my kingdom. I have moved several of my internet domains over to Google Apps. What that means is that all of my Email, Calendar, News, and soon phone calls, will be hosted (and probably indexed) by Google. In the coming weeks, I will move all of the domains I manage, both personal and professional over to this amazing service. This excludes my clients whom I manage, but do not actually host.

I’m doing this because hosting my own email is both tiring and expensive. Rather than spend countless hours being my own systems administrator and doing endless battle with the spam trolls, I decided to delegate this little bit of geekery to someone who is more proficient. When complete, this move will hopefully free up some significant mental energy (and funds for that matter as the Google Apps basic package is free) to spend on other pursuits. Besides, Gmail and Google Calendar are easy to use, easy to access, and quite a good value for the price!

Right now, I’ve only moved my professional domains, but my [family] clients and my personal domains will soon follow. When all is complete, I should no longer need to rent a dedicated server, and my monthly recurring expenses should go way down. This comes after reading Scott Hanselman’s post about doing the same thing for himself and his entire family. So, special thanks to Scott for outlining his process and his reasoning. It was a big help.

Jack Dorsey and Alex Payne stated that Twitter is working on adopting OpenID, however they still see some significant hurdles. In an interview with Geoffrey Grosenbach for the Ruby on Rails Podcast, Dorsey and Payne state that they believe that OpenID is a bad fit for developer APIs that require a unique key or a username and password to authenticate with the web service. Can OpenID be adopted in such a case? Could an asymmetric relationship be created between an artifact, in this case a program, and its author? Can this relationship then be authenticated by a third party, in this case, Twitter, the web service provider? I’m still very green when it comes to OpenID, but I’m wondering if there are any provisions for asymmetric links, one-offs, third party authentication, or whatever it should be called in the OpenID standard? I’m hoping to start a discussion here, so please hit the comments section as hard as you can!

Here’s what I’ve found so far:

1. Les Orchard has an open discussion about using OpenID in a blog comment proxy.
2. Obviously the current OpenID specification

Microsoft disclosed today that most active versions of Windows (2000 through Vista) have a serious security flaw in how they handle animated mouse cursors. In this security advisory, Microsoft describes that, clicking on a link in an email message, stumbling across a malicious website, or even a legitimate website that’s been compromised can lead to unintended code execution. Just about the only safe environment is Windows Vista with IE7 in full lock down mode. This security threat is considered a zero-day exploit, meaning that examples of exploitation have already been observed in the wild, and currently there is no fix from Microsoft. So, keep running your windows update over the next few days or check back here for links to helpful resources. In the meantime, don’t open emails from people you don’t know, and stick to surfing the websites that you do know… you know… normal, safe Internet practices.

[UPDATE 2007-04-02 1:00 PM PST ]

Security Firm eEye Research has released an unofficial security patch in response to the animated cursor vulnerability. Please see their press release for details. However, Microsoft is expected to release their official hotfix or patch tomorrow.

[UPDATE 2007-04-03 3:00 PM PST]

Indeed, Microsoft has released a security patch out of cycle in order to guard against known, real-worl threats from the Animated Cursor flaw. If you have not done so already, please run Windows Update to grab the most recent patches.