Geekularity

On Saturday night I joined at least four hundred other geeks at the San Francisco Apple Store to watch Will Wright, the creator of the online games “Sim City” & “The Sims,” as he demonstrated his new game, Spore, which is scheduled to debut on September 7, 2008.  In addition to the opportunity to truly geek out, I observed two points of Mr. Wright’s presentation style that I both admire and hope to mimic in my own presentations:

1. True passion cannot be faked. When you are truly passionate about your project, your audience will recognize its authenticity almost immediately. In terms of building emotional capital with your audience, this is possibly the largest deposit you can make in a single transaction. Mr. Wright shows tremendous passion for this new game and we, the audience, locked in on it within the first 60 seconds of his presentation.
2. You can be “The Dude” with out ever having to utter the word “dude.” First of all, it helps to wear a Tommy Bahama aloha shirt when you’re at the podium. But even without it, Mr. Wright was incredibly laid back while at the same time eloquent and responsive to the audience and to the game. During his demo, he exposed several significant bugs and without batting an eye, he called them bugs, and moved on. The tone of the presentation was simply: I’m going to show you some really cool #$%^! Not all of it’s going to work, but who cares? Let’s have some fun!

When all is said and done, this was effectively a sales demo. I bought a copy of the Spore Creature Creator, a character editor program which serves two goals. First, it’s a teaser for the upcoming game. Five minutes playing with the editor and you’ll be hooked. Second, it’s a community catalyst. The fan base gets to help build the species inventory for the game a good three months ahead of the game’s release. Here’s my first creation Zweedoo:

I can’t believe I missed this last month. Shortly after Garry Gygax passed away, Adam Rodgers of Wired Magazine posted this great op-ed piece complete with a brilliant flow chart of geek passions. The article is well worth a read, and the flow chart is a fun one to trace your own path through.

I spent the better part of the day and night yesterday recovering from a power failure at one of my client’s data centers; a failure that we did not have a contingency plan for. A transformer inside of the building, one that we thought was solid-state and not a concern, decided to self-destruct in the middle of a busy day, bringing down half of the office and all of the IT infrastructure.

We had battery backups and a diesel generator in the event of power loss, but the diesel generator connects to the building on the other side of the dead transformer, so once the batteries died, we were down. The only thing that could have saved our bacon would have been keeping about 1,000 feet of industrial extension cords on hand to run between the generator’s auxiliary ports and our most critical systems. I think the IT director is putting in for a purchase order this morning.

So, with this little nightmare behind me, I thought I’d try to open a thread of top lessons learned while implementing and supporting server infrastructure. These can be hard-learned lessons, or near misses that you’d like to see IT professionals think about and revisit periodically. I’ll start it off with my top 10, which I’ve picked up in my decade of IT industry experience:

1. BACK IT UP!.. AND THEN CHECK YOUR BACKUP. I’ve sent way too many hard drives to a clean-room laboratory to try to resurrect the data off of the failed media, all because the victim either didn’t back up their data, or never bothered to check that their backups were working. This is a sophomoric mistake, but it may take a $20,000 invoice from the data restoration company for a systems administrator to finally get religious about backups.
2. Get servers with redundant power supplies. Plug each power supply into a separate UPS, and each UPS into it’s own breaker, preferably on different legs of the building’s 2 or 3 phase circuit. This will allow you to swap out UPS batteries without bringing down the system and minimizes your exposure to building infrastructure problems (like I experienced last night).
3. Label every outlet in your data center. At minimum, each outlet should have the breaker number and panel location clearly labeled. You don’t want to be searching franticly for the correct breaker when your UPS units are beeping at you.
4. All electrical circuits used by your core IT infrastructure should be dedicated circuits. Do not use shared circuits, especially in locations where office tenants could plug in appliances like coffee makers and space heaters, or housekeeping could plug in a vacuum cleaner.
5. If you have more than one server, label it on the server. If the servers share the keyboard, monitor, and mouse through a KVM switch, make sure that the switch is also clearly labeled, and that the labels are correct.
6. Have a startup and shutdown procedure for bringing all of the systems down and bringing them back up again. Your data center is an organism, and there are critical services on some machines that need to be up and running before other systems can function. Make sure you know which servers or appliances are hosting DHCP, DNS, SysLog, Active Directory, etc., and make sure those devices are high on the boot order.
7. For the love of all that is good in the world, use velcro or zip ties to clean up the cabling around your servers. Keep the wires as short as possible and try to prevent any sort of rats nest wherever you can. It will pay huge dividends later when you are trying to isolate essential from non-essential power cords. Plus, clean wiring will promote air flow and will prolong the life of your equipment. If a cord hangs down below where it’s plugged in, it’s too long. If it touches the floor, it’s too long. Any loose cord that’s near the floor or at about hip level where most geeks keep their blackberry holstered, will undoubtedly get pulled out accidentally if not properly secured.
8. Disks will fail. It’s not a question of IF, it’s a matter of WHEN. So, for every RAID array you maintain, keep one or more spare drives on hand and readily available. If you use your spare, it is imperative that you order a new spare on the same day. Do not put this off.
9. Have an emergency resource guide inside your data center with phone numbers and reference information. Check and update this information regularly. Phone numbers should include electricians, HVAC, plumbing, fire sprinkler contractors and your building’s facility services hotline. Also include the cell and/or home phone numbers of any company executives that you may need to get emergency purchase approval from. Reference materials must include at the very least, all telco and ISP account and circuit IDs. If the resource guide is securable, you may want to include root and administrator login information for your critical systems.
10. I don’t drink coffee any more unless it’s in a traveller mug with a close-able lid. My son calls it “Daddy’s sippy-cup.” Even still, coffee, soda, water, or whatever should never enter the server room. I can still recall, in vivid detail, the day the CEO of the company I was working with, dropped his mug three feet in front of an open server cabinet. In slow motion, I watched as a splash of coffee arced gracefully from the shattered mug and into the front of a server’s hot swap drive bay. The result… refer to the invoice mentioned in tip #1.

So this list is just a starter. Please add your tips and tricks to the comment section below!

Ignoring that faint voice in the back of my head that tells me that Google may indeed be Skynet, and that our robot overlords are just a few years away from their planned invasion, I decided to hand Google the keys to my kingdom. I have moved several of my internet domains over to Google Apps. What that means is that all of my Email, Calendar, News, and soon phone calls, will be hosted (and probably indexed) by Google. In the coming weeks, I will move all of the domains I manage, both personal and professional over to this amazing service. This excludes my clients whom I manage, but do not actually host.

I’m doing this because hosting my own email is both tiring and expensive. Rather than spend countless hours being my own systems administrator and doing endless battle with the spam trolls, I decided to delegate this little bit of geekery to someone who is more proficient. When complete, this move will hopefully free up some significant mental energy (and funds for that matter as the Google Apps basic package is free) to spend on other pursuits. Besides, Gmail and Google Calendar are easy to use, easy to access, and quite a good value for the price!

Right now, I’ve only moved my professional domains, but my [family] clients and my personal domains will soon follow. When all is complete, I should no longer need to rent a dedicated server, and my monthly recurring expenses should go way down. This comes after reading Scott Hanselman’s post about doing the same thing for himself and his entire family. So, special thanks to Scott for outlining his process and his reasoning. It was a big help.

I didn’t know that RAID was a bad word, much less a bad acronym, unless you are an insect. But after reading the Drobo product literature and watching Robert Scoble’s interview with the Data Robotics, Inc. management team, it became clear that Drobo’s marketing machine wants nothing to do with the term RAID. Apparently RAID has become synonymous with enterprise-level, expensive, hard to setup, and hard to maintain. Wanting to target small businesses, small offices, and home users, Data Robotics, Inc. calls their product a Data Robot (Drobo for short); a protected storage solution which uses industry standard data protection methods. Marketing gloss aside, I think that the Drobo product embraces the best of what RAID was supposed to mean: Redundant Array of Inexpensive (or Interchangeable) disks.

The Drobo unit accepts up to 4 disks, and will take any 3.5″ form factor Serial ATA disks you already have, or purchase and optimizes them to run as one large virtualized disk. No matter what size the disks are (unlike other RAID solutions, the disks do not need to be the same size in the Drobo appliance), the Drobo operating system will build the largest volume possible while providing at least a minimum level of protection against hardware failure. Whether it’s just two disks, on which it builds a mirror (equivalent to RAID1) or three or more disks where it creates a striped set with parity (equivalent RAID 5), the Drobo OS takes care of all of the configuration for you. It will even build a redundant mirror when a single disk is inserted. However since a single disk mirror will still fail if the single disk stops spinning, I do not recommend relying on this configuration for hardware data protection. The Drobo appliance will even rebuild and resize the partitions when additional hard drives are inserted in real-time without the need to bring the volume offline.

I’ve been evaluating my Drobo unit for two weeks now, and I set it up to run attached to a Mac Mini, which acts as my office server and media center. I can then share my files with my other computers using several protocols including Samba, Apple FileShare, or SSHFS. The unit was easy to set up, and runs very smoothly. With the exception of some noise when the cooling fan ramps up to full speed, the unit is normally very quiet. I purchased the $500 Drobo appliance, but I already owned the four disks that I put into the unit, all 200GB drives. This configuration gives me a total of 550 GB of space to hold my data. The remainder of the space is reserved for data protection and for system overhead. As drives fail, or as I need to expand my storage, I can simply upgrade the disks one by one and without taking the appliance offline. I will simply slip out the oldest or smallest drive and insert the new one. After a few minutes, the Drobo appliance will have formatted the drive and resized the volume to utilize the new disks.

Although I am very happy with the data protection that the Drobo appliance provides, I do not consider it an excuse to not backup. A protected storage appliance like this will not protect me from fire, flood, or other disaster, so I still backup all of my critical files nightly to a removable drive which I swap out each week and take with me off site. In the future, I plan to setup an rsync relationship with a computer at my house and move all of the changed files across the wire each night through a cron job. But that’s a project for another day. All and all, I’m very happy with my Drobo appliance and would recommend it for individual users and small offices with large storage needs, but who do not have large throughput needs. The Drobo appliance attaches to the host system using USB 2.0 and the specifications claim that it has a sustained transfer rate of 22 MB/s. This is enough for an individual to run a video editing project on it, but it may bottleneck when I/O intensive databases try to move large chunks back and forth. At that level, you would need to build a more enterprisey RAID solution anyway.

Thanks to Chris Messina for giving me an invite to the Skitch beta program. This is a cool little tool that allows you to capture screenshots and web cam images, annotate them, and then upload them to numerous locations. It’s very simple and intuitive, and I like it a lot so far. Right now, particitpation is invitation only. So, as soon as I get my hands on some invites, I’ll post here or on twitter for all who are interested.

This is the signage that faces out to a busy street in my neighborhood. I couldn’t help but giggle when I was stopped at the light and finally read the sign!

So, if you are absolutely jonzing for milk tea and karaoke, bring your laptop along and get some work done between your encore sets.

Justin Kan is 23 and the CEO of the newly live company justin.tv He wears a camera on his head 24 hours a day and broadcasts his life, in real time, to anyone who wants to watch. Yes, I reluctantly include myself as one of his followers. It’s really addictive! So between justin.tv, all of the buzz over twitter (disclosure: also an avid user), and a recent report from the Wall Street Journal (link - behind pay firewall sorry!) that says that more than half of the tech companies that are currently seeking IPOs are in the Red, I’m ready to admit that I am worried.

I’m listening for the *POP*

You know the same *POP* that we all heard in 2000 when all of these crazy ventures were dumped on their arses the first time. Except this time, I’m listening for the entire San Francisco Bay Area and Silicon Valley to collectively mutter “Wow man! It’s like déjà vu all over again!” shortly after this bubble explodes.

I hope justin.tv proves to be a wildly successful marketing venture; if not, one heck of a cool socialization experiment. However, it registers in my mind as technology hysteria which, while really cool, has no sustainable business model. It’s very similar to what I saw here in the Bay Area in 1999, and I think it’s just history repeating itself. Nevertheless, I’m along for the ride Justin! I just hope your sticking any money you make from this under a mattress or into the bank, because I think you’re going to need it fairly soon!

Am I too late to play in the lolcats meme? Here’s my addition to the party!

This will be what users see on 5XX server messages on some of my sites. Yay, the Internet is so fun… And so cute!

You’ve been a part of my daily routine for a year, and now I will find something new. It’s been difficult to think for myself again -blink-

Thank you to Ze Frank and The Show for such a wonderful addition to the Geek Zeitgeist!

P.S. We think Ze’s facemelting didgeridoo solo (pictured at right) produced the brown note for our son!