A Cautionary Tale

As Robert burns famously wrote "The best laid plans of mice and men oft go astray." As we moved the lab from SUNY to our new digs in beautiful New Jersey a series of failures convinced me that he was right, and that Edsel Murphy PhD had come to visit. First, we discovered that the power supply in one of our key VMware hosts, a Dell PowerEdge 2950, didn't survive the trip. Since the lab isn't a production environment, we didn't equip it with redundant power supplies, which turned out to be a bigger mistake than we would have believed as both of the replacements I ordered from eBay turned out to be duds. But wait. . .there's more.

Then the gremlins started in on the Internet connections. The lab shares a building, and internet connections, with a medical office management company. Before we moved in, they had a 70mbps point-to-point wireless link and a Sprint T-1 for backup. We ordered a Verizon FIOS line when we moved in. As our little story opens, Verizon had come to install the FIOS, but we hadn't actually integrated it into the link load-balancer or even tried to use it.

On Tuesday, the wireless link went down and the provider expected it to stay down for a week or so. Thursday, I went to the lab and found a tech from the medical folks' VAR working in the telco room, which should have made an old network hand like me nervous. 

A few minutes later, my Pandora stream died as the Sprint line bit the proverbial dust. Turns out the battery backup for the telco mux had failed rather spectacularly actually, with smoke and leaking acid. The telco stopped supporting that backup system, so the VAR had ordered replacement batteries and our friend the tech blew up the mux by changing them without disconnecting the backup power from the system.

No problem, I said, we'll just use the FIOS line. But the FIOS line was only configured to supply one IP address and even that was down. Strike three!So my erstwhile intern and I switched gears to build new golden images for the server operating systems. The Dell 1850s needed Windows Server 2008 after all. But some cheapskate (oh, wait, that was me), ordered the 1850s with CD rather than DVD drives and Windows 2008 is on DVD. Plug in the USB DVD drive, and we discovered that to boot from the USB DVD, we needed a BIOS upgrade, which we could download from the Internet. No problem, if all three Internet connections weren't down.

This all makes me wonder about using cloud computing, and especially cloud storage, for my primary apps. Even with three Internet connections, I was cut off from email, my web CRM system and storage at Nirvanix. Yes, I could have had more robust connections, and yes, we could have just plugged the mux into the 40kva ups, where it's now powered, but you really can't make things fool-proof. Fools will always find a new and inventive way to foul things up.

PS:I called the cable company so next week we'll have four Internet connections.  When all of them fail, I'll have another story to tell.