When Disaster Strikes

We have been working on moving Network Computing's Green Bay lab for more than a year, and last week finally had the opportunity to tour the new lab space as they were building it out.The reasons for our move were many, most revolving around being in the basement of a building with a sump-pump in it. I have feared a backup of that sump pump since I first came on staff.Little did I know, that the sump-pump was the least of our worries. Last night we received alarms from our NetBotz that said humidity was approaching 100% and the temperature was 245 degrees. We hopped on our Axis camera and took a look at the lab. All was dark (as it should be when no one is there), but the lights of the machines were glowing green in the darkness and there were no signs of fire - something you'd see in the dark if the temperature was 245 degrees in the lab.

We made some jokes and decided it was a non-issue.

We should have listened to our trusty little NetBotz. It seems that one of the tenants above us had a hot water heater, and that hot water heater broke. The rising humidity was the water pooling in our drop ceiling and evaporating. The temperature spike was when the ceiling gave way and dumped many gallons of hot water on our running infrastructure.

I have no idea how we escaped without a fire. Dozens of servers were dowsed in gallons of water.

We spent our day trying to dig out of the mess. Steve and I missed all of our calls for the day, and Lori had to send her vendor home - there just wasn't any way to test.We were taking down machines and switches that were four or more feet off the ground and pouring water out of them before they were moved to a dry location.

So now we're moving to a new lab, but have precious little gear to put into it. It appears that we can completely salvage at least 10 Dell servers, and it's possible that our Cisco Catalyst 6500, which is a long-term loan from Cisco survived, but dozens of other machines along with a couple dozen switches, storage boxes, and routers are "a wash" - pun intended.

Here's the punch line. We were in the process of configuring off-site replication with the Syracuse lab. There are no backups because of this - it was all in a state of local replication to our Adaptec Snap Server. Our Snap Server that poured a couple of quarts out when we pulled it out of the rack.

The real problem is NWC Inc. Conceived and built by Lori, with very little help from the rest of the NWC Staff and a little more help from freelancers, our entire online simulated business is gone. One of the ceiling tiles that let loose was directly over the four NWC Inc. racks, and all of the servers from those racks are soaked.

Lori has poured her soul int NWC Inc. for the last five years, making it into an astounding simulation of a real enterprise, and she's going to have to start from square one. Steve and I will help her, of course, but it is definitely a disaster.We'll keep you posted. We took pictures for the insurance adjusters and I'll get some of them up here for you as soon as we get them off the camera... Needless to say, we're a little busy.

On the bright side, we get to redesign our IT Infrastructure...