We just don't have the imagination...
Does your DR plan include a section for earthquakes? Facility fire? Power loss? Flooding? Hurricane? Tsunami? Asteroid strike? The sky falling? Is every Act of God in the 'DR crash' file, can you possibly imagine every disaster that could disrupt your business? Presumably not.
New to Southern California back in 2002 I sat in our data center with its seismically rated raised floor, braced racks, strapped down, well, everything learning as much as I could about the rolling blackouts that in 2001 had threatened our San Diego headquarters. Within a few weeks I could recite from memory every component of the power systems extending from the local utility substations down to each server's redundant power supply, and every UPS, transfer switch, distribution panel, PDU, transformer, conditioner, service vault and generator in between.
For IT managers throughout San Diego County it was the same one-two: sometime soon the ground will shake or the lights will go out, maybe both. The two active airfields a mile either side of our headquarters made a worthy footnote, having taken a quick look at local hazard maps, briefly considered tsunamis and hurricanes (both rare but conceivable in this part of the world) and talked to other local IT managers about their risk assessments it looked like we were on the right track. The ground will shake, the lights will go out - or we'll be very, very unlucky and hit by something falling from the sky.
The risk assessment changed abruptly on October 25th, 2003. In less than a day the Cedar wildfire devastated a 30 mile track from the back-country in to the City of San Diego, destroying suburbs and business parks in its path - by the morning of the 26th the fire was raging half a mile North and East of our headquarters, and the last of our on-site employees had retreated with a case full of backup tapes. Wildfires are of course nothing new, even in the heavily developed suburbs close to the city - but no-one imagined that an uncontrollable fire would travel that far in just 16 hours.
Our emergency plan, complete with example scenarios (ground shaking, lights out) didn't mention wildfires. But it didn't need to. While we consider the specific threats - and earthquakes, now joined by wildfires, make two of the more obvious examples - our BC and DR planning is based around general categories of disruption:
• loss of services in our datacenter (for example Acts of Man, power, telecom, cooling etc)
• loss of access to our datacenter (as happened during the Cedar fire)
• loss of the datacenter facility (for example localized fire or flood)
• loss of the site (for example things that fall from the sky)
• regional disaster (for days when the sky is falling, the ground shaking or both)
Disasters never follow a script - but having documented,
discussed and rehearsed responses to the general problems that accompany disasters will help you quickly piece together an appropriate response. Thankfully even Acts of God have predictable effects on businesses and the information systems that support them, I hope to be around after the sky next falls to say 'I told you so'.
--
James Deveson
Director Information Technology