Domino Effect

Sometimes, we fail to realize the degree to which systems are interdependent on one another.

DominosLast Friday, a sign appeared on the front door of the high school. “The building will be closed on Saturday due to electrical work.” The custodian clarified. The electricity at the high school will be off on Saturday between 8:00 AM and noon. If the maintenance work goes poorly, we may be out longer.

No one seemed to think that this would affect anyone except the high school. But an extended power outage like this one would have a substantial effect on the district as a whole. I sent out an email to all staff. Some major things would be broken, including the district’s web site and all teacher web sites, the email server, and our Internet connection. All of these are in the high school. Since the core switch for the data network is in the high school, data communications between the buildings would be impossible.

Then, it gets interesting. People get upset when they lose Internet access. They get irate when email goes down. But the world ends when the phones stop working. Since the phone system relies on the data network for trunking, phone service would be out districtwide except for voice service within the various buidings. But since the buidlngs themselves would still have power, their phone systems would not failover to the analog lines that they’re supposed to use when the power’s out. And since the voice mail system actually gets calls forwarded from the buildings, that wouldn’t work either.

We have a public access cable channel which carries text information about our district. That’s in the high school, too, so we would be off the air there, too.

As it turned out, they only turned off power to part of the building, and the technology and communications infrastructure was not affected. But it did serve as a wakeup call. If we were to lose the high school (or, more likely, the fiber going to the high school), it would have a crippling effect on our ability to communicate. We should make sure that our disaster management and recover plans take these things into account.