In an online discussion in July, we were talking about Moodle hosting. I commented that self-hosting a Moodle server is fairly easy for most school districts, but that you do need someone on staff who has a pretty good understanding of what’s going on. Others in the discussion felt that a hosted solution would be better, because the teachers could focus on the coursework and the tool rather than the network and the server. Dave chimed in, and claimed that network support people have an inflated sense of self-worth. What goes wrong? Once you set it up, it just works. He was probably just trying to give me a hard time. Dave has a great sense of humor. But of course, I responded by saying you need someone whose cell phone rings when things don’t go as planned. Yesterday was a great example of that.

Apparently, our high school had an extended power outage on Friday night / Saturday morning. The high school is the head end of our data network, and also houses our web and email servers. It took a few minutes on Monday morning to realize that there were some problems, but here’s what was broken:

The email server did not restart cleanly after the power outage. It was not accepting SMTP connections, so no one could send mail to it. This means that it also wasn’t receiving any messages from the outside world, and it had a two-day backlog of messages waiting to be delivered to it. Once we restarted it, it was operating at 1000% – 2000% of its designed load for the rest of the day.

We have a wireless link from our high school to the maintenance and transportation facility about a mile away. This link was down, meaning they didn’t have network access in that building. The maintenance and transportation people are about the only ones in the district who start work earlier than I do.

Two of the ten network zones in the high school didn’t have access to the rest of the network, because their fiber links were down.

The cafeteria point of sale system didn’t work, because the POS terminals couldn’t communicate with the backend server.

One of the file servers didn’t restart after the power failure because the cooling fan wasn’t working. Some quick thinking by Dan solved this problem fairly quickly after we liberated an identical fan from a doorstop recently pulled out of a library.

Another file server didn’t restart at all. The elementary school that it serves didn’t notice, because it was back up by the time they came in.

All of this happened before 8:00 AM, and most of it was fixed by then. Last night, my wife came home and handed me a new cell phone. That’s good. I think I wore the old one out yesterday.