Just call me Halon Master Chief

No that’s not a typo. I wasn’t playing the Halo Master Chief Collection last night. That would have been fun. Instead I was playing Halon Master Chief, a game unavailable on any console. A game that can only be played in a computer room. Last night around 6 PM I was able to unlock the dubious achievement of being in a computer room during a halon dump. Sounds glamorous, right? No, it’s not.

People have this misperception about halon, thinking that it displaces oxygen in a room and makes it dangerous for people in the room.  That behavior is more in line with a carbon dioxide fire suppression system.  Halon acts by disrupting combustion process. Here is what the Halon Alternatives Research Corporation (HARC) has to say about how halon works:

Three things must come together at the same time to start a fire. The first ingredient is fuel (anything that can burn), the second is oxygen (normal breathing air is ample) and the last is an ignition source (high heat can cause a fire even without a spark or open flame).

Traditionally, to stop a fire you need to remove one side of the triangle – the ignition, the fuel or the oxygen. Halon adds a fourth dimension to fire fighting – breaking the chain reaction. It stops the fuel, the ignition and the oxygen from dancing together by chemically reacting with them.

Halon is effective in concentrations less than 8% by volume, so there’s plenty of oxygen for you to breathe in that space as you evacuate the room. I’m not saying you should hang out and relax in there, but you’re not going to pass out or die. From my experience, the biggest hazard of being present during the dump is you might get knocked on your butt. More on that in a bit.

So what happened?  The main cooler in our computers room is quite old (our backup cooler is even older), and the decision was made a while back to not spend the money needed to replace them with new units. No coolers means no equipment, so we started moving our datacenter to racks in the central campus datacenter.  I’m still in the early part of that process and have to be completely moved no later than May 2015. The coolers can be finicky at times, but overall haven’t been a big problem…. until last night.

I stayed late and saw the night maintenance guy stop by to check on a low humidity alarm we’d been seeing all day. At some point I started to smell a burning, acrid smell. Not good in a computer room. I stepped out and looked into the mechanical spec and asked if he was doing something to cause that smell. He said no, and just then smoke really started to come out of the unit and fill the room. He turned off the cooler at the panel, and then the fire alarm went off.

A fire alarm in a computer room means two things — the power is going to be cut and the system is going to dump halon into the room — unless you push and hold the abort button. As the mechanic headed out of the room to get to the fire panel, I pushed the abort button that was right there by the cooler, expecting it to stop. Nothing. I push it again. It suddenly dawns on me, I’m an idiot. I have to push and hold the button in to keep the halon at bay until someone can disarm the system.

I then quickly realize that if I’m going to stand there and keep the halon at bay, I’m doing so next to a smoking piece of equipment that could actually be on fire for all I know. So I decide to do the safe thing and exit the room. Ideally at this point, maintenance guy at the panel would press and hold the abort button and give me time to get out so we could assess the situation from the safe side of the big glass window.

That’s not what happens We couldn’t communicate over the alarms, and even if we could have done so, it wouldn’t have mattered much because I forgot all about the abort button out there. Instead what happens is I get within four feet of the door, and the halon dumps. The force of the halon coming out of the nozzle (almost directly above my head) knocked me over. I managed to put my arm out and keep from falling flat on my dupa (it’s Polish, look it up). Startled doesn’t even begin to cover what I felt. I gathered myself and exited the room, papers swirling all around me like some halon-fueled snow storm.

Looking through the glass at a smoky, hazy computer room was a horrible feeling. Scenarios ran through my head, none of them good. Even if there wasn’t an actual fire (there wasn’t), at the very least I’m down to one cooler, and I have servers and a SAN that all went down hard when the power was cut.  We’ve been through an emergency power off situation a few times before and they’re always a nail-biter as you wonder what might not come back. That’s when the maintenance guy tells me he doesn’t have a key for the fire panel. Of course he doesn’t. The only key I’m aware of is in my office in the binder for the halon system. Not seeing any signs of a fire after the dump, I headed in to grab the key. So he finally gets the system disabled, and then we start making phone calls and texting.

After about an hour we open the room and get blowers to clear the air of what smoke is left, then start recovering:

• Backup cooler operational? Check
• Main breaker? Reset. (Lights and power!)
• Main UPS and panel? Tests OK. Reenergized.
• Network back up? Check
• Domain Controller w/local storage only? Check.
• SAN? Disks powered up. Controllers powered up. Check.
• Other domain controller? Check. (Users can logon)
• File Server Cluster? Check (Users can logon AND access our files)
• Print Server Cluster? Check (Users can print)

At this point, I’m feeling much better about the situation, so it’s off to Steak and Shake to get some food and get out of that room for a bit. At this point, I can see via my Microsoft Band that my heart rate is now dipping back down into the double digits from the elevated rate from earlier. (Does that count as an aerobic workout?) The next few hours were spent bringing up more systems. SQL Servers, Exchange, web servers, vSphere cluster and its VMs, and Hyper-V clusters and their VMs.

By midnight we were back up and completely operational. The only casualty was an ancient Cisco Catalyst 2900 that was being used to connect DRACs for some old Dell servers and managed PDUs.  I don’t know about you, or how often you’ve had to recover form a complete shutdown (and not a controlled one at that), but to come back without losing any disks or servers is pretty darn good, especially considering the age of some of our equipment.

They came and carted off the now empty halon tank, and the question is what replaces it. Our maintenance people are saying they can’t refill with the same material (damn ozone layer). Alternatives require some changes to the piping and apparently the price tag isn’t something they’re excited to spend. Now they’re asking how quickly we can finish the move to the other datacenter. So that’s what we’re looking at now. I’m pretty confident we can dramatically accelerate things, but that means a lot of other projects are going to get set aside for the time being.

Obviously if I could go back and do it over, I would have aborted the dump. The cooler issue was apparently caused by a broken pulley  which led to a broken belt which led to something smoking. I’m still waiting to find out the details that went from belt to smoke, but there wasn’t an actual fire. I didn’t know that at the time, however, and as big of a pain everything was afterwards, I’m going to take a halon dump and all of that hassle over possibly dying in a fire. But hey, I’m weird that way.

It probably isn’t good that we only have one key for the halon panel, and if there is one, it probably shouldn’t be in my office inside the computer room.  We should probably have also had training on the system and defined procedures covering what to do in case of various scenarios. It sounds obvious, but not unusual for that formality to be lacking in a small group like ours, especially in higher ed. Live and learn.

So that’s my story. Hopefully you enjoyed it. Not sure if there’s anything specific to take away from it, but if you do find something that you can use, great.  All I know is that I can safely say that Halo is better Halon (unless you’re trying to put out a fire), and I sincerely hope that I never have to go through that again.