I felt bad for our hometown 49ers on Sunday, but I felt really bad for the facilities team on site at the Superdome. If you’re involved in facilities, you’ve probably experienced that same sinking sensation yourself: something is going sideways in your facility, the clock is ticking, and you are the one leading the charge. Even though the eyes of the world were on the Superdome on Sunday, many facility managers face considerably higher stakes than a 30-minute delay of a game.
The blackout is still under investigation, but it looks like the official diagnosis is circuit overload due to fault-sensing equipment co-located with the utility feed. Power was quickly restored, but energy services, notably lights in the massive HID array, required half an hour of warm-up time to restore.
What are the lessons for you as a facilities professional?
The first is to manage expectations from the get go. The public reaction to this blackout is a mirror of how tenants and occupants react to an reduction in service: frustration, finger-pointing, and a dose of disbelief that this is happening to them. Most people assume the utility is at fault. The utility will respond quickly to prove they are not. Just five minutes in, Superdome power provider Entergy went public claiming they were not to blame. Most regulated entities don’t place a high priority on customer relationships, and will be chiefly concerned with public image. The blackout? It’s your problem, period.
The second is that special events are, well, special. If you don’t test from end to end using real-world demand levels, you can never be sure what to expect. The Superdome is frequently used for games, but was load a little higher than normal? 10% higher? Was that ever tested? What about the weather? The ideal for critical facilities is system-level testing for peak conditions. This is hard to do, but most facilities can benefit from more rigorous loaded testing rather than testing point pieces of the system.
Third, those of you with old facilities should exercise extreme care. Remember the roof leak during Katrina? The entire stadium was rebuilt, but do you think all the electrical infrastructure was? Any time old infrastructure is involved, special caution is warranted.
Finally, watch out for the opportunists. Everyone likes to make hay in a disaster. Check out the tweets from Kirstie Alley (funny), a solar financier, an advanced battery manufacturer, and a provider of demand response services. None would have helped during this event (solar panels?), but they’ll be phoning you shortly. Even coal giant Peabody got in on the party with a laughable press release extolling the reliability of coal. Perform a solid post mortem and put in place a considered plan for disaster prevention in the future before taking outside calls.
Nice. You’ve got to understand the problem in order to find the solution!