A Power Outage and Recovery

Well, I think I jinxed us...talking about how good the Crisis Management was getting at dealing with situations.

For the record, at approximately 12:15 am Friday morning, we experienced and electrical short that took out power to B, D, X,  G, the CDC and part of A wing.

 Photo by: Vanda Bidwell/The News Gazette   

How did this occur and what can we do to prevent it from happening again?

The cause was aging cables. There is really no way to prevent it from happening again, short of re-wiring the entire utility ring. There is a ring of underground utilities that surround the campus, accessible by manholes spaced accordingly. Between the B and D wings, the electrical cable shorted out.  We lost power, air conditioning, and eventually, the uninterruptible power supplies (UPS) that keep our network and servers online.            

Had this short occurred just one week later, we would have had our new UPS and generator ready to take over. Unfortunately, this wasn't the case. Had the new system been ready, the UPS batteries would have kept power to the servers with the natural gas-fired generator taking over indefinitely.

So our systems went down rather un-gracefully. 

AmerenIP worked through the night along with Troy Burns and Jim Bustard from the Physical Plant. Of course Gordon Green, Jim Pierce, Doug Brooks, Mark Guymon, Bryan Diller and possibly others that I am not aware, came to the College in the middle of the night to do what they could.

At approximately 11:30am, power was restored. It took another 3 hours to bring the network and servers back online.

Typically, we have some student service functions open on the Saturday before the semester begins, so we decided to extend the hours a bit. Here is the News gazette story.

Why did it take so long for computer systems to be restored? Why did they go down in the first place?

The Server Room

Bad timing. The "old" uninterruptible supply lasted for only @ 15 minutes before power ran out. This is exactly why we bid the replacement.

There are approximately 20 servers and sub-systems that need to be restored when power is interrupted. These systems must be cooled, turned on in a certain order, and verified/indexed. This takes 3-4 hours. Our thanks to Gordon Green and Doug Brooks, along with their teams for getting us back in service so quickly.

This morning (Saturday), we opened at 9am and served a significant number of students. By 10am there were nearly 80 students waiting in the Counseling and Advising area. Admissions, Financial Aid and the Business Office had steady traffic all day until we closed up for the day at 3pm.

At the risk of leaving someone out, I won’t try to thank everyone that came in on a Saturday to help out, but I’d like to recognize a couple special folks.

Billie Mitchell from the Adult Re-Entry Center served as an Admissions Advisor all day. Mary Katherine Denmark from TRiO became a Financial Aid Assistant, and in the Business Office, Lori Wendt, Kathleen Charleston, and Melanie Lewis served as cashiers. My special thanks to all of you for stepping-up today.

Days like today make me glad I work at Parkland College.

A poem by Trustee Linden Warfel:

Is this the 'zap' semester?

When things get very dark

and then the lights come on!

Brighter and brighter

and more long lasting

A shake down day

checking out emergency response

with the president 'down', well,

out of town

But the good people got it together

and the president came

and the computers hummed

after their little rest

Ready, now, for work to begin!

'charged to the hilt'!

Full power!

Let 'er rip!

Full speed ahead!

No turning back!

The light house is back!

Bring on Monday!