At approximately 10:45 a.m. on Tuesday, July 1, the Xcel Energy power transformer feeding the West Bank Office Building (WBOB) exploded and subsequently caught fire. A number of campus-wide systems and services that are housed in that building were unavailable until approximately 10 p.m. OIT proactively shut down many of these systems in order to achieve a stable power situation.
Once a consistent power supply was established, OIT staff worked throughout the day and evening to bring the systems back online.
By roughly 10 p.m., all critical OIT enterprise servers and applications were back online and verified as functioning properly.
Technical Details
The Xcel Energy power transformer immediately adjacent to and feeding the West Bank Office Building (WBOB) exploded and subsequently caught fire. Due to this, power being supplied to WBOB and the OIT Data Center (OIT DC) within WBOB was severed.
The backup power generator supplying the DC caught the power drop and functioned as required. Upon power loss from Xcel, full generator power was being supplied at all times. The power event with the transformer and the automatic switch to generator power caused the DC air chillers to automatically shut down. This is normal operating procedure. However, once on generator power, the chillers attempted to restart as expected, causing the load being requested of the generator to exceed available capacity. Because of this, the Uninterruptable Power Source (UPS) units within the DC were unable to achieve a steady-state power situation. Subsequently, the UPS units drained their available battery capacity and then went offline. These issues resulted in an unstable power situation for the data center and numerous rounds of the power to the DC itself being lost.
These events caused many of the enterprise servers to crash and a resulting loss of software applications to customers. Unfortunately, the new Enterprise Financials System was eventually affected by this power event even though the majority of its infrastructure resides in the OIT 90 Church Street data center.
In order to achieve a consistent power situation, OIT requested that all non-critical servers be shut down, allowing the chillers to come back online and put the DC into a stable power situation for the remainder of the day.
By approximately 6 p.m. Xcel Energy had a new transformer in place and functioning. At that time WBOB and the DC were switched off generator and back to line power. A notification went out to OIT and all tenants with the "All Clear" to bring up any servers.