Questions raised around Amazon’s ‘lightning’ claims at Dublin data center [Updated]

You may have read about the ‘lightning strike’ and subsequent explosion that took out Amazon’s Dublin-based data center at the weekend, forcing its EC2 cloud computing service into unplanned downtime. It affected customers for up to 2 days in some cases. Amazon said in its statement:

“The transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the backup generator plant, disabling some of them. Power sources must be phase-synchronized before they can be brought online to load. Bringing these generators online required manual synchronization.”

This is a massive data center we’re talking about here, leading many to question why such a scenario couldn’t be prevented. But in a twist, a report in Silicon Republic today has cast doubt on claims that a lightning strike was at the heart of the problem at all.

It seems that ESB Networks, an Irish electricity provider, has given a different version of events. A spokesperson from ESB Networks told Silicon Republic that the problem arose following a fault in one of its Citywest substations, noting that an alternate power source should have kicked-in automatically in less than a second.

The outage, according to ESB Networks, happened at 6.15pm on Sunday evening. The spokesperson told Silicon Republic:

“In relation to Amazon, it would have experienced less than a second (of downtime) when the centre would have been switched to an alternate supply.”

The spokesperson added that there was no report of an explosion or fire. And to compound matters, the Met Office said there was no record of a lightning strike in the Dublin area at the time.

It’s not clear yet whether Amazon was quick to jump the gun by blaming a lightning strike when something else entirely was to blame, but it’s the second major problem one of its data centers has experienced in four months, with another outage reported back in April affecting some US-based customers.

Microsoft also experienced issues with BPOS, its online desktop application suite, resulting from power issues in Dublin at the weekend. The company said at the time via a Twitter feed:  “European data power issue affects access to BPOS”, before saying a few hours later “BPOS customers are back online for EMEA customers.”

So it’s clear that something did happen in Dublin that caused more than one data center into downtime, but with conflicting versions of events there are still questions needing answered.

Update

ESB Networks has provided this comment in relation to the electricity outage on Sunday, August 7th. It says that its original assessment that a lightning strike was to blame was wrong:

“ESB Networks can confirm that at 18:16 on Sunday August 7th, a number of customers in CityWest lost electricity supply.  It’s an unfortunate reality for every power system that faults can occur due to vandalism, storms, fires, floods, plant failure and third-party interference.  In this case, the problem was the failure of a 110kV transformer in the CityWest 110kV substation.  The cause of this failure is still being investigated at this time but our initial assessment of lightning as the cause has now been ruled out.  This initial supply disruption lasted for approximately 1 hour as ESB Networks worked to restore supply. There was an ongoing partial outage in the area until 11pm.  The interruption affected about 100 customers in the Citywest area, including Amazon and a number of other data centres.  Another Amazon data centre served by ESB in South County Dublin was not directly affected by the outage, though it did experience a voltage dip which lasted for less than one second.”