How a single typo brought the web to its knees

This week, an Amazon Web Service (AWS) failure caused a massive outage all over the internet. Today, we know why: a typo.

The company released a detailed report today explaining what happened. An employee entered what they thought was a routine command to remove servers from an S3 subsystem. By mistake, they entered a larger number than intended. These servers supported two other S3 subsystems, both of which manage the storage and metadata for the entire region. Down went the dominoes.

AWS assures everyone that it’s prepared for the occasional failure. Fixing the employee’s error should have been as simple as rebooting the subsystems. However, AWS admitted that it hasn’t actually restarted those subsystems in years, and S3 has grown considerably in the meantime.

If you’ve ever rebooted an older computer and notice it chugging on start-up, you’ll understand the feeling AWS must have had while waiting for the system to come back.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

In response, AWS says it’s putting in safeguards to prevent this kind of error from happening again. Apparently, creating these safeguards had been in the works for a while, and AWS is making it a priority after this outage. Considering it even brought down AWS’s own conference mid-speech, this is probably for the best.

The moment the presenter realizes S3 is down. #AWS #S3 #Amazon pic.twitter.com/S8gY1J9lrH

— Ian Sherwood (@ian_surewould) February 28, 2017