TNW Conference 2022 will be bigger, bolder, and better! Get your tickets now >>

The heart of tech

This article was published on March 2, 2017

    How a single typo brought the web to its knees

    How a single typo brought the web to its knees
    Rachel Kaser
    Story by

    Rachel Kaser

    Internet Culture Writer

    Rachel is a writer and former game critic from Central Texas. She enjoys gaming, writing mystery stories, streaming on Twitch, and horseback Rachel is a writer and former game critic from Central Texas. She enjoys gaming, writing mystery stories, streaming on Twitch, and horseback riding. Check her Twitter for curmudgeonly criticisms.

    This week, an Amazon Web Service (AWS) failure caused a massive outage all over the internet. Today, we know why: a typo.

    The company released a detailed report today explaining what happened. An employee entered what they thought was a routine command to remove servers from an S3 subsystem. By mistake, they entered a larger number than intended. These servers supported two other S3 subsystems, both of which manage the storage and metadata for the entire region. Down went the dominoes.

    AWS assures everyone that it’s prepared for the occasional failure. Fixing the employee’s error should have been as simple as rebooting the subsystems. However, AWS admitted that it hasn’t actually restarted those subsystems in years, and S3 has grown considerably in the meantime.

    If you’ve ever rebooted an older computer and notice it chugging on start-up, you’ll understand the feeling AWS must have had while waiting for the system to come back.

    In response, AWS says it’s putting in safeguards to prevent this kind of error from happening again. Apparently, creating these safeguards had been in the works for a while, and AWS is making it a priority after this outage. Considering it even brought down AWS’s own conference mid-speech, this is probably for the best.

    Get the Insider newsletter