GitHub hit with major service outage after three minor outages three days in a row

GitHub is having a really bad week. The service experienced minor interruptions (Partial Service Outage, as classified by GitHub) on Tuesday, Wednesday, and Thursday. On Friday though, the social coding site was hit really hard (Major Service Outage).

We’ve been monitoring the situation all week, but since GitHub often has minor interruptions, we didn’t think it was worth reporting. When this red banner came on, and when we noticed the problems have been going on for some six hours now, it was time to dig in.

Over on status.github.com, we can see that today’s problem started late last night (this log is being updated, so refresh if you want the latest):

06:01 AM PST: Investigating network issues
06:17 AM PST: Access to all repositories should be restored. We’re continuing to investigate impacted network connectivity
06:29 AM PST: Network connectivity is stabilizing.
06:47 AM PST: All systems go.
08:00 AM PST: Investigating unicorns and stalled git interactions.
09:09 AM PST: All systems go.
09:54 AM PST: We are investigating an offline file server.
10:30 AM PST: We are working to bring the file server back online. Users whose repositories reside on this file server will have trouble accessing them during this time.
11:19 AM PST: We are continuing our work on resolving the network and file server issues. We’ll provide more information once it becomes available.
11:58 AM PST: We’ve recovered from a network partition and are continuing to work towards restoring access to all repositories.
01:38 PM PST: One fileserver pair remains offline. We’re performing manual verification checks on this server to ensure data is consistent, and are working to restore service to the affected repositories as soon as possible.
03:31 PM PST: All repositories are now available. Our sincere apologies for a rough day.

The company has also confirmed the problem on Twitter:

Service is degraded while we work through some network and fileserver issues – status.github.com

— GitHub (@github) November 30, 2012

We have contacted GitHub for more information. We will update this article if we hear back.

Update at 3:10PM EST – “GitHub is experiencing a partial service outage while we work through some network and file server issues,” a GitHub spokesperson said in a statement. “As of 12pm PT, we’ve recovered from a network partition and are continuing to work towards restoring access to all repositories. We will update https://status.github.com/ with additional information as it becomes available.”

Image credit: Marco Soscia