GitHub confirms major service outage across all its services following scheduled maintenance

On Friday, GitHub had some scheduled maintenance planned, but whatever changes it made appear to have caused some issues. On Saturday, the social coding site confirmed that it is experiencing major service outages across its products.

Over on the GitHub status page, the log shows the service started experiencing trouble starting yesterday. After that, the company disabled GitHub.com and Gists in an attempt to fix the problems (we’ll update this log as we see more, so please refresh if you want the latest):

13:00 UTC: We are beginning our scheduled maintenance. We do not expect this to have visible impact to customers, but will update status if that changes.
13:17 UTC: We are seeing unicorns related to the scheduled maintenance. We are investigating.
14:03 UTC: We have stabilized the network but with reduced redundancy. We are working through restoring full redundancy.
16:05 UTC: The network continues to be stable, though operating with reduced redundancy. We continue to work with our network manufacturer to restore full redundancy.
18:06 UTC: The site continues to operate normally but at reduced redundancy while we investigate the root cause of the incident this morning. We will leave the status at yellow until we have restored full redundancy.
23:20 UTC: We are investigating unicorns and repo access problems.
23:43 UTC: A number of repositories are currently unavailable as we diagnose some network problems.
0:12 UTC: We’ve disabled GitHub.com and Gists while we continue to resolve the network problems.
2:06 UTC: Our network has been stable for some time, but weve chosen to remain in maintenance mode while we continue to restore internal services. We are making good progress and will post additional updates as we have more information available.
3:28 UTC: Our entire team continues to work to restore internal services. We dont have an ETA at this point, but we are making very good progress. Well continue to provide periodic updates.
4:23 UTC: Service is now fully restored. It may take a little while for performance to return to normal. Thanks so much for your patience.
4:42 UTC: Everything operating normally

The company also confirmed the problem on Twitter:

GitHub.com and Gist are down for maintenance while we investigate problems in our network. Please visit status.github.com.

— GitHub (@github) December 23, 2012

Visiting GitHub.com results in the following friendly error message:

We have contacted GitHub for more information. We will update this article if we hear back.

Update at 9:45PM EST: GitHub is making progress but the problem has not yet been resolved.

Update at 12:00PM EST: All systems are working again.

Update on December 26: GitHub has an explanation of what happened over on its blog.

Image credit: Marco Soscia