Gliffy, a popular online diagram and flowchart Web app, experienced every startup’s worst nightmare this week when a member of its team accidentally deleted the company’s entire production database.
The issue started when Gliffy’s team found an issue in one of its backup systems and so scheduled maintenance to resolve it. When working on the issue, the administrator accidentally deleted the wrong database, in its entirety, according to its support site.
That meant the entire service has been offline, and users can’t retrieve any of their data or charts.
Having been a network engineer and software developer in the past, this is something that almost certainly happens to everyone once, but not always on this scale.
oh gosh oh gosh @xdissent @marcuswhitney i can't process these emotions. i may have ptsd. https://t.co/OQ1o70hGoP pic.twitter.com/z5LAjdDX2E
— kimberv (@kimberv) March 23, 2016
Luckily, the company has backups of the database that it’s trying to restore.
Not quite so luckily, those can take days to restore as they’re often enormous in size and are notoriously slow to complete – Gliffy’s trying different tactics, but is bound by how long that restore takes.
Gliffy has been struggling to get back online since March 21, and all of its users’ charts are trapped, since it’s entirely cloud-based. It’s confident, however, it can recover eventually.
To everyone who's hung in there with us, you're our heroes. We're inching ever closer to being back up and running: https://t.co/Vjd3pGqBHd
— Gliffy HQ (@gliffy) March 23, 2016
For startups, this is one of those risks that almost everyone’s familiar with, but doesn’t think will ever happen. Disaster recovery – or not having a single point of failure – is important for making sure you’re not down for days when accidents happen.
And trust me, these kinds of accidents are all too common.
Get the TNW newsletter
Get the most important tech news in your inbox each week.