TwitterCounter: how do we keep up with Twitter?

TwitterCounter is growing fast and is taking more and more of our time. We try to keep up with Twitter as it grows and becomes a mainstream media company. Arjen recently did an interview with a Dutch blog about the technology behind TwitterRemote with some interesting stats which I thought would be nice to share with you here too.

Currently our whole infrastructure consists of just one server at SliceHost which was tuned to give us as much performance as possible. We used to host TwitterCounter in Europe but because it has to talk to the Twitter API a lot we decided to move everything to the US. This turned out to be a very good decision as the connection time to Twitter is three times faster from the US than Europe which saving us 40 hours of waiting time a day.

Our server often handles more than 200 mysql queries per second. The database contains more than 700 million records and takes up 45 gigabytes. Our biggest challenge is updating every account, every day. With more than 1.5 million unique user accounts that need to be updated every 24 hours we would need to get the information for 62500 accounts every hour to get everything updated every hour. Previously we used to query the Twitter API more than 300 thousand times an hour. That worked fine for us until Twitter decided to limit the number of requests per hour to 20.000.

If we would do 20.000 request per hour we could only update 480.000 of our accounts. Less than a third of what we needed! We spent a lot of time analyzing the API and have come up with a way to update more than 200.000 accounts, per hour, without exceeding our API limits. That means we can track up to 5 million unique Twitter users with our current technology.

Arjen Schat, the co-founder and lead developer at TwitterCounter, came up with some pretty revolutionary technology, tricks and solutions which he will share with the audience at the upcoming Kings of Code conference in Amsterdam in a few months.

Since mid february we are generating more than 3 million TwitterCounter buttons every 24 hours and we can only see that number growing. With TwitterRemote now live and more and more bloggers joining Twitter we expect to generate close to 15 million buttons per day before the end of the year.

Last year Twitter reportedly grew 800%. TwitterCounter has been live for a little more than 6 months now. We enjoy working on it immensely and plan to keep innovating and adding interesting stats to it on a daily basis. Right now the service is generating some modest revenue, enough to pay for hosting and some office space. We expect to add more revenue generating services soon which I will talk about in a follow-up post soon.