It seems not a day goes by now on Twitter without a flurry of complaints that Twitter is “out of capacity”. But what does that really mean, and is it actually true?
The overall Twitter service is a fundamentally simple system
- Receive a new tweet from a user
- Store that tweet
- Distribute that tweet to each follower
So exactly which step are people complaining about? Is it actually Twitter’s fault, or is it the client that they are using? Client-Server applications that are served over the Internet have so many constituent parts that root-cause analysis to find the true problem is very hard.
The Twitter architecture consists of a front end (the web or the API), multiple layers of cache, and a backend MySQL database. Twitter engineers have stated that the database is just a backup, everything is kept in RAM. This makes sense given the high volumes of traffic that Twitter is expected to handle at times (Obama’s inauguration, Michael Jackson’s death) and the relatively quick access times of memory compared to physical disk.
Receiving, storing and distributing are therefore all done in memory using a queuing system. My suspicion is that, when people are complaining about “capacity” the real problem is that the queues are full. However if this were true, then all users should be suffering equally; at least all API users or all WEB users (they use different routes through the architecture). If it is only a subset of users that are suffering, then what else could be the problem?
If the users client is unable to contact the Twitter API (due to network capacity of their mobile/wireless/internet provider) then that too might be reported back to the user as “out of Capacity”. How is the user to know whether the fault lies with Twitter or their network provider?
Either way, the most curious aspect of the issue is this:
When users receive the “Out of Capacity” message from Twitter, the one place that they immediately make this known to the world is…. Twitter!