Twitter Capacity

Posted on 14/07/2010 by James Collings

It seems not a day goes by now on Twitter without a flurry of complaints that Twitter is “out of capacity”.  But what does that really mean, and is it actually true?

The overall Twitter service is a fundamentally simple system

  • Receive a new tweet from a user
  • Store that tweet
  • Distribute that tweet to each follower

 

So exactly which step are people complaining about?  Is it actually Twitter’s fault, or is it the client that they are using?  Client-Server applications that are served over the Internet have so many constituent parts that root-cause analysis to find the true problem is very hard. 

The Twitter architecture consists of a front end (the web or the API), multiple layers of cache, and a backend MySQL database.  Twitter engineers have stated that the database is just a backup, everything is kept in RAM.  This makes sense given the high volumes of traffic that Twitter is expected to handle at times (Obama’s inauguration, Michael Jackson’s death) and the relatively quick access times of memory compared to physical disk.

Receiving, storing and distributing are therefore all done in memory using a queuing system.  My suspicion is that, when people are complaining about “capacity” the real problem is that the queues are full.  However if this were true, then all users should be suffering equally; at least all API users or all WEB users (they use different routes through the architecture).  If it is only a subset of users that are suffering, then what else could be the problem?

If the users client is unable to contact the Twitter API (due to network capacity of their mobile/wireless/internet provider) then that too might be reported back to the user as “out of Capacity”.  How is the user to know whether the fault lies with Twitter or their network provider?

Either way, the most curious aspect of the issue is this:

When users receive the “Out of Capacity” message from Twitter, the one place that they immediately make this known to the world is…. Twitter!

Posted in Blog Posts, Capacity Management | Tagged , , , , , , , , , , , , , | Leave a comment

Using Freeware tools for Capacity Management

Posted on 26/06/2010 by James Collings

There are a large number of tools dedicated to Service Management of IT infrastructure.  Many of these lay claim to providing performance management or capacity planning to a business. 

But how difficult a job is this, and is it worth spending money on such tools?  Most operating systems contain bundled applications/tools that will report the current activity of a system.  The purpose of a performance or capacity tool is to record this information on a regular basis, store it, and provide analysis tools for the specialist to use.  There are limitations to the bundled applications/tools, and for this reason some vendors have written their own systems that can interrogate a running kernel directly.  This will certainly provide additional information over and above the standard toolsets, but it is unnecessary if all you need to look at is overall CPU, Memory and Disk activity.

Tool vendors also promote their ability to report and analyse application data.  Before you get excited about this, consider a very simple and fundamental fact.  If your application developers have not included any instrumentation in their application, then where will the vendor’s tool get it from?  They can’t just make it up!  Once again, the commercial tool is useful in so far as it will record the application instrumentation (assuming your developers DID put some in) on a regular basis and make it available for analysis.

Having read the above, you might think that I am against purchasing commercial tools for performance and capacity within a business.  You would be wrong.  While there are a number of free tools available, it is right to do a little “what if” analysis on their use.  As Capacity Planners, we are used to “what if” analysis.  What if the business grows by 20%?  What if this new product is highly successful?  What if we merge with a competitor? 

However, we should also consider the following.  What if our monitoring tool fails?  With a commercial product that you are paying maintenance for, while the damage will have already been done (you won’t have any monitored data for your systems), you will have some recourse to get the problem fixed.  With a freeware solution, you have no recourse to anyone.  There may be a network of helpful experts on the internet, but you have no guarantee that they will be there when you need them, or will have a solution for your problem.  If your problem is unique and requires changes made to the source, then you are reliant on there being someone that WANTS to make that change and their timescales for doing it.

An initial few hours of downtime could quickly become weeks or months.  With a commercial tool, this would not happen.

Freeware monitoring tools have their place.  For non-mission critical monitoring they provide the cheapest solution available, but where the Capacity and Performance service depends upon it, I say “commercial” every time.

Posted in Blog Posts, Capacity Management | Tagged , , , , , , , , | Leave a comment

What does the Coalition Government mean for Freelancers?

Posted on 20/05/2010 by James Collings

Well… For the first time in my (not so) short life, we have a coalition government in the UK.  On the face of it, this means that every decision made by the Government will be a compromise between the two parties.  However, in reality, all decisions made by Government are compromises… it is just that this time the compromises will be very public, whereas normally they are between two opinions held within a single party and the “deals” are “done” behind the scenes.

What does this mean for freelancers in the UK?

The outgoing Labour party have been quite openly hostile to freelancers.  They introduced IR35 which insultingly “deemed” freelancers as being nothing more than disguised employees of their clients.  This legislation has coloured relationships between the smallest of businesses and Government for over a decade.  The incoming Liberal and Conservative parties, however, have been very favourable towards freelancers.

The Liberals have spoken up in support of freelancers in the House and have led two Early Day Motions pushing for repeal of IR35.  They even went so far as to table a motion to get IR35 repealed in the recent Finance Bill (it didn’t get to the reading stage).  The repeal of IR35 has been a pledge of theirs for many years.

The Conservatives had also pledged to repeal IR35 since its introduction, however in recent years this pledge has become a little watered down to be a “review” rather than outright “repeal”.  I suspect that this difference between the Conservatives and the Liberals is in no small part linked to their relative expectations of being in power right now, and the danger of making promises that can’t be kept.

However.  They are both SHARING power.  For freelancers, this is good news.  The two parties have their long-standing opposition to IR35 in common, and only need to find a way to work together to get rid of it.  The fact that Labour stood so firm behind it is a sign of the emotional attachment that they had to the policy.  Too many people had too much “face” to lose for IR35 to be abolished during the previous administration.  This “embarrassment” factor will also give the new coalition an incentive to push for its removal.

And today, already, the decision has been made to “review IR35” and to “seek to replace it”.  The question remaining is “when?”.  The state of the country’s finances are not healthy, and any policy that REMOVES a tax (regardless of how pitiful the tax-take has been) will have to be publicly managed very carefully.  While the budget of 2011 would be great, realistically the budget of 2012 is more likely.

Posted in Blog Posts, Freelancing | Tagged , , , , , , , , , , , , | Leave a comment