BLOGCTA_AQT_20130726_1_Coverage_Profiling_936x110

Breaking it Down with Scott Barber: How Ellen Degeneres Broke Twitter

Welcome to Breaking It  Down with Scott Barber. Each month we will  recap a software quality issue that broke our apps and made the news. Performance expert Scott Barber dissects the incident and provides some important insights on how you can prevent the same issue from occurring on your servers.

This week, Scott will break down exactly how Ellen Degeneres was able to break Twitter with her record-breaking Oscars selfie.


SmartBear SE Tips:

1. Know your workflows and understand your expectations

In this video, Scott highlights the average number of tweets that Twitter expects to handle. Back on February 26, they increased from an average capacity of 6,000 tweets per second to 24,000.

These are important business metrics to understand, because they directly related to why Twitter was not ready to handle this event. You might think that a 4x improvement in capacity would help to ensure less downtime, but as we all know from the event, this didn’t help in the case of mass tweeting pictures.

In your own tests, it is important to record all of your key business workflows, then test them in combination with each other, rather than individually.

2. Load test boundary conditions, not just your typical workflows

Twitter’s typical ratio of text-only tweets to picture tweets was crushed by the quick flood of selfie tweets during the Oscars. While the total number of tweets per second around the world didn’t really exceed planned capacities, the total throughput required to distribute all that picture data in such a short period of time increased beyond what Twitter’s servers could handle.

Remember, all our “normal” tweeting was going on everywhere too, and with this spike in required throughput, the infrastructure tested to handle “normal” maximum capacity simply couldn’t keep up. Also, Ellen didn’t tell the folks at Twitter about her plan ahead of time, so they probably didn’t feel the need to include the “selfie bomb” in their capacity planning and performance testing.

You are not off the hook though! Your website or apps often have many separate workflows and situations that incur varying patterns of traffic against your infrastructure. Testing using multiple activities in combination with each other is vital to getting accurate load testing results.

LoadUIWeb can help you simulate real-world traffic patterns by using multiple VU groups to design combinations of different scenarios into a single test.

3. Number of virtual users vs. diverse traffic patterns

As we saw from the Twitter downtime event, it is not enough to simply multiply your expected number of visitors by arbitrary amount to ensure success. You must also diversify your performance tests with:

  • multiple key traffic patterns in realistic and varying amounts
  • multiple client types (browsers and connection speeds)
  • test data that represents varying inputs to your site or app

Sit down with someone from your Web operations or analytics team to review and prioritize your key business workflows, then record them in LoadUIWeb. Once you do that, you can create tests with combinations of key traffic patterns and change their browser and connection speed settings. Finally, you can use the data-driven testing features of LoadUIWeb Pro to simulate input that is more similar to how real visitors would use your site or web application.

4. Use combined types of tests, not just bigger tests

Going back to the problem Twitter had, it is also vital to simulate varying patterns in the amount of traffic your server handles at certain points in your test. A steady load test may show things like memory leaks and help to prove server capacity, but a spike tests can show how your site or app handles fast and furious activity as well as how it responds after the flurry of traffic is over.

To run mixed load profile types, use the LoadUIWeb Pro custom load profile shape editor to draw things like the “soak-n-spike” test below:

mixed-load-profile-types

See also:

 

subscribe-1

  • matthewheusser

    Amazing. You’d think the RT’s would contain a pointer to a binary, so the tweeting itself wouldn’t cause the problem. Maybe it was the /downloading/ of all the RT’s on all the clients? Re-listening, it sounds like that might have been the problem; Scott didn’t really specify.

  • http://pretenseofknowledge.com/ speedmaster

    Fantastic post and explanation. Thanks