Exploratory testing has become a (well deserved) cornerstone in agile testing strategies. Although automated and manual tests are great for creating a safety net for every change made directly to your application, they don’t catch errors outside of the scope of what they are testing. Manual and automated tests aren’t good at what your users excel at: unexpected or erratic behavior; canceling when one shouldn’t, combining actions in a way that wasn’t in your user-story, or configuring your application in an unintended, but technically possible, way.
Exploratory testing is just that – “exploring” your application in imaginative and user-centric ways. It means performing actions that users do (or don’t do) before they have to call support because they found a bug that wasn’t found by your automated tests. Think of it this way: an exploratory test is like taking your new car out for a (spontaneous) drive into town. You park like you usually do, accelerate and break on your way, and make U-turns where you usually have to when dropping the kids off at school. Perhaps you will notice that the combination of these actions (all abiding by traffic laws, of course) expose minor but vital flaws in the car. Maybe the rear-view mirror doesn’t give you the angle you require, or the turning radius is far lower than what you expected. These may be flaws that isolated tests of the car’s functions in a controlled environment wouldn’t have found, but are made apparent when tested sequentially.
Load-testing is almost always performed as “un-exploratively” as technically possible. You set up a fixed load (perhaps with a ramp up and ramp down), run it over a certain period of time, then analyze the results, giving you an understanding of how your application performs under load in an extremely controlled environment.
This approach definitely has its merits but in most cases it’s not a very realistic simulation of how load will affect your application. In keeping with my car analogy, load testing is like driving at a fixed speed and direction for a certain amount of time and then analyzing the results (heat emission, fuel consumption, etc.). This might be good to test the basic traits of your car, but is not a practical simulation of the car’s actual performance. Wouldn’t it be better to put your car in the hands of the Stig and let him put it through its paces on a racetrack for a couple of hours? I’m talking frantic gear shifting, sporadic breaks, frequent accelerations, power-slides – the works!
For your web application or API interface that would translate to ramping up and down multiple transactions simultaneously, with varying number of virtual users and from different load agents. It’s simultaneous, realistic, and passionate. Does your real-time monitoring show you that your network is hardly being utilized? Increase the attachment-upload and file download transactions. Is your database server busy doing nothing? Increase the CRUD transactions that make heavy use of it – preferably several different ones at the same time.
Hopefully, you will notice that certain combinations of transactions at certain loads will trigger unexpected bottlenecks in your infrastructure. Perhaps the simultaneous load on the database from multiple transactions resulted in database locks that only occur under these conditions; perhaps you will notice that the bandwidth of your local network is too low to handle a high number of transactions that upload images and make calls to an external API for rendering maps. These are all vital findings for creating an operations runtime that meets the needs of your end users.
I’m not saying that automated “static” load tests shouldn’t be part of your testing strategy. On the contrary, these tests can be extremely valuable. For example, when it comes to detecting which changes to your system are having an impact on performance; are your nightly automated load-tests going 20% slower since you upgraded your AppServer? Did your refactoring of the ORM layer result in a 15% increase of speed for search-transactions? Exploratory Load-Testing probably wouldn’t have found that for you – it’s just another very important way of putting your system through its paces, and not a replacement for the tests you already have in place.
You knew I was going to talk about a SmartBear Product I am passionate about, didn’t you! Well, as it happens, our API Load Testing tool LoadUI was built exactly with exploratory load testing in mind. It lets you do all the actions required for exploratory load testing; add and remove transactions and virtual users, increase or decrease load, change load characteristics, monitor server performance and client response times – and all in real time while your test is running – allowing you “play around” with your applications performance in a realistic and spontaneous way. It even lets you distribute your transactions to load agents running in your network or in the cloud – while the test is running! Want to see how your network throughput handles load coming from an Amazon cloud node in Asia? Set up your loadUI agent accordingly and just drag-and-drop it to this node in the UI; once again – while the test is running and you can see the graphs for network utilization from your server react – no restarting required.
Of course loadUI allows you to still run the traditional “generate X transactions per Y from these Z agents for U minutes” test (and you should!) – But I firmly believe that both approaches are necessary to get a realistic “performance profile” of your application – be it Web, API, or in the middle (don’t ask).
So, should you leave your automated load tests in the dust and go full frontal on exploratory load testing? Of course not! Should you consider exploratory load testing if your system needs to handle multiple reasonably complex transactions at various loads simultaneously? Probably you should! Did I write this blog-post to get you interested in a SmartBear product? You bet! Have a look at loadUI which is uniquely positioned to fill this gap in the load-testing tools market – and best of all, it’s free and open source!
So should you try it out? Definitely!
Do you agree? Or is this the most ridiculous thing you’ve read since sliced bread? Please comment and get the discussion going!
Thanks for your time!