The default set of statistics
loadUI comes with roughly 30 statistics per component and about 100 statistics per server monitor. This is unless you’re using the soapUI Runner, then there are eight more statistics per soapUI TestStep. Of course, if you run a distributed test, each agent holds its own set of statistics, and when comparing two results the total amount of statistics double. Overall, a typical load test gives you access to roughly 1,000 statistics.
It is not enough
No amount of default statistics will ever be enough. I know this after having many
conversations with users and customers; service level agreements are often not, and should not be, written based on which values a specific load testing application can spit out.
I recently got a question from a user who wanted to make sure that their web service handled 98% of the requests correctly within two seconds. I discussed this with my colleague, Dain, and we realized that this would be possible if the condition component had statistics – actually, that would let the user define any statistic on their own!
Define your own
I decided to add the following statistics to the condition component:
- True occurrences – The number of times that the condition has passed.
- False occurrences – The number of times that the condition has failed.
- True percentage – The percentage that passed.
This makes it able to meet the user’s requirements by creating the following scenario:
As you can see, I have configured the condition component to pass all requests that took a maximum of 2,000 milliseconds to complete. I also created an assertion that will fail if the true percentage is lower than 98%. Finally, I can set the entire load test to fail if any assertion failures occur, using project limits.
If you have used the condition component before, you might know that it has an advanced mode. This means that you can now get statistics for, and therefore assert, anything that you can express as a Groovy condition!