A small contingent of SmartBears are down in the heart of New York attending CAST 2014 – the annual meeting of The Association for Software Testing. A sold-out crowd of about 250 testers from around the world have piled into the Kimmel Center, right in the middle of Greenwich Village, to learn and share from their peers.
The first day of the conference consisted of several all-day tutorials with some of the most well-known names in the field. In the spirit of this year’s theme, “The Art and Science of Testing”, I decided to dive into the tutorial “The Art of Performance Analysis”, hosted by Mark Tomlinson. Some of you may remember Mark from our SmartBear on Air podcast, or as part of the PerfBytes team. As someone relatively new to the field, the prospect of participating in an interactive tutorial seemed a bit daunting at first, but it turned out to be an extremely comprehensive, interactive and rewarding session.
True to the title of the seminar, Mark wove together a series specific tactical elements and strategies and discussed how they fit into the larger art form of analysis, synthesis and reporting. The key takeaway was the massive human element required to effectively manage performance based on situation and context. This applies to such varied skills as philosophical as question formulation, to the ability of identifying and mitigating bottlenecks and the interpersonal and communicative skills required to effectively communicate results and manage crises.
In one of the more illustrative segments, Mark used his recent cross-country move to illustrate some of the key causes of resource bottlenecks. In this analogy, various bottlenecks can be represented in the following ways:
- Processor bottlenecks: having a limited number of movers/family members to help pack up your belongings
- Disk bottlenecks: lack of storage capacity, whether in the moving truck or in a storage unit
- Memory bottlenecks: only being able to carry a limited number of boxes in your hands at one time
- Network bottlenecks: running into an accident on the highway en route to the new house
However, this analogy breaks down a bit when applying to system performance, as it may not be readily apparent what the source of a bottleneck. Often enough, one performance issue will mask other ones downstream. This is where the human element is critical. According to Mark, there are four main steps to addressing these issues:
- Analyzing the problem to identify the root cause and its consequences
- Synthesizing reporting outcomes into meaningful data
- Communicating the information in a manner appropriate for the audience
- Escalating/remediating the issue based on severity
There is a large degree of precision required to carry out performance tests, but the variability around the design, execution and remediation elevates it into an art form.