We talk a lot about the importance of quality products. But what does the word “quality” really mean? Is it a subjective or objective trait? And how can anyone strive to achieve high quality without first understanding what that term actually entails? Over the next few weeks, we invite you to join us as we explore the meaning of the word “quality,” and try to understand the role it plays in our everyday lives. In this post, Ken Godskind, Vice President of Monitoring Products at SmartBear, discusses the four metrics he believes define quality.
My co-workers have been writing a lot about what “quality” means to them for the last several weeks. This should be no big surprise, as we all work for a company whose tools and solutions help improve the quality of code, perform functional and regression testing, manage the testing process, perform load testing and monitor quality in terms of availability, performance and user experience – woo, that was a mouthful. Our tools and SaaS offerings help to improve not only software quality but the quality of user experience delivered by that software.
I have worked in the end-user segment of the monitoring industry for close to 15 years and have developed my own philosophy about what defines quality.
My philosophy, while obviously colored by my professional experience, has been greatly influenced by a term shared with me by a large telco service provider during a product overview I was delivering almost 13 years ago. They talked about “demonstrable measures of quality.” It is my belief that we can define demonstrable measures at each stage of the software development and deployment life cycle, but I’d like to wax on a little about what demonstrable measures of quality means from a business perspective.
Companies that have transformed their respective industries have done so by focusing largely on the customer experience or user experience. And those improvements and changes, whether delivered by people or websites or APIs, are supported by the cumulative work of development, QA and operations.
My career experiences have led me to define some demonstrable measures of quality from a business perspective. These measures are hard metrics that can be measured, trended and correlated to business performance.
1. Service Availability and Performance
From a user experience and Web performance perspective we need to address service availability from an end user perspective. During what time window do you expect end-users to be able to interact with the website, mobile application or Web services?
These days that’s usually pretty close to 24 x 7 x 365, with rare exceptions.
And we need to understand performance both in technical and perceived terms. Consistency of performance is also a meaningful metric. I’d rather be a tenth of a second slower on average if it meant a lot less variable and a more consistent experience for each user.
2. Visitor Analytics
Other measures also indicate you’re delivering great customer experiences. From Avinash Kaushik’s blog there are three key areas:
- Acquisition: How are you acquiring traffic via earned, owned and paid media, and how are you prioritizing that traffic and spending?
- Behavior: Are you getting the desired behavior supported by metrics like bounce rates, time on site, page views and conversions?
- Outcomes: Are you measuring what outcomes signify value delivered to the business bottom-line?
3. Business Outcomes
Business metrics measure whether or not the applications delivered the desired business outcomes.
For retailers that means orders and revenues from sales of product. For insurers that might be policies quoted and bound. For lead generation efforts that might be the number of individuals who viewed a webinar or downloaded a trial. For banks it might be the number of checks deposited through the mobile app versus coming into the branch, which has a much higher cost to service.
4. Impact on Support
Another direct measure of user experience and application quality, especially at release time, is support case load or online chat inquiries. Categorizing and understanding these user interactions can provide an immediate feedback into the success of new feature releases. Other indirect measures of quality for application releases, like support case load, also factor into the equation.
Taken together, all of the above are Demonstrable Measures of the Quality for an application. Of course these measures need to be reviewed in concert and in the context of the objectives for the business and project.
To me, the most meaningful measures of quality do not measure any individual aspect of developers, code, software, hardware and networks but rather how they function as an orchestra to deliver an experience to the users and how that impacts the performance of the business.