Ingrained Biases in Software Testing That Work Against You

Ingrained Biases in Software Testing That Work Against You

Software testing is loaded with unchallenged ideas from the last 20 or so years, and changing its perception works against ingrained bias and prejudice.”

I recently came across this line in a blog post by Keith Klain and, as someone who loves to challenge ingrained biases and prejudices anywhere I can, it definitely caught my attention. The blog goes on to illustrate some interesting new ways of testing software, highlighting one company in particular that’s really getting it right.

But, by the end of the blog post, I was only more curious about that one line and the journalist in me started getting antsy… so I arranged an interview with the author.

Before I share the entire conversation, I wanted to address the original point that pulled me in… Just in case it piqued your curiosity, too.

What are the Major Biases Holding Back the Software Testing Industry?

According to Klain, the whole industry is underpinned by a false analogy that’s made between software testing and manufacturing. It may be false, but it’s not hard to see where it came from. Software is a product people buy, so it makes sense that it’d be treated like every other thing on the assembly line. To be fair, it may have worked perfectly fine when software was limited to being embedded in devices or shipped on CDs. But, like everything else, the Internet and wireless revolution pulled the strategy rug out from under software testing. So, if you’re actually testing Web based software today, that analogy—and the bias and protocol that result from it—is probably a major source of frustration for you.

This Is Not a New Conversation

Before we get into the details about why the bias towards seeing software testing as a manufacturing process doesn’t fly and what we can do about it, I’d like to step back a bit and acknowledge that this isn’t a new discussion. I found this blog from a few years ago calling for “manufacturing no more,” with the astute suggestion of replacing the manufacturing analogy with an attending physician for web applications:

“We just don’t write or release software the way we used to. Software isn’t so much built as it is grown. Software isn’t shipped … it’s simply made available by, often literally, the flip of a switch. This is not your father’s software. 21st century development is a seamless path from innovation to release where every phase of development, including release, is happening all the time. Users are on the inside of the firewall in that respect and feedback is constant. If a product isn’t compelling we find out much earlier and it dies in the data center. I fancy these dead products serve to enrich the data center, a digital circle of life where new products are built on the bones of the ones that didn’t make it.

In our father’s software and Deming’s model we talk about quality control and quality assurance while we play the role of inspector. In contrast, my job seems much more like that of an attending physician.”

I really like the physician analogy, even though it may be a bit of a stretch. For now, it’s a healthy one. We need to see test cases more as living organisms that must respond to changes and adapt to input. We need to keep records and track data in relationship to a dynamic system that lives in a habitat so to speak, not an unchanging object. With that in mind, let’s dive into the issue.

Why the Analogy Between Software Testing and Manufacturing Doesn’t Work

Many people who are involved in the software industry but don’t test software think you should (very easily) be able to break a test system down into predictive components. Often they are the ones looking to either make or save money by counting and quantifying all the steps it would take to sufficiently test software. So, it necessarily needs to be predictable—you can’t determine how much you could save or make if you can’t predict it. Nothing wrong with wanting to make and save money, of course.  It’s the insistence that predicting test cases makes for adequate test cases that ultimately undermines quality efforts.

This is essentially trying to create a commodity around a test case so that companies can sell testing services by promising to break down a system into hundreds or even thousands of test cases and charge for each one. Or, for a QA manager, this strategy might be understandably motivated by the need to plan a budget. But that approach robs a tester of much needed flexibility, the freedom to adapt to previous test results and their stakeholders, to change things as they get to know the software, and ultimately to produce intelligent tests that work well at identifying defects.

The other major influence supporting this old paradigm is the trend of shipping software to be tested overseas or off site. The need to control from afar and give directive guidance has deepened the manufacturing correlation rut tremendously. Again, it’s understandable. We humans usually equate control with security. Unfortunately, it’s not true in this case.

The Enormous Cost of Software Defects

A recent Cambridge University study cited that the U.S. economy looses $312 billion per year from software defects. That’s a huge number, but there are about as many ways to calculate those costs as the number itself. It’s like trying to count all the impoverished children in the world—it depends so much on how and where you look and often by who is doing the looking. So, I personally prefer to track individual stories, and particularly appreciate this slide show Bloomberg put out a few weeks ago on the most damaging software defect catastrophes over the past 50 years. The point is, either way you look at it, we have a lot of room for improvement.

What’s Being Done About It?

We all have biases, be they from language, education, culture, nation, gender, age or anything else that makes us human. In some ways, we can never be completely free from bias, but we can and need to be aware of its influence – especially when it becomes an obvious roadblock to our development. Luckily, the bias towards a manufacturing model in testing has already proven to be something we can change—with a little education, patience and persistence.

Keith Klain and others like him are taking a few different approaches to shifting this tide—one from a leadership or managerial perspective that involves a major overhaul of entire testing teams, and the other through educating the individual tester.

Only a few years ago, Klain took over Barclays’s Global Test Center. While he found that the team was extremely proficient in test management, “their misaligned priorities had the effect of continually hitting the bull’s eye on the wrong target.” Keith put a system in place “to foster testing talent and to drive out fear—abandoning worthless metrics and maturity programs, overhauling the training regime, and investing in a culture that rewards teamwork and innovation.” To hear more on how Klain implemented these radical changes, check out this YouTube video.

Obviously, the kind of overhaul Klain directed at Barclays takes a highly motivated, dedicated and persistent leader—let’s be frank, that’s a rare person. However, he and other influencers in the quality industry are working another angle as well. And this one not only is dedicated to revolutionizing software testing, they are out to put a dent in poverty as well. Through software testing training programs targeting under-served youth and adults in several major U.S. cities, Per Scholas, is doing both. Per Scholas trains testers to be contextual thinkers from the beginning and to bring the “manufacturing no more” values to their future work places. In the process, they are seeding a new trend that has the potential to shift old biases and pull the entire industry out of a pretty deep rut.

Let us hear your thoughts, biases, experience, frustration, etc. about your own work as a tester or test manager in the comment section. And stay tuned for the full video interview with Keith Klain, as well as an exciting new video series SmartBear is producing with him (think leaders in the software testing industry… debating).

See also:


  1. Thnk you a bunch ffor sharing this with all folks you actually recognise what you are
    speaking about! Bookmarked. Please additionally seek advice from my website =).

    We could have a link exchange arrangement among us

  2. I started my present job about 3 years ago with the mind-set that I had had for the previous 12 years — namely, the manufacturing approach that you describe. However, one of the things my boss encouraged me to do early on was to read blogs and such, something I was hesitant to do because there seemed to be so much crap out there that it wasn’t worth the effort. Well I am grateful to my boss for encouraging me to do this because I discovered some gold amidst the crap, Context-Driven Testing (CDT), championed by Keith Klain and his associates in that school of software testing. As happy as I was to make this discovery, I suddenly found myself on the outside. Here I had interviewed, standing on the proud achievement of having documented and maintained 1000’s of test cases in the course of my career and yet now I was saying, “Nope, I’m not doing that anymore. I’ve figured out that it is a total waste of time to write down test cases, time that I could be using for (surprise, surprise) actually testing.” The latest development is that there is now a push (not from me or any other tester, mind you) to “improve test documentation” in the group. I as yet have not been privy to what is actually meant by this but can only speculate that it is something along the lines of, “write test cases covering 100% of the software so that we can hand them off to people unskilled in testing as checklists for them to go through as they execute periodic regression testing.” I am at a total loss as to what I can do, if anything. Perhaps I will be relegated to stand back and watch the carnage as others go down this dismal path.

Speak Your Mind