Best Practices of Context-Driven Testing


First of all, before anyone’s head explodes, let me explain what I mean by “Best Practices.” I know that, particularly in the realm of context-driven testing, this term is looked upon as a major misnomer that is only spewed by the ignorant and uneducated. But I don’t think that has to be the case.

Not everything should be quantitative. Not everything has to be proven without a shadow of a doubt. Most of all, we shouldn’t be afraid of being wrong. For years people have subscribed to this idea that “there are no best practices,” and have genuinely feared the shunning that would come with stepping outside the lines of that very narrow, absolutist mindset. Particularly in an industry so tightly tied to quality and context, it seems bizarre that no one is willing to allow any gray area when it comes to a slightly subjective term like “best practices.”

For the sake of this blog post, let me briefly explain what I mean by “best practices.” I’ll do so by paraphrasing Cyrus Shepard, who I think has a good explanation for how we should look at this term, and what we should expect from the guidelines that fall under it:

  • Best practices are a set of rules or guidelines that have consistently shown superior results for a practitioner. This doesn’t mean that best practices are the only way you could or should accomplish a task, just that they have generally shown consistently higher results than other techniques.
  • Best practices can help to serve as benchmarks for the industry they’re applied to. They should be seen as goals for practitioners to strive for in order to raise the standard of work and enable an industry to mature over time.
  • Best practices are temporary. Saying that something is a best practice today doesn’t mean that I’m forever bound to that methodology. Rather, best practices should be expected to evolve over time. What is a best practice today likely won’t be a best practice a decade from now. And that’s okay. That means practitioners have improved with time and have created a new standard for the expected quality of their work.

And finally, best practices can, and sometimes should be, be ignored. As is the fundamental element of context-driven testing, you have to decide whether or not they fit into each project you work on. Sometimes you don’t have the time or resources to do all of them. But just because they can’t be blindly applied to every situation doesn’t mean they don’t exist. They do. And here is my take on five of the best practices of context-driven testing.

Ask Questions

This has to be the one thing that I hear context-driven testers emphasize more than anything else. Ask questions of stakeholders. Ask questions of the development team. Ask questions of your fellow testers. Without asking a slew of questions it’s extremely difficult to understand the context of a project, which in turn makes it very difficult to obtain maximum test coverage.

Asking questions can also be a major driver for improving your career beyond a specific project. Contantly asking questions and questioning the status quo allows junior testers to learn from their mentors, and it allows mentors to learn from junior testers. As Keith Klain explained to me during an interview in March, the most successful testers are generally the ones who have an unquenchable curiosity and are able to spread the knowledge they’ve gained over the years to those around them:

The people I’ve seen successful in the [tester mentoring] role are folks who are conduits to knowledge. So it’s like, “Let’s learn from my experience, and ask a lot of questions.” I think the Socratic approach to mentoring is really important to get people to learn things on their own and help them be very self-reflective and figure out what you are contributing to this problem and how you can help them tease out their own solutions. Because that’s ultimately what you’re trying to do anyway.

Plan Ahead

What good is all that knowledge if you don’t have an efficient way to put it to use? Creating and sharing your test plan with the rest of your team and project stakeholders not only makes you more efficient, it builds rapport with the rest of the company and spurns more meaningful conversations.

No, you shouldn’t be expected to take feedback from every junior developer or project manager in the organization, but sharing your initial test plans with the most prominent stakeholders gives them real insight into the type of return they should expect to see. It shows them that their are some guidelines to what your team is doing, and that any changes in their own plan will impact the testing strategy.

As JeanAnn Harrison explained it during her most recent visit to the SmartBear office, “If you plan out your testing strategy and figure out what kinds of tests you want to do then, really, you will go ahead and create a very efficient process.”

Adjust Your Plan Accordingly

Just because you’ve made a plan does not mean it ‘s set in stone. Rarely does a software project go entirely as expected, so you have to be ready to make adjustments as they come. Failing to be at least somewhat flexible with your test strategy will likely lead to all around frustration and less thorough test coverage. Schedules change, features are added, new priorities arise, and your strategy should adapt accordingly.

That said, if you’ve laid out your testing plan and shared it with stakeholders, you shouldn’t be expected to bend over backward to make up for mistakes in the earlier stages of the project. Remember that your ultimate goal is to achieve the maximum test coverage you can achieve given the parameters at hand. If those parameters change through the course of the project, you will doing yourself and your organization a disservice by staying true to a plan that no longer serves that ultimate goal.

Stakeholders Decide when a Project is Over

This is for everyone’s sake. It gives the stakeholders the power, and thus the responsibility, for deciding what timeline everyone on the project is working toward. That’s, in theory, part of their job. It’s also good for the testers who, as Dawn Haynes explains in the clip above, shouldn’t be making the decision to hit or miss a deadline.

Your job is to test the software as thoroughly as you can within the limitations handed down by the stakeholders, and then to provide as much information back to them as you can.

Don’t Blindly Apply Any Practice

This is probably the most prominent and obvious pillar on which the context-driven ideology is based. As I stated in the introduction, best practices (including the ones listed in this article) will not work for every situation. Sometimes you simply won’t gain much information from asking questions of project managers. Sometimes the project is in such dire straits that there really is no time to create a detailed test plan ahead of time. And sometimes it’s better to stick with your guns and push back against stakeholders that are making  a horribly detrimental decision. In the end, it’s all about doing as much as you can with the information you have at any given time.

And it’s exactly that kind of flexibility that will continue continue to elevate context-driven testers to the forefront of their field for decades to come.

See also:


  1. Okay, first the definition:

    “This doesn’t mean that best practices are the only way you could or should accomplish a task” – Then they’re not the best.

    “Best practices can help to serve as benchmarks for the industry they’re applied to. They should be seen as goals for practitioners to strive for in order to raise the standard of work and enable an industry to mature over time.” – How can a benchmark for one company apply to another one in the same industry but with different methodologies and practices and technologies and tools and culture? Moreover what about the individual disagreement about what constitutes even a good practice, let alone a “best” one?

    “Best practices are temporary.” – Then in what sense are they the best? And how temporary are they?

    “Saying that something is a best practice today doesn’t mean that I’m forever bound to that methodology. Rather, best practices should be expected to evolve over time. What is a best practice today likely won’t be a best practice a decade from now.” – Not just a decade, perhaps in a week or a day or 5 seconds time the context might change to devalue a best practice into a poorer one, which means that it cannot be the best, it can just be good for the current context.

    Now some examples of ways in which these best practices might be not best practices:

    “Ask Questions” (I’d also include asking questions of ourselves, and asking questions of the product and project).
    Fails when:
    * The answers are wrong
    * The question is offensive, pointless or illegal
    * The person one chooses to ask isn’t the person for that question
    * There’s nobody else to ask
    * We don’t use the answers to good affect

    It’s not important to ask questions, it’s important to ask good questions of the right people at the right time in the right way and to do something good with the information we get back.

    “Plan Ahead / Share Your Test Plan”
    Fails when:
    * It might be that your test clients will disagree with your test plans for political reasons or misunderstanding of testing.
    * It might not be possible to share your test plan, or your test plan may not be read by anyone
    * There may not be time to share your test plan – you may not be able to communicate it, or it might be that you have to document it in order to share it which takes more time that it has value
    * Over-planning is risky. Testing is an exploratory activity, so what is revealed by testing may change, add to or completely invalidate the test plan.

    It’s not important to plan, it’s important to plan the right amount at the right time in the right way for the given context.

    It’s not important to share the plan, it’s important to give to (and get from) the right people the right information at the right time in the right way about the kind of testing you intend to do, if appropriate and within the context.

    “Adjust Your Plan”
    Fails when:
    * If there’s a contractual agreement to your plan it may not be possible to change
    * If there’s a legal constraint on the plan it might not be possible to change it in the way you want to
    * Adjustments to the plan might exceed constraints of cost or time
    * Adjustments to the plan might be wrong, or the plan might be correct and be changed mistakenly

    It’s not important to adjust the plan, it’s important to make good decisions about testing coverage, oracles and techniques given what we know at the time about the product and the project within the context and communicate this well to the right people at the right time if and when it’s necessary and useful.

    “Stakeholders Decide When The Project Is Over (Testers Do Not)”
    Fails when:
    * The testers ARE the stakeholders

    “Don’t Blindly Apply Any Practice”
    This is why there are no best practices, only good practices relevant to a context. The value of any practice is dependent on its context and cannot stand alone. Having admitted that there are no best practices here I don’t see what use the term has.

    And how could we ever know that these practices are the best? We cannot compare them to all possible practices.

    I don’t see why we need the term “Best Practice”. It’s incorrect and misleading and I don’t know why it’s important to preserve it. Instead of redefining “best practice” to maintain a cognitive crutch why not lose the term in favour of having to think about the value of our practices in context? What does “best practice” do for us that “good practice for this context” does not do equally well or better?

    • Ben Austin says:

      Your comment actually hits on the exact reason I wrote this article. You are so caught up on, and biased against, those first two words of the headline that you can’t see the value in the rest of the article. Because of those words, you spent an undisclosed amount of time deconstructing everything in the article, rather than simply building off of it and adding to the conversation in a constructive manner. You found a way to show scenarios where each of my suggestions wouldn’t work, but failed to mention that each one is extremely beneficial the majority of the time – that’s a clear bias.

      This deconstructive nature is good for individual testers, but it’s holding back the industry as a whole. Yes, context is extremely important, as it is in most other professions, but it’s not the only thing that matters. It’s not helpful to boil down every conversation about testing to, “it depends on the context.” Imagine other professions sticking to that mentality… You never hear about context-driven surgery or context-driven bus driving because those professionals have evolved beyond that benign conversation. Context is still incredibly important to those professions, but at some point someone said, “alright, but what are some actual practices that I can do to be a better surgeon?” If someone had just said “it’s all about context” every time they asked they likely wouldn’t have learned much about how to perform surgery.

      In the end, it all comes down to a bias against a silly little term, which stems from an industry-wide trend to deconstruct. “Good practice in this context,” is just as subjective as “better practice,” which is just as subjective as “best practice.” If I had used the term “good practice in this context” in this article, you still could have come in and deconstructed each section in the exact same way you did above and neither of us could ever prove 100% that we were right. But that’s not helpful, and that’s not the conversation we should be having.

      We should always be answering the question, “how do I do better testing?” If our own biases against those two words continues to hold us back from agreeing on anything more than, “depends on the context,” then this profession will be very slow to evolve.

      • “Your comment actually hits on the exact reason I wrote this article.”
        You wrote an article about best practices. There are no best practices, and I argued against your redefinition and your cited examples. Am I caught up in the term, or do I simply disagree with it? You claim that there’s value in keeping the term. I fail to see what value the term is adding, I disagreed with your reasoning for keeping the term, I went on to explain why the term doesn’t work using counterexamples to the examples you provided, and I queried why we need the term or why we need to defend it. I wrote to disprove your argument by reasoned counterexample. Did I not make a series of arguments against the use of “best practice”? Why are these arguments dismissed as “biased”? I could claim that you’re too caught up in keeping the term, and biased FOR it, but it wouldn’t get us anywhere.

        Can you please also explain why “bias” devalues in any way the arguments that I’ve made? You haven’t dealt with the points I made about why “best practice” is a bad term, you’ve simply called me biased. I’m reminded of psychics countering scientific arguments against them with the claim that the people making them are “closed-minded”. I think we should debate with the value of our arguments, no matter how biased they might be. I’d welcome good reasons as to why “best practice” is more useful than harmful.

        The term “best practice” is actively harmful to a serious craft. I think that’s an important thing to say. I’m not saying that your “best practices” don’t have value, I’m saying that we shouldn’t call them “best practices”. I think it’s important to change a plan when necessary, but there are occasions that make it a bad idea. Understanding that empowers me to make more considered and informed decisions. If I established it as a “best practice” I’d be blinded to those pitfalls.

        “but failed to mention that each one is extremely beneficial the majority of the time”
        So… not the *best* practice, then? You’re making my point for me here. I’m showing how the term “best practice” does not make sense, and how your claims of best practices are, in fact, not best practices. Yes, they’re good most of the time, but you didn’t claim that they were good practices, you claimed that they were best practices.

        I like that you call it a “silly little term” – because that’s also what I think it is. Of course there is no practice that is better than any other possible practice! That’s why I think it’s time to move on and change our conversation from talking about “best practices” to talking about practices in general, and their worth for the context in which they’re used. It also raises our conciousness about the influence of context on our practices and teaches us to think rather than to emulate. It teaches us not to be absolutist in our thinking and admit the possibility of heuristic failure.

        “Best” is the superlative of (and “better” is the comparative of) “good”. They all mean completely different things. The word “best” infers that there is no better of something. The comparative and original adjective are not absolute descriptors in this sense as they’re used to ascribe a comparison or a value judgement. That’s why subjectivity is important, and why “best practice” doesn’t make sense. Subjectivity, to me, is an argument against “best practice”, rather than for it.

        If you had changed “best practice” to “good practice in context” and I came along and deconstructed it, showing when each one would fail I would no longer be arguing against the term “best practice”. In fact, I would be supporting your article with ways in which context can influence the value of a practice. “Yes, I agree, there are only good practices in context, here are some practices and the ways in which context changes their value in order to help to make that point”, I would say.

        I’m not sure what “100%” right means, here. I am claiming that the term “best practice” is nonsensical and harmful, supporting it with my reasons. I think that’s an important conversation to be having. Not being able to prove something doesn’t mean that the discussion has no value, or (given that you say you could never prove yourself 100% right) you presumably you wouldn’t have written the post in the first place.

        “How do I do better testing?” – by saying what I mean, and thinking about the value of what I’m doing. I don’t understand how disagreeing with the use of the term “best practice” invalidates any agreement about any other thing. You’re talking about the “context-driven school” in your article – a school driven by context. Why would we NOT want to talk about context if it drives us? What context-free absolutes are you looking for? We navigate the complexity of the world with heuristic (fallible) thinking. Understanding our own fallibility is a vital component to good, thinking testing. I don’t think the nature of our minds and the limits of our language have slowed our progress in the testing profession any more than any other. We must make assumptions every second of every day to survive with a brain that cannot perceive the world as it is and it does us very little harm. However when we see something in our thinking or discourse that is harmful then I think we should call that out and discuss it, as I have done here. I think it would be good to put my perceived bias to one side and discuss the dangers and merits of this term with reason and evidence.

        I would like to ask again why we must be resolute in defending “best practice” as a term. Why do you think we need to save it? What is it doing for us? And why redefine “best practice” when we can just say what we actually mean?

        • I think you should read Ben’s reply again. You’re quite biased in the way you write and that’s holding you to see what the real “context” of this post is referring to.

  2. Jeffrey Martin says:

    Semantics can be important. Language is in many ways the context of our thoughts, and the brain is VERY context driven. “Best Practices” is a misnomer and the phrase implies certain concepts such as completeness, suitability, and general fitness for the use at hand. The article says that “Best Practices” are temporary, can be ignored, etc.. all of those points make a great case to retire the phrase “Best Practices” completely, they simply are not “Best.”

    What smart practitioners of any field generally actually use them for is as a starting point, especially valuable for people new to a given practice, and then modify and adapt the practice as needed to their context.

    Calling this the “Best” implies, even if only subconsciously, that it cannot or should not be improved. They are better, perhaps, named “Starting Practices” or even “Operating Practice.” And they do serve some purpose, as it’s always smart to learn from other peoples successes. While what is successful for you might not work for me, it’s not a bad place to begin, especially as a rookie; as long as I don’t think it’s an immutable “Best.”

  3. Gregory Mooney says:

    I’d like to add that I don’t think this article should be taken too literally. I cannot speak for Ben, but from what I gather, this article was created for the sole purpose of sparking a discussion that’s been going on for quite a long while. Unfortunately, these discussions tend to turn into heated debate and sometimes get overly political.

    Personally, I’ve never used the term “best”, but whether you use it or not isn’t the point. There are certain principles that are highly valuable to newcomers of the software testing space, but context needs to be explained. Blindly following suit because someone “more experienced” says it is always the best route of execution; that is being a puppet and not using critical thinking which is the most important aspect of testing.

  4. Both Ben and Chris are right at their own Context 😉

    Nice article Ben ! We get your point.

    Nice debate Chris ! We understand what you are referring to.

Speak Your Mind