TestComplete PPC Ads TC10 - CS6-02

Your Software Flunked Testing. Now What?

software-test-fail

The good news is your shiny new software is back from testing. The bad news is that it flunked. Maybe the QA department determined that the software is too slow, maybe it uses too many resources, maybe it just doesn’t work right.

 What do you do now?

There are a lot of things you can do when your custom application fails the QA testing process. What you do depends on the exact circumstances, including available resources, criticality of deadline, and the importance of the software to the enterprise.

None of your choices are likely to be optimal. You and the company are going to lose some skin in this game, and you have to balance the competing factors to minimize the pain.

Taking your time to fix the problems, even if it delays the release, is the choice that many developers would make – while muttering to themselves, “You can have it fast, cheap, or with all the quality and features you want; pick two.” If the tests demonstrate that the quality is out of whack, in most cases that’ll affect the “fast” or “cheap” goals.

At one end of the spectrum: You can do nothing. Put some lipstick on the pig and try to sell the porker before anyone notices. Then take the hit. Generally, this is held to be the worst strategy, although it is likely to have its advocates – cynically: especially in the executive suite.

“It is a huge mistake to just throw the product out there and ‘work out the bugs’ when it is already live,” says Aaron Watkins, president of Appency, an app marketing company.

This is an especially bad strategy in the highly competitive mobile app market, where most of the sales come through the app stores and their associated communities.

“Your negative reviews will stick around long term and even with major product improvements you will still have that bad smell of buggy software associated with you,” Watkins says, and an app’s history affects its ability to be found in app store searches.

Although some market segments are more tolerant of buggy software than others (insert obligatory Microsoft joke here) it is still a bad idea to ship the software without fixing the bugs. This seems obvious upon relaxed contemplation; it is a lot less obvious when the pressure’s on and Marketing just wants to get something – anything – out the door.

Christopher Swenson, a backend software engineer at Simple.com who says he “lives and breathes testing,” suggests, “Unless there is a terrible deadline where you will be losing a lot of money if you don’t ship it now, you have an obligation to your users to ship great software. It should be fast and lean, and as bug-free as possible. Take your time and do it right.”

A related “solution” that is sometimes possible is to strip out the buggy parts and ship the scaled-down software, followed in short order by Version 2 with the original features fully implemented.This is not a good idea, but it may be the best idea available to balance cost, quality, and time.

The other possibility is to go back and do it right. Figure out the root cause of the problem and fix that.

“Go back through and start benchmarking to find the slow parts and the memory hogs,” says Swenson. “Find out which of your external libraries are resource intensive, and work on fixing them, replacing them, or rewriting them.”

How much of a chore this is depends largely on where the project ran off the rails. In some cases, the problem turns out to be quite easy to fix, sometimes almost laughably so.

In other cases the problem is deeper. Perhaps the original specs were stated unclearly or there was a fundamental misunderstanding between the developer and the customer (internal or external). In these cases, it can take major surgery to fix the problem – which means more time and money.

At this point the question of how much work you want to do – or can do –raises its head. Often you can patch semi-successfully by exchanging modules, adding more buffer space to the database, or other methods. Alone, they may not fix the problem but they can get the performance up to make the product usable. One danger to this “Let’s patch it for now” technique is that, far too often, nobody ever goes back to fix the underlying problem. If the team makes the mistake of calling this not-quite-ready launch an Agile iteration, it can cause users to hate the development staff – as well as Agile!

Finally, there is the nuclear option: Write off the project and abandon the whole thing.

This is everybody’s least favorite “solution,” but it does put an end to the misery. Sometimes given the constraints of time and money on the project, it is the best solution in a bleak landscape. No one wants to pull the plug since it is a black mark on everyone’s record and it can be embarrassing if the word gets out.

Still, sometimes it is the only real solution.

See also:

  • disqus_YMVJ9Xy1x0

    In any decent Agile organisation, it would be impossible to pass an untested “complete” piece of software to QA for testing. Software is only complete when it has passed testing and it should be continually tested while development is taking place on a per-story basis.