This is the fourth installment of blogs regarding the Top 5 Agile Testing Challenges.
You can view the prior blogs or download a more detailed white paper, here:
4. Five Challenges for Agile Testing Teams: Solutions to Improve Agile Testing Results (white paper)
Agile development is a faster, more efficient and cost-effective method of
delivering high-quality software. However, agile presents testing challenges
beyond those of waterfall development. That’s because agile requirements are
more lightweight, and agile builds happen more frequently to sustain rapid
sprints. Agile testing requires a flexible and streamlined approach that
complements the speed of agile.
Inadequate Testing for your Published API
Many testers focus on testing the user interface and miss the opportunity to
perform API testing. If your software has a published API, your testing team
needs a solid strategy for testing it. API testing often is omitted because of
the misperception that it takes programming skills to call the properties and
methods of your API. While programming skill can be helpful for both automated
and API testers, it’s not essential if you have tools that allow you to perform
testing without programming.
Getting Started with API Testing
Similar to automated testing, the best way to get started with API or Web Services testing is
to take baby steps. Don’t try to create tests for every API function. Focus on
the tests that provide the biggest bang for your buck. Here are some guidelines
to help you focus:
- Dedicated Resource: Don’t have your manual testers
develop API tests. Have your automation engineer double as an API tester;
the skill set is similar.
- High Use Functions: Create tests that cover the most
frequently called API functions. The best way to determine the most called
functions is to log the calls for each API function.
- Usability Tests: When developing API tests, be sure to
create negative tests that force the API function to spit out an error.
Because APIs are a black box to the end user, they often are difficult to
debug. Therefore, if a function is called improperly, it’s important that
the API returns a friendly and actionable message that explains what went
wrong and how to fix it.
- Security Tests: Build tests that attempt to call
functions without the proper security rights. Create tests that exercise the
security logic. It can be easy for developers to enforce security
constraints in the user interface but forget to enforce them in the API.
- Stopwatch-level Performance Tests: Time methods (entry
and exit points) to analyze which methods take longer to process than
Once you create a base set of API tests, schedule them to run automatically
on each build. Every day, identify any tests that failed to confirm that they’re
legitimate issues and not just an expected change you weren’t aware of. If a
test identifies a real issue, be happy that your efforts are paying off.
API testing can be done by writing code to exercise each function, but if you
want to save time and effort, use a tool. Remember, our mission is to get the
most out of testing efforts with the least amount of work.
When considering API testing tools, take a look at
SmartBear’s soapUI Pro. It’s easy to learn and has scheduling capabilities
so your API tests can run unattended, and you can view the results easily.
Which API Metrics Should You Watch?
Focus on API function coverage, API test run progress, defect
discovery, and defect fix rate. Here are some metrics to consider:
- Function Coverage: Identifies which functions API tests cover. Focus on the
functions that are called most often. This metric enables you to determine if
- Blocked Tests: Identify API tests that are blocked by defects or external issues
(for example, compatibility with the latest version of .NET).
- Coverage within Function: Most API functions contain several properties and
methods. This metric identifies which properties and methods your tests cover to
ensure that all functions are fully tested (or at least the ones used most
- Daily API Test Run Trending: This shows, day-by-day, how many API tests are run,
passed, and failed.
What Can You Do Each Day to Ensure Your API Testing Team Is Working Optimally?
Testing teams should perform these things every day:
- Review API Run Metrics: Review your key metrics. If the overnight API tests found defects, retest them manually to rule out a false positive. Log all real defects for resolution.
- Continue to build on your API Tests: Work on adding more API tests to your arsenal using the guidelines described above.
For tools to support these practices: