lifecycle-management-api

My Acceptance Testing Saga – How Would the Customer Test This?

acceptance testing saga

I guess I was pretty late in finding test-driven development, but when I did it was a major game changer. Finally I had an outlet and a method for focusing on quality while developing software. Unit testing became my weapon of choice. Sadly it became my only weapon. I thought I had found the silver bullet, and I tried very hard to convince others that this was so.

Even though TDD and unit testing can give you a decent design, it doesn’t really say much about the quality of your software. TDD ensures that we do stuff right, but it doesn’t tell us if we’re doing the right stuff. I think that quality lies not only in delivering stuff that works, but the right stuff that works. A customer will take for granted that the system works, the hard part is ensuring that what it does is actually what the customer wanted.

And that’s where acceptance tests come in. Acceptance tests work on the highest level possible, testing things that are the most important to the customer and to our product. These tests will tell us if the system is actually doing what we and the customer agreed on. Automating these tests enables us to evolve the system with confidence, without fear.

At the company where I work, we use SoapUI for some of our acceptance tests. We do a lot of integration projects and this tool is very useful when it comes to Web service based solutions – when the main deliverable actually is a Web service.  Even though SoapUI is a great tool for testing, creating good acceptance tests with it (or any other tool, for that matter) requires some skill. We need to understand what we’re testing first, then we can go ahead and implement it in our tool.

It’s easy to lose focus of what we are testing and as I’ll try to demonstrate in this post, the tool and your experience with it can sometimes affect your test design. I know I’ve fallen into that pit many times.

The ACME E-commerce integration

Let’s consider a simple integration example. Imagine we are developing an integration to our own e-commerce platform, ACME. Our customer will integrate with it to place orders for their Web shop. This is the basic design: 

Acme system design

We serve our customer’s app with two things: A Web service for placing orders and an inventory file for letting the customer know what products are available. The Web service is used online in the customer’s app, the inventory file is a one-per-night transfer.

We’ll start with the simplest happy test case that provides value for the customer: Placing an order for an item that exists in our inventory. An item with enough stock should be successfully ordered from our customer’s Web shop. Since I use the Gherkin language for my tests (see this blog post on how to implement it in SoapUI), this is what my test will look like:

GIVEN there are more than two items left of an item
WHEN I place an order for two of that item
THEN an order is created
AND the inventory of the item is reduced by two

That looks pretty clean and simple, doesn’t it? It’s just a matter of finding an item, placing an order, asserting the result is okay and that the inventory was updated. Easy-peasy. With the basic test design done, it’s time to implement the steps. Let’s ignore the order for a while and start with the easiest step, the WHEN step.

WHEN I place an order for two of that item

This will be our Web service call, the main reason for us selecting SoapUI for these acceptance tests. The contract at hand looks like this:

<PlaceOrderRequest>
…some authentication

<CustomerNumber>90210</CustomerNumber>
<OrderLines>
   <OrderLine>
     <OrderLineNumber>1</OrderLineNumber>
     <ItemNumber>1001</ItemNumber>
     <Quantity>2</Quantity>
   </OrderLine>
</OrderLines>

</PlaceOrderRequest>

In order to place an order, we need a couple of things:

  • A customer number.

    This could very well be hard-code since there will probably exist a set of “test customers” in the database. Customers in this case are what I would call static data, meaning data that rarely changes. Of course, there will be more customers – let’s hope! But a particular customer’s basic configuration will not change frequently.

  • An item number.

    This is trickier since it is transactional data, meaning that it will be affected by our transactions. We could hard-code an item number into the tests, but what happens when we run the suite 10 times? 100 times? I try not to hard-code this type of thing, so we’ll need to get it from somewhere else. That’s what the GIVEN step is for. We use that step to setup the context for our scenario which in this case means finding that item.

GIVEN there are more than two items left of an item

So where do we find that item? Since we’re using SoapUI, the first thing that comes to mind is a Web service request. Unfortunately such a service is not available in this solution. We need to think of something else. What other options does SoapUI give us for implementing test steps? Quite a few actually…

All SoapUI test steps   JDBC circled

Hey, there’s something we can use – the JDBC test step. We have the inventory information at our disposal in the SQL database, so why not get it from there? Said and done, we create a new JDBC test step and hack some SQL selecting an item with an inventory level larger or equal to two. Then we use the get data feature to copy the result of the SQL into our WHEN step. Smooth!

THEN an order is created

We have found an item and a request has been sent to the Web service. Now we need to assert that an order was created. Let’s have a look at the response of the Web service call:

<PlaceOrderResponse>
<OrderNumber>O20130125ERX001</OrderNumber>
<OrderLines>
   <OrderLine>
       <OrderLineNumber>1</OrderLineNumber>
       <Status>Ok</Status>
   </OrderLine>
</OrderLines>
</PlaceOrderResponse>

The order came back with an order number and the status of the order line was “ok”. That seems pretty awesome! But wait a minute… did our system really create an order? I know the inner workings of the ACME platform, and a simple “ok” doesn’t do it for me. 

Our systems internal structure is pretty dated and has had its problems, so I think we’d better make sure that everything is in order. Since it was so easy to hack that SQL in the GIVEN step, why not check the order in our database instead? I happen to know what fields should be set for everything to work smoothly with the rest of the system, so I’ll create a clever SQL query that makes sure everything is in place. I’ll need to join in those other three tables and that invoicing table just to make sure. If this step passes, I’ll know we’re in the clear!

AND the inventory of the item is reduced by two

Dead simple. Copy-paste some code from the GIVEN step, use the item ID as a parameter and assert that the item inventory is now two less than before. That JDBC step rocks!

What have we done?

The process I described above would have been very characteristic for my own testing a while back. This process allows me use my knowledge of the system and my dazzling SQL skills. But I’ve realized this way of working has some problems:

  1. What are we actually testing? The acceptance tests should be concerned with what concerns the customer. In this solution our customer cares about two things: The Web service and the inventory file. We did use our Web service in the WHEN step, but we didn’t think that the response was good enough. As for the inventory file, it hasn’t been used at all.

  2. Our test relies heavily on SQL and the inner structure of our database. This is not part of what we are delivering to the customer, it is only implementation detail. When this test comes up red in our nightly test run (you have one of those, don’t you?) how will we know it was actually an error in our code and not an outdated schema use in our tests?

I think we should ask ourselves one important question before we start implementing any test:

How would the customer test this?

They don’t have access to anything except what we’re delivering, and they need to make an assessment based on only that. I think acceptance tests should at least try to restrict themselves to the same limitations that we’re giving the customer. That way we’ll create more robust tests that will fail for the right reasons. Good acceptance tests target the interface – not the implementation.

Test case – reworked

Let’s give the test another iteration, keeping in mind that we’re creating an acceptance test that should be robust and valid. The WHEN step is still fine. It uses the customer’s interface, but it needs to have its parameters from another source. That’s where the GIVEN step comes in.

GIVEN there are more than two items left of an item

If we can’t use the database, this step leaves us only one option: the inventory file. The customer will pick that up from a network share each night and our test could do the same. Instead of blindly selecting the pre-made JDBC step in SoapUI, we’ll use a key gem in this tool: the Groovy script step.

 

SoapUI test steps   Groovy

 

 

I’m not an expert Groovy developer, but there seems to be a lot of those online. I guess I’d say that I’m more of a “Google-based” Groovy developer. I’ve managed to create a number of tests using Groovy that collect files, reads them and extracts data from them. In this case, collecting the file from that network share shouldn’t be a problem. Once we have the contents in our Groovy script, the rest should be easy enough.

For those of you now thinking “but wouldn’t that require the test to know the schema of the file?” – Yes, it would. But that’s our interface to the customer and therefore it’s our acceptance test’s concern.

However, I would also try to make the Groovy script trigger a new inventory file before our test. Our customer might not be able to do this, but I don’t consider this “cheating”. It’s only a matter of using our possibilities in a smart way. If we don’t do this, we might end up with that scenario where the 10th or 100th test run fails because of stale data.

THEN an order is created

My reasoning before was kind of silly. Of course we should trust our systems “ok” result to assert that an order was created. That is what we are telling the customer and that’s what we should believe as well. If we have quality issues or other concerns with our platform, they should be addressed. Just not in the acceptance test suite.

This step now becomes a lot easier. We’ll create an Assertion TestStep that checks the result of our WHEN Web service request, verifying that we had an “ok” result.

 

SoapUI test steps   Assertion

 

 

I don’t use the assertions available in the Test request step, since that would be mixing the WHEN with the THEN. I try very hard to follow the “arrange, act, assert”-way when structuring any test and as I mentioned before I use the Gherkin syntax for SoapUI.

AND the inventory of the item is reduced by two

Reconsidering my scenario, this step would be excluded. It all comes back to the very important question of “what are we testing?” This scenario is only testing that we can place orders for items with enough stock. Asserting what we do with the inventory when an order is placed is another scenario. Any test case should have only one reason to fail and I try not to have multiple THEN steps in my tests. Multiple assertions in any test should be a warning sign that says: “What are you testing!?”

Summing up

I’ve realized that acceptance tests are an important part of our quality assurance. We need all the other tests as well (unit tests, manual tests, performance tests etc.) but these tests are our main concern simply because they are our customer’s main concern. They are what we should communicate with the customer and preferably write together with the customer before we start implementation. Designing these tests well is key when it comes to delivering quality software.

Asking the question of “How would the customer test this?” helps us to keep our focus on what is important. It also forces us to “dog-food” our own software, which in turn makes it easier to find testability issues and improve the overall design. We want robust tests that pass or fail for the right reasons. That’s why we should keep them on the highest level possible.

Some might argue that we should actually test the ACME solution using the customer’s Web shop. But we can only place tests on components we control, things we are responsible for. The customer might have their own quality issues and we don’t need additional test dependencies that reach beyond the limits of our own software.

Some might think that I’m asking a lot of test writers when it comes to knowing the tool. Sure, Groovy scripts means code and code means maintenance, but if our interface is a file then we need to use it. Maybe someday SoapUI will provide a “file step,” or maybe there’s already other ways that I’m not familiar with. My point is that we shouldn’t let the tool decide what we test. We make that decision.

In this post I have tried to exemplify what I think good acceptance tests are and what problems I often run into and what basic guidelines I try to follow. Making up examples is hard, but I hope I got my message across. If nothing else, take these things with you:

  • Write tests against the interface, not the implementation. How would the customer test it?
  • Don’t be too curious or too clever. Ask yourself what you are testing.
  • Don’t limit yourself to what you have done in a tool before or what is most convenient. Groovy scripting is a Google sport.
  • Let your tests drive your design and vice versa, iterate!

 

Thank you for your time!

 

See also:






 






  • Jimmy

    Wow! I have always used SoapUI for simple manual exploratory testing but this article takes to the tool to a hole other level!  
     
    Very informative, detailed and good thinking about different testing disciplines and customer involvement. 
     
    I don’t think you will become a good programmer. You’re already better than most because you focus on the right things.

  • Greptor Takken

    Spend more time in the business, you will learn a lot of interesting things in time.  
     
    Keep up the good work, you might eventually become a good programmer.