The Value of Continuous Integration at SmartBear

continuous integration

At SmartBear-Austin, we develop CodeCollaborator, CodeReviewer and PeerReview Complete. We’re often asked about our own internal development process – how we use continuous integration, how we integrate CodeCollaborator into our own processes as we develop, etc.

For us, and for many of our customers, continuous integration (CI) is an important part of the development lifecycle. CI has become an integral part of the Agile development landscape precisely because it enables teams to incrementally push, build and test the code to minimize risk and ensure working software at all times. The tool that we have selected to help us achieve continuous integration is Jenkins. After Jenkins has done its work, it’s up to our QA staff to take it further. But before any of that takes place, though, code has to be written, tested and reviewed.

Here’s an example of the steps we take.

Step 1: Improve it

When people think of development, what usually comes to mind is code. Pull out your editing tool of choice and write some code! However, the code that we ship is not the only thing involved in this step. Documentation, tests and internal tools might all need our attention here.

Step 2: Test it

Our unit tests and integration tests are built right into our build system. Running them is as easy as compiling the code; however, the more important test at this step is the sanity check. Do the end results of our effort make sense? Do they do what we thought they would do?

Step 3: Review it

Once we’re satisfied with our changes, and convinced that they’re right, it’s time to get other people’s input. As a general rule, changes need to be reviewed before we even attempt to merge them into our main git repository. (If you’re curious about our use of git, check out this article on best practices for using git and CodeCollaborator. Using a distributed source control system allows members of our team to commit their code locally before even starting a code review, so most of us commit early and often. For all code review, we use (what else?) CodeCollaborator. When we’re ready, we review the entire batch of commits at once. When the feature branch is complicated, our “Review Unpushed Commits” command helps keep it simple.

Step 4: Request a merge and let Jenkins take over

Having satisfied all of the reviewers, it’s time to combine our work with other people’s. This is where we ask Jenkins to take over. We could simply push changes to the master branch in our main repository. Instead, we ask Jenkins to do more testing. This way, when we request the latest version of our code, we get something that works. Jenkins automates and drives the effort of starting an Amazon EC2 instance, merging the latest changes with master, and re-running the unit and integration tests on it. If this process works, we end up with a master branch that compiles, runs, passes tests and includes the latest work that we’ve done. If that process doesn’t work, we end up having to solve the problem ourselves, rather than breaking the build for everyone.

(My co-worker, Cody Casterline, is working on a more detailed description of this part of the process. You can check back later his more detailed description of our merge requests.)

Step 5: “Dogfood”

If you’re familiar with the notion of “eating your own dogfood,” then the fact that we call our internal CodeCollaborator server “Dogfood” should make sense to you. Jenkins takes one last step every night: It checks out the tip of the master branch, compiles it, tests it and, if it runs, upgrades our internal server to the latest version. This means that, when we ship a product to you, we’ve been using the changes we added — usually for quite a while.

If you spend your time collaborating on code and aren’t using a continuous integration system, I recommend that you investigate some options. Jenkins lets SmartBear-Austin spend more time developing and less time worrying about whether our test suites are being run.

See also:


  1. It’s interesting to see that Testing team is not taking part to the CI flow. I’m curious to sew whether it’s not an option or you’re not there, yet:)

  2. Thanks – have always been curious about how Smartbear does their stuff.  
    I found this link from someone at Smartbear who said “A quick blog entry outlining how we use our own products internally” which, thankfully, isn’t really what this is about. Gotta say there are some elements of Smartbear which come across as completely out of touch. Plus all the click-through ads I see now for document review are kinda turning me off.

  3. Ben Austin says:

    Jimbo, we really do appreciate feedback from our readers, however, it’d help if you could be a bit more specific as to why you think we’re out of touch. Feel free to email me ( if you’re willing to elaborate.

  4. Ardei – our testing team has a parallel workflow. Our continuous integration is used to give the development team quick feedback. For example, the internal server updates every night. The testing team, on the other hand, take selected builds and verify things more deeply. That deeper verification tends to take longer. Some of our automated testing used to take more than 40 hours to run; those had to be done in parallel if we wanted to keep doing nightly updates.

Speak Your Mind