This post was part of a series I wrote for Improving and originally posted on the Improving’s Thoughts Page.

First Principles

The earliest science-fiction novels are littered with examples of machines which lift the load of work off the human back, providing increased ease and great rejoicing… until those same machines suddenly decide they’d like to be the ones in charge. Just like our fictional robot overlords, Automated Software Testing tools promise to increase our productivity. Tales abound of benign automated software testing systems turning on their keepers. And in the process, they bog entire projects down in the chaos. This gives rise to some very reasonable questions: Should I even automate my testing? I have everything I need right now with manual testing, so why take the risk? Automation is a broad subject. It promises many efficiencies and radical gains in productivity. But like Lean and agile before it, the industry is full of horror stories of Automation gone awry. So to answer whether you should automate your tests (hint: you should), we must look deeper.

Before we can answer whether to automate testing, we ought to review the goal. Why do we test? And specifically, what does testing buy us? Well…would you ever release software that hadn’t been tested? I’m willing to bet the answer would be NO!, in all caps. In most cases we won’t release untested software because we are not sure if it will work, or if it will have bugs in it. That is our confidence that we are delivering value in a particular release is low without testing. As a result of low confidence, we perceive a higher risk in that release. This risk might appear in the form of extra work to fix new bugs. With scarier alternatives like lost and unrecoverable data or other large impact failures. Fundamentally, the reason why we test comes down to mitigating the risks and increasing our confidence.

On adopted pets and friendly robots

With that in mind, we do need to strike a balance between perfect confidence and actually getting the product out the door. After all, software sitting on the developer’s machine doesn’t generate value. It must get into the hands of the customers! We cannot spend forever testing every inch and every possible use-case of a given piece of software. And this is where automation comes in. When applied correctly, automated test execution allow us to more thoroughly test the current version of our software within a given amount of time. Note that I said, ‘when applied correctly’. Not every team is capable of applying a sweeping automated system right out of the gate. There are several disciplines which need to be grown within the development, testing and even in the operations side of the team to support an effective automation solution. Now the good news is that automation doesn’t have to be an all or nothing game. In fact, there is a way to approach it so that the system can grow with your team as their discipline and skills grows accordingly. You can think of an automated testing suite like a pet. The usual story is that for some celebration you adopt a pet for your family. The children will need to learn to care for it, and clean-up after it. The family learns to adjust around the pet, and with proper training the pet brings joy to the family. But… if you fail to train it properly, you end up constantly needing to clean up little puddles in the corner, and deal with the malodorous after-effects. With that lovely analogy in mind, I want to suggest that in most cases, a well-trained, well-maintained automation suite is beneficial to the team that relies on it.

Now, one thing that every decision maker has to consider when looking to adopt a pet, or create an automation suite, is cost. And not only the start-up costs either, but the ongoing maintenance costs as well. After all, dog food isn’t free. Nor is electricity. When it comes to costs and benefits with automation systems a lot of the answers have to boil down to ‘It depends’. How disciplined is your team? How much experience do they have with such frameworks? How collaborative are your development, testing and operations teams? But let’s set those questions aside for a moment. Most of them can be addressed by how you approach automation, and we’ll explore that topic in a future article. Instead, let us focus on something more fundamental. Are you using the right tool for the right job?

To be clear, by ‘right tool’, I am not referring to which framework you should use. Instead think about how you are assigning your team. For this model to work, consider your automation suite to be a little robotic companion from one of those sci-fi shows. Your automated testing tool is highly skilled at exactly one thing. It is excellent at following exactingly specific directions. Not only can that little bot do the work faster, it can also follow those instructions with fewer mistakes than any human could hope to match. For testing that means less human error, and faster test execution. But the robot is not good at learning what is important to test. Nor is it particularly clever in determining if a given test passes or fails without explicit instructions on how to decide that. Naturally, this allows us to highlight the strengths of your testing team. Being human, one of their greatest gifts is their ingenuity. You can hand them a system, or even a description of a system and ask how you might confirm the behavior. Most QA analysts will be able to tell you how to do what you’re asking. Not only that, they’ll even look around at corners or oddball situations that you hadn’t thought of yet. This is called exploratory testing or ad-hoc testing and it is a field where humans excel!

If our goal is to build confidence in our release most efficiently, then our challenge is how to most effectively distribute our resources. ‘How do we get the most bang for the buck?’ as it were. The ideal situation would be to leverage each team member to the best effect in their area of specialization to support the needs of the team. The reality is often very different. Most software projects start as a small- time operation. They have just enough testers to manually test the app as it grows. Little by little the project grows. As the team hits its stride, the testing group becomes hard pressed to both test the new functionality, and discover if any existing features broke. This is the point when most teams begin to consider automation. They hope that by building up an automation suite they can reduce the burden on their existing test resources, without having to greatly increase the overhead and expense of getting a release out. At first glance the backlog of things which you want to test and cover can be daunting. But remember the automation process is just that, a process. The suite can and should grow with your team’s capacity to steward and maintain such a system.

Getting started is sometimes the hardest part. If interested in learning how to best implement automated software testing into your team process, reach out to us for guidance or come back for part 2 of this series where we’ll explore how to start building an automation suite with increased collaboration between your developers and your testers.