One of the most important aspects of Test Automation is getting into the rhythm of running everything regularly with continuous testing. This doesn’t mean kicking off a test run once a week or even every other day. We’re talking at least every day. Getting into that rhythm of running at a steady cadence that helps you improve your process and your tests.
We discuss the why’s and where’s of continuous testing below. If you’re looking to get into the practicalities and learn the fundamentals of continuous automated testing with Jenkins the we have this covered for you too.
Why Test Continuously?
What we want to help you with here is understanding why you want to get to this state of testing continuously. Once we understand the ‘why’, then ‘how’ best to go about achieving this and ‘what’ to achieve it with. Before we start on the why, let’s give away the ‘how’ up front. Whatever your tooling approach for writing automated tests we recommend a tool called Jenkins. Jenkins is an Open Source CI/CD tool that lends it’s self perfectly to the process of managing and controlling Continuous Testing. We go into this in detail in the sections below.
Let’s be clear though. This doesn’t have to be part of any larger continuous development, integration or deployment process. That might be you’re end goal. However, you can get into this habit on your own and without waiting for others who you might think you’re dependent on. In fact it’s usually better if you setup this continuous testing process and system in isolation so that you have complete control everything in the early stages. What we’re really talking about here is putting yourself in a position where you are in the habit of continuous testing.
The Discipline
This needs to become a discipline that you embed in your way of working. Continuous Automated Testing needs to become your number one habit for the following reasons:
What is Jenkins and How to Install Jenkins
Checking Test System Environment Before Starting Your Tests
Test Automation Execution based on App Version Updates
Re-running A Failed Automated Test Automatically With Jenkins
Running Data Driven Tests with the Matrix Project Plugin
1
Automated tests take time to bed in and stabilise
It’s madness to think that you can write an automated test and that once it’s run the first time successfully you’ll never have to touch it again. Automated tests are code. Code takes time to evolve and become stable. Your tests will take time to evolve and become stable too. More often than not they’ll fail over and over again in the early days. They’ll fail because there are conditions you never thought of. Issues will crop up because you may have done a poor job first time round. Failures happen for lots of reasons. Running your tests in a continuous test environment will help force you to tweak and perfect those tests over time. You’ll need to be diligent about this. You’ll need to work just as hard make your tests reliable as you did to write them in the first place. Having a system in place that runs them continuously and automatically will support you in that effort.
2
Environments need time to stabilise and deliver consistency
It’s not until you start running those tests continuously (and automatically) that you start to see the issues with the environments you’re running them in. Over time you’ll see your tests fail, not because there’s anything wrong with what you’ve written, but because your environment isn’t stable. Environments can be just as fragile as code. This fragility leading to yet more failures. You have to identify issues with the environment. Going on to resolve these issues will help push you in the right direction to get the execution side of things stable. Let’s be honest here this is a lot of work. Work that’s usually not anticipated as part of this sort of project. It’s worth the effort though. And once you’re aware that this effort is all part of the process it’s usually easier for you to work your way through it.
3
Processes need time and effort to evolve
You have a release process for the applications and software you build for your customers (if you don’t you’re in a mess and this really shouldn’t be your first priority). You need to develop a process for developing, testing and releasing your automated tests too. Doesn’t matter if you’re going to plug this into a larger, project wide, CI or CD process. You need to get your automated test development process in order first. It’s usually easier to get that in place if you build your own development and test process. Keep it concise and efficient so that you have full control over the process. Adjust, tweak and perfect your local process. Automate that process as far as possible.
The Aim
You want to get to the stage where you can….
i. Develop the automated test locally
ii. Check it into a source code control repository
iii. Have it run automatically (lots of times)
iv. Get an automated report on the results
v. Tweak the test, environment and/or process
vi. Repeat the above steps until everything is stable
You might have to go round this loop several times. You might have to go round this loop ten or 100 times. Either way you have to get into this habit of working to make everything reliable and consistent.
Think of it all as automatically testing your automated tests! Making everything work correctly, reliably and consistently. Getting everything stable before you use it in anger. Stability is everything!!!
And if you’re still wondering if this is worth the effort, ask yourself this…
“How can the rest of your team have confidence in what you’re doing if, what you’re doing doesn’t work reliably?“
If you do want everything to work reliably then you’d be advised to start automating your process of improving the quality of your own work with Jenkins. To that end, in the sections below, we provide some resources that will help get started with Jenkins. Some resources that’ll give you that baseline of techniques and tactics to set Jenkins up to provide a Continuous Testing capability.
But How?
We’d recommend starting out by installing Jenkins in your own dedicated environment. This is about using Jenkins to iron out issues with your own work and your own processes. Repeating what we said above, this is the setup for automatically testing your tests and your processes. Once things have stabilised and proved reliable in your environment you can look at taking the next step as part of any larger CI or CD setup.
Our series of articles on Continuous Automated Testing with Jenkins have been written to help you get started with Jenkins. Written to help you get set up with Jenkins as your Automated Testing control panel. These articles work your through example job configurations and help you build your own Continuous Testing environment.