Running Data Driven Tests with the Matrix Project Plugin

Overview

Goal: To selectively Run Certain Data Driven Permuations as part of a data driven test run

Jenkins Plugin You’ll Need:

  1. https://plugins.jenkins.io/text-file-operations
  2. https://wiki.jenkins.io/display/JENKINS/Matrix+Combinations+Plugin

Objective: configure a Jenkins automated test execution job so that we can selectively run certain test permuations for a particular test.

The assumption here is that you have one test and that this test can be run multiple times with different data sets. What we want to do is have one Jenkins job that is dedicated to running one automated test. However, that one test can be driven with data and can be run many times with these different data sets.

In this example then we’ll simulate our test case as a command line call to complete a calculation from a Windows command prompt (other flavours of command prompts are available but aren’t covered here).

REM this build step simulates our test
REM in the real world it would be....
REM a call to run a test case externally
set /a 2*2

We’re just using this to simulate a test case in this example. In the real world this could be a call to a Selenium test or a LeapWork test.

What’s unique though is that this test case is data driven. This data driven component could be in the form of a CSV file with lots of permuations. Those permutations including test input values and the expected results. In our example we’ll use the following:

input_A, input_B, expected_result
2,2,4
2,3,6
2,4,8
2,

In the real world this data is likely to be contained in a source code repository or a database. Either way one test is repeated many times, each time using 1 permuation from the data set.

What We’re Aiming for

We’re aiming to split up test runs into more managable run chuncks. Essentially if you have a group of similar tests that are either data driven or driven by some sort of configurable parameter then you can get Jenkins to run the test multiple times with the different data/parameters.

If you run the test multiple times then this will of course create multiple test result records. We’ll see how Jenkins provides one overall test result value but also allows you to drill down and look at all the individual tests.

Install Additional Plugins

All of this depends on the Conditional Build Step plugin. First we need to install this plugin. From your Jenkins Manage Plugins page find the ‘Conditional Build Step’ plugin along with any dependencies

You’ll want to find these two plugins (names have evolved a little):

Text File Operations

Matrix Configuration Parameter

Simplest way to install them is just select them on the Plugin Manager page under the ‘Available’ tab then click the ‘Install without restart’

Or you can download the latest .hpi file using the link below and install it manually in your instance of Jenkins

  1. https://updates.jenkins-ci.org/download/plugins/text-file-operations/
  2. https://updates.jenkins-ci.org/download/plugins/matrix-project/

Note that we’re only installing the Text File Operations plugin because we want to use this in our example below to create a data file for our tests.

Setup

First we need to create a new Multi-configuration Project. Click on ‘New Item’ link on the Jenkins home page

And then select the ‘Multi-configuration Project’ option. Provide a name for your job and click the Ok button

Then we need to make sure we ‘Parameterise’ this build. The para… DO WE?

Next we’re going to add a few build steps to simulate a data driven test and create another build step that produces that data. You should be aware that these build steps will be replaced with steps that actually call YOUR tests. We’re building these just to demonstrate the principals here.

Add the first build step ‘Create/Update Text File‘. Enter the file name ‘TestData.csv‘, set ‘Create at workspace‘ to True, Overwite for file options and enter the following for the content:

test_id,input_A, input_B, expected_result
#1,2,2,4
#2,2,3,6
#3,2,4,8
#4,2,5,9

You should end up with a build step that looks like this:

All we’re doing here is creating a CSV file that contains a few data perumtations. Each permutation identified by a unique tag or test_id (for example #1). We have two input values and an expected output value. NOTE that the final data permuation, #4, is designed to fail (the expected result is incorrect).

Add the second build step ‘Execute Windows batch command‘. Enter the following batch commands

REM Find the data permutation we want to run the test with
REM and place that permutation in a temp file
type TestData.csv | findstr #4>temp.txt
type temp.txt

REM Load the contents of the temp file (e.g. one data permutation)
REM into a variable called PERMUATION
set /p PERMUTATION=<temp.txt
echo %PERMUTATION%

REM Split the value out from the PERMUTATION variable into
REM separate variables (this is very crude and wont scale!)
set INPUT_A=%PERMUTATION:~3,+1%
set INPUT_B=%PERMUTATION:~5,+1%
set EXPECTED_RESULT=%PERMUTATION:~7%

REM Write our test conditions and expected results out to the console
echo INPUT_A is %INPUT_A%
echo INPUT_B is %INPUT_B%
echo EXPECTED_RESULT is %EXPECTED_RESULT%

REM Run the test and save the result in the Result variable
REM Also write the result to the concole
set /A Result = %INPUT_A% * %INPUT_B%
echo %Result%

REM Test to see if the result matches the expected result
REM Then write the test result value to the console
if %Result% == %EXPECTED_RESULT% set res_name=PASS
if NOT %Result% == %EXPECTED_RESULT% set res_name=FAIL
echo %res_name%

REM if test passed then exit with return code 0 (pass)
REM if test failed then exit with return code 1 (fail)
if "%res_name%" == "PASS" exit /b 0
if "%res_name%" == "FAIL" exit /b 1

You can read through the comments in the script to get an understanding of exactly what is happening. In short though we’re taking one data permuation and running the following test:

%INPUT_A% * %INPUT_B%

Then we check the result against our EXPECTED_RESULT value. Depending on the result we exit from the batch commend with either a=0 or a=1. If it’s 0 jenkins takes this as a pass. If it’s 1 jenkins takes this as a fail, and sets the build result value appropriately.

We can actually run this now before we’ve created the Matrix conditions in the jenkins job. Click save and then ‘Build Now

You should see this fail.

Go back into the job and change this line in the ‘Execute Windows batch command’ build step:

type TestData.csv | findstr #4>temp.txt

to

type TestData.csv | findstr #1>temp.txt

Instead of having the test hard coded to run perumutation #4 now we’ll run with #1. #1 has the correct expected result. So when you run again the next time (save the job and click ‘Build Now’ the job should Pass.

Now we’ve check the default job we’re ready to configure the Matrix part of the job. The part of the job that will run all 4 permutations and record individual results for each data permuation we run the test with.

Configure‘ the job and in the ‘Configuration Matrix‘ section select ‘User-defined Axis‘.

Give the name as ‘test_id‘. Then enter all the test id values in the values text box as follows:

Only one thing left to do. In our ‘Windows batch command‘ build step, where we run the actual test, we need to update it with our test_id variable. In the ‘Execute Windows batch command’ then, modify this line again,

type TestData.csv | findstr #1>temp.txt

to

type TestData.csv | findstr %test_id%>temp.txt

Now what Jenkins will do is, for every value of test_id defined in the Configuration Matrix section (e.g. #1, #2, #3 and #4) repeat the execution of this Job replacing the %test_id% value with the different test id values. Thus we repeat the test four times, each time with the differen csv data row values.

Save the job and you should see each of teh configurations listed:

Now click the ‘Build Now‘ button to run the tests. You should see your overall build history increment by 1 and a list of the pass/fail results for all the permuations

And hey presto we’ve run the same job 4 times, each time with a different data permuation. You have an overall test run result (the overall job/build/run result) and you have individual test results for each permuations. You can drill down into each of these individual test permuation runs by clicking on the test_id value.

And then on the individual permuation page you can drill into the console view and see exactly what happened with this test run instance…

Summary

Whilst this is a pretty simple example it’s not difficult to build this out with more complex tests with larger data sets. You can even build out the matrix to have multiple matrix values thus developing quite a clever why of testing with many data values. Taken a step further you could go as far as having many Jenkins client machines and run the different matrix combinations across a pool of different machines so that you’re running all the tests in parrallel. It’s not difficult to build this out and scale things up once you’ve grasped the concept of defining the Matrix combinations and making sure you feed those values in to the build steps that call your tests.