Types of automated testing framework


Before we start looking at the various types of automated testing frameworks, it’s worth clarifying what a framework is. The definition we use for an automated testing framework is:
*The ‘framework’ is the structure that gives everything shape and form. ‘Components’ are the elements you need in order to build the overall system. And the ‘architecture’ is how you go about slotting those components together within the framework in order to produce a working system.*

We’ll be using the terms ‘framework’, ‘components’ and ‘architecture’ to distinguish each type of framework in our discussion below. These terms are defined in greater detail in [Test Automation Architecture One Point Maintenance](http://www.automated-testing.com/one-point-maintenance/).

Advantages of Using a Test Automation Framework

Typically when you don’t work within a framework you end up with spaghetti code; that is, code that’s badly organised, difficult to understand, full of duplicate code, time-consuming to work on, limited in its ability to scale and, in short, nearly impossible to update. You can read more about what a framework is and why you should be using one in [Automated Testing Frameworks](http://www.automated-testing.com/frameworks/).

If you put the effort in up front to build your code within a framework, your life will be a lot easier as your system grows. Your code will be easier to update because you’ll know where to find the relevant sections. It will be easier to reuse pieces of code (sticking to the age-old principle of DRY – don’t repeat yourself) because you’ll be able to find the pieces you need and they’ll be well isolated.

And, of course, if you use the right type of framework it will be easier to data-drive tests, and easier to build keyword-driven tests that can be used and modified by non-coders.

Above all, it makes your code easier to maintain. As your systems grow – and they will – maintenance becomes vital. Maintenance involves adding new tests, modifying old tests, and fixing issues in your tests. And it’s not just the growth of your code and tests that needs to be considered. The team using your automation code will grow, too, as the success of your automation project grows and more people want to be involved in it – non-coders (manual testers) who want to make the leap into automation, and developers who see the worth of the automated testing and want to assist with scaling it up. Either way, if you have spaghetti code, it will be difficult to share your creation with others. But if you’ve structured your code well, it will be easier to explain it to newcomers, and developers join the project will be happier to work on it.

Ultimately, if you’re the one maintaining the code, then you’re the one who has a vested interest in having a well-designed automated testing framework.


Before we get into the details about the different types of frameworks, let’s define the terminology we will use throughout this article.

*Modularity* – the use of individually distinct functional units
*Library* – a collection of methods, routines or functions available for immediate use
*Abstraction* – the act of taking away or separating based on some common qualities

Some of the other core terms are explained in our core list of [automated testing definitions](http://www.automated-testing.com/frameworks/#definitions).

Main Framework Types

We cover the following 7 types of frameworks in the sections below.

1. Linear
2. Modular
3. Library
4. Data-driven
5. Keyword-driven
6. Hybrid
7. Behaviour-driven Development


The dictionary definition for the word ‘linear’ that applies to this context is:

*Progressing from one stage to another in a single series of steps; sequential.*

In a linear setup, a list of automation steps are simply run in sequence, from top to bottom.

The system is not broken up into components – it’s just one big component that runs from start to finish, one command at a time. Because there are no separate components, there’s no architecture worth mentioning either. There’s also no logic to deal with different scenarios that the script might encounter.

In many instances these linear automation scripts are created using record and playback tools. When you record a sequence of actions on your computer, the automation tool converts those actions into a list of commands, which will then run in sequence whenever you execute your linear test automation script.

Linear record and playback scripts get bad press; however, this method can be very effective when used correctly. For example, if you record a small linear script, add some error handling, and package it up as a module, that module can then be added to a modular testing framework (see below). This is a very efficient way of rapidly building a library of modules. So do not dismiss the record and playback approach simply because everyone else does. Find ways to use this incredibly useful tool to build well-constructed automation solutions fast.

But be warned: if you do create long linear scripts with no modularity, code reuse, or error handling, you’re well on your way to building a solution that has an architecture that won’t amount to much. If you build something with no components or modules expecting it to run end-to-end without any hassle, then you’re in for a bucketful of disappointment.

Linear scripts are notoriously difficult to maintain for several reasons. First, you end up with duplicated code. Second, when the application under test changes, you’ll have to make lots of changes throughout your code.

Often when running tests against a system or application, you’ll need to repeat certain actions frequently. For example, the process of saving a document: you might be saving different documents each time but the steps taken are always the same. In a linear script, to repeat an action requires duplicating its code. If that action (e.g. saving the document) changes in the application under test, you’ll have to update your test code in multiple places. This is very inefficient and multiplies the opportunities for errors to creep in.
At the slightest change in the underlying application your scripts will grind to a halt. Even a single data value changing its state from one run to the next could render your linear automation script useless. Unless, of course, you update it. So you’ll be forever tweaking and adjusting your testing scripts to keep them running. This is no way to build a scalable automation framework.

However, used in the correct way – as a starting point for a modular system – creating small, linear units of code that you then modify can be very useful. So don’t discount this approach. It can speed up the development of your automated scripts. Just don’t expect the first pass to be of much use. To make them useful, you’ll need to keep them as small modules and embellish them to include error handling and other logic.


This brings us to the modular automated testing framework. The dictionary definition of the word ‘modular’ that applies in this context is:

*Composed of standardized units or sections for easy construction or flexible arrangement.*

This is next step up in the food chain, so to speak. Building a modular setup involves break up both the application under test and the automation code into contained units. Therefore, you need to identify the parts of the application that can be run in isolation, and then write dedicated automation scripts for each of those parts. You can then build your test cases by linking those pieces of code together in a chain.

Sometimes it works out with a neat 1-to-1 mapping between the application modules and the code units. Sometimes it doesn’t work out quite so neatly.

There are three aspects to modularisation: breaking up the application under test, breaking up the automation code units, and breaking up the test cases/steps. How you link these elements together is what forms the architecture.

A modular framework allows you to keep the architecture fairly simple. You can create a single controlling script that makes calls to run the individual units of code. And if you break up the controlling script into sections intelligently, you can base those sections on your test cases. You end up with a relationship like this:

Test cases -> controlling scripts -> code units -> application modules

When arranged like this, the architecture is defined by the relationships between the code units written for specific modules of the application. These code units are called by the controlling script.

This is still quite simple, but is a significant leap forward in maintainability and scalability. Firstly, you don’t have repeated code. If you want to run a piece of automation many times, you simply call the relevant code units many times in the controlling script. This makes if far easier to build up your test cases. Need a new test case? Simply link together a different sequence of code units in a new section of your controlling script.

If the application under test changes then, in theory, you only need to update the code unit that relates to that part of the application. All test cases using that code unit will pick up the change as a matter of course. Therefore, you only need to change one code unit, and all automated test cases should continue to run.

Library-based Framework

This framework type extends the modular framework a little further by constructing it so that each module relates to a specific part of functionality in the application under test. So a library will contain a number of functions where each function relates to a specific feature in the application. For example, you might have library functions for the login process, the add record process and the log out process, respectively.

The modular framework focuses on breaking up the code and scripts into units that make sense at the test level. Each module is responsible for completing a set of test actions that make up a complete test case. The code relating to the module covers that whole test case. A different test case would have its own dedicated module, which may cover similar parts of the application as other test cases.

The library framework is focused on building modules for each section of an application’s functionality. Functions in the library can then be called by different test cases that need to utilise the same function to run a specific part of the application.

Libraries of functions can even be split up into multiple libraries, where each library contains related functions. For example, you might have a library of functions for the user account management functionality in an application, which would contain functions for changing passwords, chaining user names, deleting accounts, etc. Another library might contain functions for creating new records. How functions are grouped really depends on the application under test, but the goal is always to make the test automation code as manageable and well organised as possible.

Both the modular and library approaches touch on the concept of abstraction. Abstraction is

*the act of taking away or separating.*

In the context of automated testing, this involves separating parts of our system into logical sections. We place related functionality into different sections so that it’s easier to conceptualise, and to maintain our system. We’ve separated our controlling script from the code units that interact with the application under test. This is an important concept to understand in automated testing, and coding in general. It is far easier to understand and maintain your code/framework if you abstract out similar types of functions into separate layers. We might, for example, implement the following three layers of abstraction in our automation code:

– Controlling script
– Functional modules
– Interaction scripts

The lower layer is responsible for interacting directly with the objects in the application under test. These interaction scripts enter and gather data from the application. For example, you might have a function to click buttons, another function to enter text in a text box, etc.

The middle layer is a set of scripts responsible for completing a set of actions against the application. For example, a functional script may be written to complete a login process. That login process needs to enter text in both the username and password fields. Therefore, it will call the enter text function in the interaction layer twice – once for each of these text fields. It will also call the interaction scripts responsible for clicking the login button and for checking the data displayed in the dialogue box to see if the login was successful.

The controlling script, at the top layer, is responsible for chaining together the execution of a number of actions from the functional modules layer. The controlling script might call the login routine, the enter record routine and the logout routines of the functional modules layer. Calling a chained sequence of functional modules makes up the full test case for the login and create record processes.

From this you can see that each layer separates out common types of functions into different areas of our automated testing system. Now, if we need to automate a different part of the application, we know we need to create some functional modules in the functional module layer. Then to create a new test case we need to define a sequence of calls at the controlling script layer.

The concept of abstraction is an important one to grasp in test automation. It’ll crop up again in the other types of framework we’re going to discuss shortly.

Data-driven Framework

The downside to the linear, modular and library frameworks is that the data needed at execution time is all contained within the code. If you need a different data set for subsequent test runs, or want to repeat the test runs multiple times with different data sets, then you need to modify the code. This approach doesn’t scale well. Therefore, we need to add data-driven capabilities to these frameworks.

With a data-driven framework you have two significant enhancements. First, you need to have a data set stored in a text file, CSV file, Excel file, or database. Then you must enhance your scripts so that this data is passed to the modules/functions at run time. In this way you are no longer hard-coding your scripts but enabling them to accept whichever data values are passed in.

This approach requires significantly more planning and design work up front if it is going to work well. Consideration needs to be given to which data values can be hard-coded (there are always some that can be hard-coded) and which should be stored in a data file. The format of that data file is important, too. Sometimes a simple text-based format like CSV is necessary because it is easier to manage with source code control tools; for example, it’s far easier to diff a couple of CSV files than to compare two Excel files. In other cases a more complex database storage mechanism is more practical (especially when you’re copying data sets from existing production data sets). Once you know what data you need to store and how you’re going to store it, you need to consider how that data will be read and passed to the modules or library functions that will be using it. For example, if the data is stored in CSV files, it may only require a simple file read process. Different modules/functions will need different pieces of the data, so how you divide that data and pass it around your code is important. Some of the data may need to be manipulated before being used as part of a data entry routine; for example, you may need to increment a date value or deal with unique ID values.

As you can see, the data-driven approach creates significantly more levels of complexity. This additional complexity is usually worth it because the main benefit of automated testing is that automation scripts are very good at repetitive and reliable data entry that adds to your test coverage capabilities.

Although repeating test scenarios with different data sets increases test coverage, it may not increase it by very much. It’s usually the different functional capabilities of an application that need testing rather than repeating the same functionality just with different data values. Scripting the test automation for the different functionality is the part that takes the most time and effort.

Keyword-driven Testing Framework

A modular framework involves creating modules corresponding to test cases. A library-based framework is based on a library of functions related to small functional parts of the application, and a higher-level script calls a particular sequence of these functions to complete a test case. The keyword approach extends this by yet another step.

You could say that we extend existing frameworks by adding another layer of abstraction. That layer of abstraction processes a list of keywords. Each keyword, when processed, results in a call to a module or a function in a library. You can extend other types of frameworks with the keyword approach, but we’ll stick to discussing the extension of the modular and library types here.

In the keyword approach, each action is driven by three things:

1. Keyword
2. Data (optional)
3. Object

First, and critically, is that you’ll need a list of keywords that relate to each module or library function. The automation code reads through those keywords and then makes a call to the corresponding module or library function. For example, you might have three keywords for login, create record and logout. These keywords could be contained in a text file, Excel file or even in a file or code unit in the automation tool itself. The point is that this list of keywords creates a layer of abstraction above the modules and/or functions that makes it easier to list all the actions the automation scripts need to complete. A list of keywords could even be defined or edited by someone not involved in writing the automation code.

Second, then, is that each keyword can have a number of data values associated with it. For example, the login keyword would be stored together with the login username and password. When processing the list of keywords, the script would read in the login keyword along with these two values. It would then call the login function, passing these values to it. This makes it easy to reuse the login function in different contexts (test cases) with different values.

Thirdly, each call to the underlying module or function should reference the object it needs to interact with. For example, you may have a keyword that carries out text entry in a text box. You can define the ‘enterText’ keyword in your table, and with it provide the text to be entered. However, your application could have hundreds or even thousands of text boxes. So you’ll also need to identify which text box object the function should interact with.

Although you are calling these modules/functions with data – potentially different data every time you make the call – this is not the same as a data-driven framework. We’re not repeating the test case with different data sets here. We’re just reusing the function called by the keyword in different contexts.

The real difference between the keyword-driven framework and the data-driven framework is this. In the keyword-driven framework the data is contained alongside the keywords. The data is removed from the modules/functions but is then hard-coded in the list of keyword instructions. In the data-driven approach the data is abstracted out to another layer – contained within one (or a few) files. A script is then responsible for reading that data and passing it to the modules/functions. That same set of calls can then be repeated for each of the data records.

So whilst it doesn’t have the same capabilities as the data-driven framework, there are a number of significant advantages to the keyword-driven approach. First, data is no longer hard-coded in the modules/functions; it is stored externally alongside the keywords. This makes maintenance far simpler, and makes the framework far more scalable. It becomes simple to copy a sequence of keywords and modify the data calls. We end up with a kind of pseudo data-driven capability. Second, it gives you the ability to let others write the test cases just using the keywords along with some data values. That person doesn’t need to be able to code; they just need to be able to design a test case and construct that test case out of a list of keywords and data values. Of course the test automation engineer will have to have developed the code behind those keywords. Once that code is in place, though, it becomes reasonably simple to string test cases together without any knowledge of coding.

The other advantage of this approach is that you can copy and paste parts of your keyword lists to create new ones. You can then modify the data and/or object name to create a new variation of the test case. This is a simple way to create new permutations quickly and easily. Building new test cases becomes almost as simple as writing manual test cases (assuming you have already written the code behind the keywords).

Comparing the Data-driven and Keyword Approaches

It isn’t a question of one being better than the other outright. We need to consider which approach is best for the job at hand. Let’s assume that you’ve built the test case and need to run it many times with different input data each time. The data-driven approach is clearly best in this scenario. If you need the flexibility of being able to script different paths through your application, then the rigid data-driven approach, which has a fixed path through the application, is not the way to go.

If you need the capability of scripting various scenarios representing different paths through the application, then the keyword-driven approach will be most suitable to your needs. It allows you bolt together different variations of keywords to drive the different paths through the application. The downside is that if you want to repeat a few scenarios just with different data values you’re into cut-and-paste territory and will end up with many similar sets of keywords. If the application under test changes, you will then need to update many similar blocks of keyword scenarios.

One solution to this problem is the hybrid test automation framework.

Hybrid Framework

As the title suggests, the hybrid framework combines parts from multiple frameworks. Of course with so many different frameworks to mix and match, the permutations are endless. We’ll simplify things here by focusing on the two frameworks that we consider viable solutions in the real world: the data-driven framework and the keyword-driven framework. Let’s consider the approach where we develop a keyword framework and wrap the data-driven capability around it.

With the hybrid approach we build out the capability to drive our test cases from keywords that control the flow of the actions carried out by the script. Those keywords also pass data to the underlying modules and functions where appropriate. Note that in this setup we might not store the keywords in an external file; it’s usually simpler to have them in the scripts.

The script that contains the keywords can then have a data-driven script wrapped around it. That data-driven script reads the data from an external data source and passes the data to the script containing the keywords. The data set passed then provides all the data parameters required by the keywords in the test scenario.

This approach does of course add another layer of complexity to the test automation setup. We’re implementing multiple layers of code to provide further capabilities to our framework. In the case of the keywords we’ve now embedded in our scripts, we’ve removed the simplicity of building out test cases and scenarios in a simple spreadsheet-based table. So the extra functionality does come at a cost.

And this is often the balancing act a good test automation engineer is working with – balancing ease of use (e.g. ease of creating test cases) with functionality that greatly increases test coverage.

Behaviour-driven Development (BDD)

BDD is a relative newcomer to the automated testing framework game. BDD really only came onto the scene around 2005. BDD itself is touted as a framework, but really it’s a framework in a different context. It encompasses many things, including the way people communicate and work together.

The key feature of BDD is that it’s designed to let you define application behaviour in plain English. It’s similar in approach to our keyword test framework. With the keyword approach we define our tests with an action keyword, some data (optional) and the object to interact with. With BDD we still define our tests with an action, data, and the object acted on, but instead of using just keywords, we use an English sentence structure that conforms to a consistent construct: Given, When, Then.

– Feature: User authorization
– Scenario: The user can log into the system
– Given: The user is registered in the system
– When … the user enters a login username and password
– Then … the user is logged in

What we need from our test automation framework is the capability to read and interpret tests defined using the BDD construct. With the keyword approach we read the table containing the keywords. With BDD we read the English sentences created in the Given, When, Then construct and convert those into actions that the automated testing tool can carry out.

The big advantage of this approach is that we move towards a test specification method (the Given, When, Then construct) that is more focused on testing the behaviour of the application. More than this, we’re starting from a definition that is usually agreed upon up front between the business, developers and testers. Therefore, we’re in a position to write the BDD specifications used in our automation framework well in advance of the application being delivered. We could even start to produce the code needed to enact these specifications.

The downside to this is that we’re adding more complexity to our automation framework. Our automation code needs to be able to process and interpret the more complex Given, When, Then construct. There are various libraries and tools available to simplify the process, but this still adds a degree of complexity. So, is the additional complexity offset by the advantages that BDD delivers –of having shared understanding up front?

Different Concepts of Frameworks

It’s worth mentioning at this point that a framework can be considered at different levels. Consider, again, our definition of a framework:

*…the structure that gives everything shape and form.*

At a high level you have the overall automated testing system consisting of various automation tools (GUI, API, unit testing, etc.), source code control tools (Git, SVN, etc.) and execution tools (e.g. Jenkins). This can be considered a framework at the system level – a system within which you build out a selection of tools and processes. This is a high-level framework within which everything from code management, execution and reporting operates.

Then, at a lower level, we might view the code that is built within the test automation tool as a code-based framework that could be implemented as a module, data-driven, hybrid or BDD type of framework. This type of framework is localised to the tool or coding language within which you create your tests.

Then you may also consider the concept of unit test frameworks, where a unit test is defined as an atomic test that is run against a unit of code that is isolated from the rest of the code for the purpose of testing. Unit test frameworks include NUnit, JUnit and TestNG, among others. These types of frameworks are at the lowest level of framework, as they are usually embedded in the code being tested and run as part of the build process.


In short, there are many ways to conceptualise the term ‘framework’. From a test automation perspective the term framework is usually used to describe the way in which the code is structured to help us develop, execute and maintain our automated tests. Selecting the right type of framework and implementing it effectively is what determines success. You’re almost better off having no framework at all than implementing one badly.

Therefore, it is important to understand the underlying concept of framework, and to understand the different types of frameworks and how to implement each one effectively. We’ll explain the implementation part of the process, using practical examples, in our next article on test automation frameworks.