Contents
Before we look at some of the different automated testing frameworks in details we need to understand the definitions and context of several key terms first. It’s difficult to define what an automated testing framework is without putting it in the context of key aspects like architecture and components too. We don’t believe that you can effectively explain what a “framework” is unless you explain it in context with the concepts of “architecture” and “components”. Let’s ensure that we’re all working from the same definitions before we start exploring the different types of frameworks.
Automated Testing Framework Definitions
A few fundamental definitions from the Oxford dictionary then:
What is Jenkins and How to Install Jenkins
Checking Test System Environment Before Starting Your Tests
Test Automation Execution based on App Version Updates
Re-running A Failed Automated Test Automatically With Jenkins
Running Data Driven Tests with the Matrix Project Plugin
Framework
An essential supporting structure of a building, vehicle or object. A basic structure underlying a system, concept or text.
Architecture
the complex or carefully designed structure of something. The conceptual structure and logical organisation of a computer or computer-based system
Component
A part or element of a larger whole, especially a part of a machine or vehicle
From this then we can provide an overall definition that puts the main concepts of automated testing in context:
”The framework is the structure that gives everything shape and form. Components are the elements you need in order to build the overall system. The Architecture is how you go about slotting those components together within the framework in order to produce a working system.”
Let’s start by looking at a analogy so that we can clarify these concepts and see how they relate to each other. We’re going to start out with an analogy that’s easy to understand for everybody. Easy to understand because we all have one; the human body. A little odd but stick with us… it’ll make it easier to understand when we look at the terms in the context of automated testing.
You and I have a skeleton. Think of this skeleton as the framework. This skeleton gives us form and structure.
It’s there to help us construct a complete human system. That skeleton is built up with things like a spine, ribs, skull, etc. This skeleton is designed to hold all the components. Your skeleton gives you something to plug all the components into.
Those components are things like your liver, lungs, heart, etc. Without that skeleton your components are going to be like a bag (the bag being your skin) of bits and bobs. A bag of bits and bobs all splodged on the floor. No decent form if there’s no framework to support them. Your components won’t function at all without the structure and support that the framework gives them.
So you have this skeleton and that skeleton holds together lots of components. Within that skeleton you plugin the heart and lungs. You squeeze in the intestines and you wrap everything in skin. Without the skeleton or framework nothing maintains it’s form. Nothing knows it’s place. In order for the components to operate efficiently they need to be constrained by the framework. Remove the framework and there is no form and no structure for the components to operate in logical manner.
Good Architecture
However, even if you do have a framework, just randomly plugging in components to the skeleton isn’t going to build you a functioning human body.
How we position, and setup the relative relationships, between those components within the framework is critical. This relationship, interaction and position of the components within the skeleton can be thought off as the architecture. With the right framework, right components and the right architecture you have what is quite an amazing system. The human body.
Poor Architecture
Just to drive home the concept lets stretch this analogy beyond sensible. Say you have a good skeleton and you have a set of perfect components (heart, lungs, muscles, etc). Then you plug everything in to that skeleton. However, in ‘architecting’ the overall solution (your body) you go and put the liver on the outside of the skeleton. You have a good skeleton, good components, but you have POOR architecture. In this example it’s very obvious that you have poor architecture. What I want to highlight just how important good architecture is.
Be aware that the framework can be considered part of that architecture too. If you get the architecture of the framework wrong then you don’t stand a chance of getting anything else right. If you do get the architecture of the framework right then that framework will help drive you towards a good overall architecture. It doesn’t guarantee you’ll get a good architecture. You can still do daft things (like put the liver on the outside of the skeleton). However, get the framework/skeleton right and it does generally help drive you towards getting the architecture of all the components right.
NOTE: I’m no human biologist! I’m a test automation engineer. If the references I’m making to particular parts of the body are flawed I make no apology for that.
Now we understand the concepts we can should find it easier to put these terms in the context of a test automation system.
In the same way that we have different types of skeletons we have different types of frameworks. Taking our human skeleton analogy even further we’ll extend the skeleton concept. There are many types of animals too that have skeletons. Let’s take a Cheetah for example. This is an animal that can reach speeds of 40 mph. You could say it’s skeleton or framework is optimsed for speed.
Different Types of Frameworks
In this same way we have different test automation frameworks optimised to perform different functions. Common test automation frameworks include:
- Keyword Driven – optimised to drive the automated test actions based on a stored set of key words that dictate the actions carried out.
- Data Driven – optimised to test the same scenario or functionality many times with different data sets
- Hybrid – A combination of different frameworks optimised to meet multiple needs in automated testing.
There are many more. We’ll go into the specifics of the different types in a later article. For now though to help define our concept of a framework lets take the Data Driven framework.
An Example
The data driven framework would test scenarios scripted to be driven by many different data sets. One test scenario, repeated many times, entering different data each time. To achieve this the automated testing framework might consist of these generic high level components:
- Input data
- Expected result data
- Data reading routines
- Data entry routines (test scripts)
- Actual result capture routines
- Data comparison routines
The framework in this instance would consist of six high level components that deliver the ability to feed data in and analyse the data coming out. That framework would dictate things like…
a. format of the data we need to store (e.g. csv, excel, DB, etc)
b. test script construction (e.g. classes, methods, etc)
c. data comparison utilities (e.g. data difference utilities)etc, etc
How these components then fit together, how these components pass data between themselves, how they are organised on the file system, etc is then considered the architecture.
Same Framework, Similar Components but Different Architectures
You can have many instances of data driven frameworks. All those automated framework instances consist of the same generic components. What really sets these different instances apart is their architecture; the way they are constructed.
A framework for a share trading system may consist of many Gb of database tables containing the different test data sets of share trading information that’s used as the input data. The framework for an online insurance application form might be lots of Excel data sheets containing applicants insurance data. The way these different data sets are stored, the way the data is entered and the way in which the expected vs. actual results are compared will be totally different. Both are still data driven frameworks but both with totally different automated testing architectures.