STC 2009- Building an Automation Framework around Open Source Technologies - 1 -

mindtree Page 1 11/12/2009

-1-

Title - Building an Automation Framework around Open Source Technologies

Paper : Building an Automation Framework around Open Source

Technologies

Conference : STC 2009

Author : Muralidhar Yalla

E-mail ID : muralidhar_yalla@mindtree.com

Address : MindTree Ltd.

Maruthi Chambers, No. 17/9C, Rupena Agrahara,

Hosur Road, Bangalore - 560 068,

INDIA

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 2 -

mindtree Page 2 11/12/2009

-2-

1. Abstract

Re-usable test automation frameworks coupled with open-source tools and technologies is a key
solution to shrink test cycle times and related costs. This paper covers an approach for creating a script-
less web application automation framework around SeleniumRC which is robust, flexible, and
extensible. Our experimental results on this framework show increased productivity and ROI, reduced
learning curve and dependency on skilled resources, eliminated usage of commercial tools and manual
configuration. The framework provides ease of script generation & management, concurrent and
remote execution, test data generation and customized reports with screenshots. Preliminary statistics
indicate that the automation framework reduces ~ 40% of total test execution effort per release cycle
under a given scenario.

2.Introduction

Frameworks are built at each level of the testing phase/cycle viz. unit, integration, and system testing.
Each framework is built on a set of rules which are derived based on need analysis.
This paper describes the methodology used in developing a robust web based automation framework
and supported by a case study on how this framework is tuned for ease of use, robust and scalable
framework.

What are the guidelines in building a framework on top of open source test automation technologies?
We need to perform a detailed automation need analysis study, to evaluate the scope and expectations
of the automation framework to be developed. The need analysis is an extensive checklist of generic
questions like “Supported browsers IE6+, FF2+, Safari, Supported platforms, protocols“ etc., which
will enable us to take a decision on go, no-go for developing an independent test automation
framework.
Framework / Automation Objectives: Identify the basic and must have features to be supported by
the framework, which will be the result of your detailed need analysis that gives us a broad wish list of
test automation objectives:
• Powerful framework for organizing test design, automation, execution around keywords, test
report generation and test data generation
• Highly Productive approach of Test Script Generation from English like language organized
as user actions, objects and values
• Framework should offer Library of Reusable Functions, like color validation, image
comparison, file Upload etc., which are extensible
• Easy to use and maintain Framework scripts by business users
• Generates test results in html format consisting of test summary, detailed reporting and with
screenshots.
• Supports component based model for object repository
• Supports Remote and Concurrent execution, which will reduce the total execution time

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 3 -

mindtree Page 3 11/12/2009

-3-

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 4 -

mindtree Page 4 11/12/2009

-4-

How should I start to identify the best open source tools / frameworks for my project?
This is a common question that customers / project teams ask. To accomplish the automation
objectives, the test managers / framework developers should adhere to the best practices as sited below
• Identify categorically the nature of application under test say, thin client / thick client / web
Services etc.,
• Current scope of this framework is considered for a thin client (Web application) automation.
• Prepare a checklist of features that the open source tool should support, ensure that the
automation objectives met or >~50%.
• Identify the open source / freeware tools that best suits your automation objective, say
Selenium-Grid / TestNG for creating concurrent test script execution.
• Perform a functional automation tool evaluation for details refer table1.1, this mechanism for
test automation projects is to correlate, identify the best fit single / multiple combinations of
open source tools that maps your automation objectives.
Figure 1: Sample data depicting the tool evaluation process

Figure 2: Instructions to fill Figure 1 template with real weighted scores

Tool evaluation provides a comprehensive overall-weighted

score, with which we can clearly identify the best possible
combination and as a best practice we ensure to architect the
framework with a combination of open source tools.
MindTree has leveraged its years of automation
expertise with open source technologies to create effective
but easy to use automation framework called Selenium
Automation Framework (SAF). Figure 3: High level architecture - SAF
MindTree Selenium Automation Framework (Figure 3) has strengths of SeleniumRC, TestNG,
Java, ReportNG and ANT. This combination ensures that the framework developed meets the
automation objectives.

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 5 -

mindtree Page 5 11/12/2009

-5-

3.SAF Architecture and Design Approach

This paper describes the “Plug and Play” hybrid framework architecture on top of open source tools; as
a best practice this architecture ensure that the
advantages of the multiple tools can be leveraged.
As per the current design and implementation
shown in Figure 4, this framework is scalable
because the controller and the execution scripts are
designed as individual modules / utilities which
give extensibility and ease of maintenance provides
the best scalable solution to run tests in parallel Figure 4: Work Flow-Module level architecture

How do we create a “Plug and Play” framework that best suit my automation objectives, project needs?
Basic components of an automation framework consist of controller, object repository, test data,
execution engine, reporting and error logging and recovery.
• There are three broad approaches in designing a controller. Spreadsheet driven, XML driven
and database driven.
• Spreadsheet driven is proven for its ease of use by manual test engineers and XML driven is
proven for its extensibility, say supports web services testing and database driven is proven for
information security, say automation of Financial projects
• Controller is the core of the framework which is used to govern the execution work flow and
modules listed below

• Configuration Module: This module will load the required test script execution environment to
govern the execution work flow

• Test Data Module: Test data stored in

spreadsheet format (Figure 5) and this will
provide input for the automation test scripts. Figure 5: Sample Test Data

• Object Repository Module: Web elements / objects stored in spreadsheet format (Figure 6) and
this will provide the same as input to the retailer test scripts

• Libraries: This is used to load generic and

application specific methods, reusable
components grouped as a method calls in
the spreadsheet as shown.

• Logging Module: This module manages

logging across the work flow, Log4j can be used for debugging and error handling.
Figure 6: User Defined Methods Template

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 6 -

mindtree Page 6 11/12/2009

-6-

• Reporting Module: Custom html reports with screenshots, ReportNG is used which is a plug-
in to TestNG framework

SAF is used for web application testing. The above architecture was chosen as it catered to most of the
core requirements, such as a hybrid driven framework addressing concurrent test script execution,
multiple browser support, extensible, easy to use, data driven testing, execute only failed or passed test
cases and with good reporting. SAF is driven by a simple spreadsheet where the application
configuration, test data, test scripts and other inputs required for the execution of the complete
automation cycle are achieved in simple steps:
• Environment setup
• Develop Test cases
• Master test suite Configuration
• Execute tests
• Report Results
Figure 7: Layered view - SAF
Figure 7 – SAF Layers explains integration of SAF with open source technologies and how each of
these technologies interlinked, revealing the potential of SAF. It contains the following components:
 Test environment initialization – Plug and Play design
 Test execution engine – Controller driven
 Error management and recovery
 Status and reporting engine

Test Environment Initialization

This module will set all the required parameters for a test run. The central framework engine will read
the global configuration file and will set the environment required for the application under test (AUT).

SAF user has the flexibility to update the configuration file to select and execute tests by customizing
the following parameters (Figure 8):
 Base URL of the AUT Type of
Browser
 Remote or Local machine
 Single or Multiple machines
 Test Suite execution speed - fast /
moderate / slow and
 Remote test case execution
monitor, which helps capture
intermediate screenshots
Figure 8: Sample Test Suite Configuration
SAF controller / execution engine will start the Selenium-RC server and confirm the status of server
back to the Test Execution engine.

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 7 -

mindtree Page 7 11/12/2009

-7-

Test Execution Engine

Execution engine clearly separates test case design and test framework execution. SAF provides an
easy way to organize test cases and test data. The test execution engine uses ANT to consolidate the
targets and execute them sequentially from starting the selenium server and then running the TestNG
suite file which will execute the test cases in the excel sheet and then shut down the selenium server.

MS Excel was chosen given its ease of use and its capability to choose keywords from the list of drop
down and check boxes. User has the liberty to execute a single step or multiple steps in a test case or
single test suite or multiple test suites.

The test design is performed, based on keywords, web objects, test data and a flag to exclude or include
the test steps / tests cases.
Figure 9 shows a sample test case to launch the application browser, performs login and user actions.
Here one single excel sheet can consists multiple test scripts
Figure 9: Sample test cases filled spreadsheet

Steps to fill the sheet are as given below:

 Write the test case id under TCID.
 Select Action “start” from the dropdown in Excel to specify the start of test case.
 Select the open, type, click action as required
 Use Selenium-IDE, Xpather, FireBug, DOM Inspector to identify the web element ID’s to fill
the Object columns accordingly.
 Select End under action to specify end of the test cases.
 Select Stop under action to specify end of the test suite execution.
In the above steps for designing test cases, the process of capturing the web elements using DOM or
Xpath and validating the same for all the intended browsers is time consuming, but this would be a
onetime effort. and as we use excel sheet for keywords / web objects the maintainability of the test
cases is comparatively very low for the sub-sequent QA cycles.

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 8 -

mindtree Page 8 11/12/2009

-8-

If the web application under test is undergoing frequent change, SAF user has to update web
components identified web ID/Xpath/DOM in a separate Object Repository spreadsheet in Figure 10
and refer the user defined custom name for that object in all his
scripts this will help to maintain global object repository per
page / module which will help manage/update scripts at one
single place instead of all the individual test scripts. Figure 10: Sample Object Repository

SAF test execution engine helps us to execute the test cases in parallel or in sequence using TestNG
framework and will return the test case execution results in html, email report.

Error Handling and Recovery Engine

The Test Execution Engine helps in recovery process. It would invoke the application from existing
state and continue test execution from the last aborted test script. TestNG on the other side will help
execute only the failed test cases in a test suite and thereby reducing the verification time. Selenium RC
and TestNG have mechanisms to capture the error logs. In case of a failure or crash, the error log will
be sent as an email along with the test case execution report.

Reporting Engine

The default report of TestNG is comprehensive, but it’s not easy to understand at-a-glance; as a result
SAF is integrated with a ReportNG listener which is a simple HTML reporting plug-in for the TestNG
framework. After the test run is completed an email alert will be sent to the stake holders.
Figure 11: SAF - Test Results Summary Report

Figure 12: Report showcasing Data Driven Test Results

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 9 -

mindtree Page 9 11/12/2009

-9-

4.Case Study

A Performance & Talent Management System Case Study snapshot. This is a web based Employee
Performance & Talent Management system that provides companies with superior control over the
lifecycle of performance appraisal. The system has several modules like career development
planning, total goal management, recruiting etc. The purpose of the case study is to pilot the run of
itest Selenium Automation Framework in addressing the challenges listed.

• Testing AJAX components • Identified 5 complex test scenarios which have

• Testing floating menus and custom web
elements
• Testing dynamic popup windows
• Time spent in manual test case execution
• Browser Compatibility testing
• Worked with dynamic pages / objects
• Searching a particular user or customer when
logged into application as Super user which
contains more than 270 users and in particular
no search button option is available.
all the listed challenges
• SAF hides the underlying complexity in
developing test suites in Java / Perl / PHP /
Ruby / Python / Selenese programming
language
• Thus, SAF could offer fast test case design and
a cohesive way of grouping of test cases.
• Given SAF's ease of use, it helped develop
complex test cases using simple keyword
actions, web objects, test data, and test
execution
• In one go we were able to get results in html,
excel report generation which used the TestNG
framework and sent a detailed report in an
email.
• SAF is used to execute part or full test suite
like only failed test cases, etc.

• Establishes a Regression Test Automation Advantages for the end-users of SAF:

framework which is flexible, scalable and
maintainable – Protection of investments
• Automated regression suite reduces costs,
improves quality and makes release process
Predictable, Faster and Reliable
• More than 75% productivity improvement
using SAF
• SAF is an extensible framework, where
future enhancements are just plug-ins to the
existing code base and ease to maintain.
• SAF could capture the state of the remote
application by capturing the screen shots
and the same is sent along with the test
repots in an email.
• Created a component based test framework
that can be bundled together based on Page
Modules
• Automation can be achieved with minimal
coding knowledge
• Built over open source tools/ libraries/
frameworks
• Support for sequential and concurrent
execution
• Support for remote execution and
monitoring
• Selective execution of test scripts
• Selective test script generation for expert
view

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 10 -

mindtree Page 10 11/12/2009

- 10 -

5.Conclusion

Thus we successfully developed a framework that helped us to exceed preset framework/automation

objectives and could execute test scripts concurrently, reducing the execution time from 2 Man days to
60 Minutes, bringing in 75% productivity improvement using SAF on a continuous basis. We believe
this approach can be carried across companies to significantly reduce the automation execution cost
without compromising on Quality. The future work of this initiative is directed towards further
evolution of SAF, SAF-Flash and “SAF as a SaaS” development model.

6.Definitions, Abbreviation and Acronyms

Acronym Description
URL Universal Resource Locator
AUT Application Under Test
DOM Document Object Model
UDE User Defined Exception
SAF Selenium Automation Framework
Controller Framework Core used in conjunction with execution engine

7.References
Item Description
Reference1: http://www.openqa.org/selenium/
Reference2: http://www.testng.org/
http://agiletesting.blogspot.com/2006/01/useful-tools-for-writing-
Reference3:
selenium.html
8.Biography of the authors

Author1: Muralidhar Yalla (Murali), Sr. Test Specialist in the Innovation Labs team at MindTree.
He has over 10 years of Software Industry experience working on open source functional automation
testing tools like Selenium, Sahi, Watir frameworks like TestNG, PHPUnit, and others, TCM tools and
code coverage tools for Java applications like EMMA and others with various languages including
Java, Perl, PHP, Ruby, and Python on WinXP.
Earlier to MindTree, Murali was with Yahoo Corporation, for 2 years, where he has worked on a
number of projects involving API testing, web services testing and functional automation using various
languages viz. PHP, Perl, shell and others on FreeBSD. He has previous stints at Synopsys, Zensutra,
Havells and Protel where he has worked on world class client / server products and worked in the area
of application development and testing of Device Drivers, worked with various languages like C, IPC
and others on Red-Hat Linux. He also enjoys working with more dynamic, productive, multitasking
and fun languages.

MindTree Limited
STC 2009- Building an Automation Framework around Open Source Technologies - 11 -

mindtree Page 11 11/12/2009

- 11 -

Co-Author 2: Ramanath Shanbhag (Ramanath), Program Director at MindTree Ramanath heads

the Innovation Labs, Delivery Excellence and Delivery Assurance activities of the Testing Business
Unit. He has played key roles in business needs analysis; organizational assessments; assisting in
building Test Competency Center, solution accelerators and frameworks; training; and consulting. His
recent contributions have been Testers’ Toolkit, Pricing Models for Outcome based engagements and
Test Quality Index prediction (TQIp) model to name a few. He has exhibited thought leadership
through external and internal evangelism like presenting papers at Step-In Conferences, and conducting
Test Leadership Programs within the organization. He is an active member of Technology Solutions
Group, and is involved in proposal creation and pre-sales presentations. He also works towards
promoting organic growth in-line with the company’s competency framework through effective
training and talent transformation programs. He has expertise in incubating complex projects, and
growing those engagements into large teams through delivery excellence and value-additions. He has
been successful in growing many projects into Multi-Million Dollar engagements. Prior to joining
MindTree Ltd., he has worked with prestigious companies like Aditi Technologies, Microsoft
Corporation, and Accord Software (a consultant to HP).

9.Appendix

Some core definitions

• TestNG is a testing framework inspired from JUnit and NUnit. It introduces powerful and
easier to use functionality. The framework covers all categories of tests: unit, functional, end-
to-end, integration, etc
• Selenium is an open source functional and acceptance testing tool for Web applications. It
supports multiple browsers, platforms, complex AJAX-based web user interfaces under a
Continuous Integration system. It facilitates users to write their own tests in java / PHP / Perl /
Ruby / Selenese / Python.
• DOM Inspector/Firebug is a browser plug-in, used to access the live DOM of web
application under test.
• ReportNG is a HTML test results reporting plug-in for TestNG framework

MindTree Limited