March 31 – April 2, 2009

Optimizing the Testing

Effort with Keyword-
Driven Frameworks
Brian Massey, IBM
ƒEnter Text/graphics here
ƒEnter Text/graphics here
ƒEnter Text/graphics here
 Quick Bio

ƒ Product Manager
ƒ No worries – not here to sale you anything (but to ensure the kids
get fed I have to mention them)

ƒ As part of Rational Services organization helped develop and deploy

frameworks for our customers

ƒ Established Software Quality Centers of Excellence for Companies

ƒ Experience as Developer, Tester, Architect, Dev Manager, QA Manager

ƒ Used to build Air Defense Systems for a living

ƒ Developed and tested embedded systems

ƒ Father of 6 so prepared for anything

 Objectives

ƒ This class presents a framework for keyword-driven testing.

ƒ How to easily plan and create manual tests using a
framework
ƒ You’ll learn how keyword bridge manual testing to
Automated testing
Agenda

ƒ Identifying the problem

ƒ The Basics
ƒ Testing Frameworks – how and why?
ƒ Solutions in the industry

ƒ How will Keywords help me

ƒ Pro and Cons of Keyword testing
ƒ Assessing Automation Success
ƒ Metrics
 The Problem
ƒ Companies want to ultimately achieve Test Automation
ƒ Faster release cycles achieved due to concepts such as
Agile
¾ UI not stable for automation early on
¾ Have to compress test cycle
ƒ 80% of Testing still done Manually
ƒ Most of the skills in any given Test Organization are Manual
Testing
ƒ Companies want to see fast return on Investment for Test
Automation solutions
Skills Separation

ƒApplication Experts
60% ƒ Plan and Design Tests
ƒ Manage Test Data

30%
ƒImplementers
ƒ Built scripts based on modules and
utilities

10% ƒCoders
ƒ Write modules and utilities
ƒ Could be developers “on loan”
“Cockpit Alignment” Problem

Proper Alignment with Each Tester’s Skillset

Test Automation Manual Testing

traditional record/playback
“caught in the middle”
Why is Keyword Driven Testing
Needed?
ƒ More productivity desired from the non-technical base of users
ƒ Test automation tools need to be easier to use for business analysts and
subject matter expects
ƒ Details of scripts are hidden from the user
ƒ Users don’t have to learn the script language to write scripts

ƒ ROI
ƒ Companies want faster ROI on purchasing of Automated Test Tools
ƒ Companies want investment is test professionals to provide quicker
returns

ƒ Need to reduce the cost of developing and maintaining scripts

ƒ Need to improve script resiliency

ƒ Reusability
ƒ Common script development techniques
 The Problem
ƒ Where is the Easy Button

IBM
Agenda

ƒ Identifying the problem

ƒ The Basics
ƒ Testing Frameworks – how and why?
ƒ Solutions in the industry

ƒ How will Keywords help me

ƒ Pro and Cons of Keyword testing
ƒ Assessing Automation Success
ƒ Metrics
The Basics – What is a
Framework

ƒ Keyword works with an automation framework but just

using a framework does not make Keyword Driven testing

ƒ Frameworks
ƒ Frameworks are simply Function Libraries extending the
Automated test tool being used.
ƒ Libraries, functions, etc created to interface with Applications.
ƒ Automation Scripts utilized these Libraries
ƒ Very proprietary to the organization and often to the AUT
ƒ Do not resolve the usability from the non-technical tester
ƒ First step to the evolution of Keyword Driven testing
The Basics – What is a
Keyword
ƒ A Word that provides a simplified
representation of a more complex
concept

ƒ E.g. Keywords Encapsulates

¾ Business Logic
¾ Windows
¾ Objects
¾ Screen Functions
¾ Services

ƒ Thus, a Keyword is A logical

representation to one of the
components above

ƒ Keywords need to be properly

defined for Automation Experts to
develop proper framework
Many Names in the Market

ƒ Many Names one Concept

¾ Action words / Actions Based testing (Hans Buwalda, LogiGear)
¾ Test Frameworks (Linda Hayes, Worksoft)
¾ Third-generation test automation (Edward Kit)
¾ TestVerb technology (TestQuest)
¾ Keyword Driven Testing/Business Process Tester (IBM
Rational/Mercury)
¾ Table Driven Testing
¾ Data Driven Testing
OK – show me a picture!!

User
Uses Creates

Keyword Keyword Map

Keyword Test
Script
Test

Framework Keyword Test Libraries to

Parser drive keywords

Test Application Automated

Automation Object map Test Tool

Application AUT

15
The Basics – Keyword Driven
testing

ƒ Keyword Driven Testing provides a mechanism for mapping

an Application Under Test (AUT) to a set of defined and well
understood keywords.
ƒ Allows testing of an application by interfacing only with the
keywords.
ƒ Underlying Automated tool at a different level of abstraction
ƒ AUT is at a different level of Abstraction

ƒ Division of labor between Application Experts (BA’s, SME’s, etc) and

test automation experts

ƒ Creation of modular, reusable test components that are built by test

architects and then assembled into test scripts by test designers
Why Keyword Driven Testing?

ƒ Provides Clarity

ƒ Teamwork and
optimization of
resources
ƒ Working Smarter not Harder
Agenda

ƒ Identifying the problem

ƒ The Basics
ƒ Testing Frameworks – how and why?
ƒ GUI Automation 101
ƒ Making Automation tool Better
ƒ IBM Example
ƒ Solutions in the industry

ƒ How will Keywords help me

ƒ Pro and Cons of Keyword testing
ƒ Assessing Automation Success
ƒ Metrics
What To Automate

ƒTests that need repeat execution

ƒTests that cover existing, stable
functionality
ƒTests that represent simple designs
ƒTests that offer numerous opportunities
for data input variation
 GUI Automation 101
ƒ Important Things to Know
ƒ Test Automation is a SOFTWARE PROJECT
ƒ Architect, plan and develop your code for the long term
ƒ Writing good automation is challenging but worth it
ƒ GUI automation is a special skill set and a career niche!
ƒ Automation is a long-term investment with extended ROI
ƒ It WILL take longer to create an automated test case than a manual test case
ƒ It typically takes 3 release cycles to see substantial gains from automating a
test case
ƒ Automation needs buy-in and help from development
ƒ Automation blocking defects need to be fixed, even if customers won’t see
them
ƒ Knowing about application changes that break automation in advance
reduces fire drills for the test execution community
ƒ Having developers run automation prior to check-in really improves code
stability
ƒ Automation ensures today’s build works like yesterday’s and can
allow late changes to the product
 GUI Automation 101
ƒ Important Things to Know
ƒ Applications Under Test WILL change.
ƒ Your automation needs to be architected for maintenance
ƒ Separate what your application does from how it does it
ƒ Keep the ‘GUI’ definitions in a single place
ƒ Understand your application, besides GUI changes, what else ??
ƒ Know that GVT testing is coming – externalize strings for lookups
and verifications or build the support in right away
ƒ Record/playback is impossible to maintain release to release
ƒ Isolate GUI ‘tasks’ as functions or classes that work on a small
portion of your application
ƒ Dynamic Verification reduces maintenance: Use your test data
to figure out what to verify.
ƒ Naming conventions and package structure can really help as
your project gets more complex
ƒ Define your development practices “Rules of the Road” (ROTR)
Testing Frameworks – why are
they needed?
ƒ Framework eliminate the need for
mundane/monotonous tasks to be
done manually

ƒ Shared test among projects that

don’t seem to have anything to do
with each other.

ƒ You likely have one (by a different

name) if you do Test Automation
IBM Internal Framework Benefits
ƒ Out-of-the-box business-logic based framework on top of RFT
ƒ Reduces total cost of ownership for mixed teams that want to develop a
framework that encapsulates their business logic

ƒ Jumpstart Agile and/or large automation projects

ƒ Simplifies coding tasks with easy-to-read library code and standardized
implementation
ƒ Re-useable code for multi-interface applications (Web, Eclipse & API)
ƒ Business Logic reduces design work; integrates with existing RFT Keywords solution

ƒ End-user validated
ƒ IBM internal users are continually adopting, enhancing and building new features
for the frameworkcode base

23
Framework Methodology
ƒ Build automation focused on the application’s business logic

ƒ Can be used with keywords

ƒ Factory implementation can support multiple client interfaces with

maximum code re-use (graphic on next slide)
ƒ Even supports testing API, CLI

ƒ Provides easy to read test cases

ƒ Allows non-programmers to write test cases
ƒ Keywords/methods are meaningful to Domain Experts

24
Framework Architecture

Re-useable code
for all clients

25
WAIT -- Keyword Automation

ƒ When “Not” To Use Keyword

Automation
ƒ Fringe Activities where reuse potential is
low
ƒ Complex Tasks where other frameworks
and/or approaches fit better
(e.g. Testing API Interfaces, etc …)
ƒ Lack of dedicated automation resources
Keyword/Framework Solutions
Why SAFS QTP/BPT Certify RQM/RFT Homegrown

The Real Point for “Keyword Driven Testing” is at “Why”, not at “What-is”

“Keyword” Context (in Keyword Driven Testing) has been created for around 10 years,
working as:
Standardized Interface in Native language Across Different Contexts (such as, manual tester’s,
automation developer’s),
Mapping to a unique, reusable, “Function” in programming
Able to Share Across Teams/Projects, SUTs

„ Why Keyword?
Aim to Deliver an Effective Test Automation Solution

Seamlessly Integrate Expertise on All Ends of the whole lifecycle, including

“High-level ¾ From Business Analyst (Expertise on User Scenarios)
Component” ¾ From Manual Tester (Expertise on Product & Environment)
¾ From Automation Developer (Expertise on Scripting/Programming)
“Keyword”
Keyword/Framework Solutions
Why SAFS QTP/BPT Certify RQM/RFT Homegrown

ƒ First, Create .Map file. This is a set of keywords and how they map
to your Functional tester Objects
Keyword/Framework Solutions
Why SAFS QTP/BPT Certify RQM/RFT Homegrown

ƒ Step Table (.SDD Files) tells functional tester what to do for each step as it
reads the application map file. It is the Test
ƒ Suite (.STD files) group Step files together
Keyword/Framework Solutions
Why SAFS QTP/BPT
ƒ Advantages
¾ Enables non-technical
testers create tests using an
application such as notepad
based on keywords defined
in the map file.
¾ Tests can be created quickly
and easily
¾ No knowledge of underlying
automation framework is
needed
Certify RQM/RFT Homegrown
ƒ Disadvantages
¾ A lot of setup required for
mapping keywords
¾ The framework requires
customization for each
automation testing tool
¾ Not very fancy, have to
maintain things in text format.
¾ Lot of work needs to be done
external to the tool
¾ Keywords need to be mapped
to objects in the automation
tool
Keyword/Framework Solutions
Why SAFS QTP/BPT Certify RQM/RFT Homegrown

Mercury EULA
You may not...provide externally or to third
parties any oral or written communication
that describes or summarizes the features,
functions or performance characteristics of
the Licensed Program and Documentation,
or that compares the Licensed Program
with any other similar product of Yours or
any third party;
Keyword/Framework Solutions
Why SAFS QTP/BPT Certify RQM/RFT Homegrown

ƒ 3 components to Worksoft Certify

¾ GUI Client to plan, design, execute and maintain tests
¾ Central DB server for maintaining all assets
¾ Function Library (e.g. Framework)
ƒ Manual Tester/End User creates a test from an application
map
¾ Application Map Created
¾ Users specifies the object and operation from the map and any data
needed for a sequential flow through the application
¾ Specify actions that can happen on each step (e.g. On failure
continue executing, end test, Jump to another step, etc)
¾ If objects in application map changed test steps using changed
objects are flagged
¾ Test results log maintained in DB
Keyword/Framework Solutions
Why SAFS QTP/BPT Certify RQM/RFT Homegrown

ƒ 3 components to Rational Quality Manager Solution

¾ Manual test tool used to creation and execution of manual tests leveraging
Keywords
¾ Automation tool for creation and execution of scripts to tie to keywords
¾ Keywords that represent application Business Logic
ƒ Manual Tester/End User creates a Manual test
¾ Keywords defined in RMT
¾ Framework (Automation scripts) built in RFT and tied to keyword
¾ Based on what is scoped by keyword definition
¾ Application logic captured here
¾ Users build Manual tests leveraging keywords
¾ Specify Data to use in the Manual test Script for variableized content
Keyword/Framework Solutions
Why SAFS QTP/BPT MTP v3.5 RQM/RFT Homegrown

The Real Point for “Keyword Driven Testing” is at “Why”, not at “What-is”

One Customer Example

ƒ Goal
¾ Make Scripts simpler to read and maintain
¾ Want scripts to only look like Manual tests
¾ Improve ROI
ƒ Develop a manual test
ƒ Create external object maps maintained in CSV file.
¾ Consists of Object, Action, Value. (Use test Object Inspector to find object names)
¾ Assumption is that object names don’t change
ƒ Hidden properties of Manual Test Step to designate the object, action, value on each step
ƒ Execute Automated Test where all testcases execute same generic automation script
ƒ Have a Automation Framework
¾ Parses each line of manual test and extracts the object and operation desired
¾ Calls library files passing the object and operation on the object to drive application.
¾ Manual test tool used to creation and execution of manual tests leveraging Keywords
¾ Automation tool for creation and execution of scripts to tie to keywords
¾ Keywords that represent application Business Logic
Commonalities across solutions
ƒ All solutions require creating a map of the AUT
ƒ Some work with an application map directly
ƒ Others map Keywords to the Test Tool objects and/or Transactions
ƒ Test Tool Objects still mapped to AUT objects

ƒ Coding is required on the back end, especially for custom controls

ƒ Application Experts need to know the Keywords and types of

interactions that can occur on the Keywords
ƒ Even if a keyword is mapped to the object, customers need to
know the keyword and operation on the keyword

ƒ Faster ROI is a primary driver

ƒ Testers want to justify their purchases and existence

ƒ Primary outreach is for the Business Analysts or Non-technical Tester

ƒ Want to utilize the work being done by the BA’s and make tools
easy enough for all types of users
Need A balanced approach
Meet the needs of the business in a collaborate
manner

Streamline Manual Testing

ƒ Create living self-maintaining documentation
ƒ Reduce dependency on subject matter experts
ƒ Minimize inconsistencies of manual testing

Business Relevant Automated Testing

ƒ Flexible and scalable framework
ƒ Demonstrate value quickly

Agility To Meet Business Needs

ƒ Strategically utilize staff and the business
ƒ Maximize investment in manual test assets
ƒ Minimize automation maintenance over time
ƒ Keep the entire test team on the same page
Agenda

ƒ Identifying the problem

ƒ The Basics
ƒ Testing Frameworks – how and why?
ƒ Solutions in the industry

ƒ How will Keywords help me

ƒ Pro and Cons of Keyword testing
ƒ Assessing Automation Success
ƒ Metrics
Benefits of Keyword Testing

ƒTesting done earlier in lifecycle

ƒ Define Tests before application stabilized
ƒ Increased ROI on Testing Tools
ƒ Create Tests from specifications earlier in the Software LifeCycle

ƒBridge to Test Automation

ƒ Use Application experts (Business Analysts, SME’s) for testing
ƒ Use common terms to define test cases and scenarios earlier in the process
ƒ 80% testing done manual – optimize that investment

Manual Tests Steps tied Keywords Test Automation

to realized by
keywords automation

3838
Keyword Testing Goals
ƒCreate manual tests
easily
ƒAutomate pieces of a
test
ƒ Have “hybrid” manual-and-
automated scripts

ƒAssemble the pieces into

new scripts
ƒ Create fully automated scripts
without coding

ƒLet an automation
expert automate the
pieces
Keyword Automation within Agile Processes
ƒ Release Engineering/Build Processes
ƒ Automation is considered a best practice
ƒ Quicker detection of regressions with each
build

ƒ Functional Test
ƒ Automation results can allow development
teams
to react faster to defects and fulfill the agile
vision

ƒ Pitfalls
ƒ Automation is very difficult with 1.0 Projects
ƒ Higher level of code churn which increases
pressure on upfront investment
Metrics to collect
ƒ Things to understand to better evaluate our ROI in
automation:
ƒ How Long
ƒ How long it takes to run the tests manually
ƒ How long it takes to analyze the test results (ran
manually and ran automated)
ƒ How long it takes an unmanned machine to run the
automated suite or tests.
ƒ How Many
ƒ How many tests we plan to automate (an actual count)
ƒ How many tests we actually have automated -- at given
intervals
ƒ How many resources involved in running the tests
manually (including setup, clean-up and test analysis)
ƒ How many resources involved in running the automated
tests (including setup, clean-up and test analysis)

Note:

Automation isn't always the best answer. These metrics will help determine the best course of action.
Key Solution Drivers
9 Ease of Use
• Utilize Manual testing for Business Analysts – easy to use shell

9 Little to no programming
• (Automated Scripts recorded, or created by test automation engineers)

9 Represent tests to different roles in format they are most comfortable with (BA’s
see tests in Natural Language)
• High level manual tests and keyword view

9 Faster creation of a robust, repeatable and low maintenance test

• Keyword mapping and automated script generation

9 Ability to create tests without an application – create from specifications earlier

in the Software LifeCycle
• Keyword Mapping to objects and headless components

9 Unattended playback
• Automated test associated once manual test has been keyword enabled

9 Quicker Return on Investment for customers buying our solution

• Creation of tests earlier and faster

9 Reusability
• Keywords reused and metrics collected
Keyword Automation – Summary

ƒ Not a silver bullet for all testing

ƒ Requires dedicated resources

ƒ Keyword Automation Frameworks integrated
within the tooling help reduce the upfront
burden

ƒ Certain types of testing have an affinity

for certain types of automation

ƒ High levels of keyword re-use and

incremental test coverage expansion
indicate increasing benefit from keyword
framework.
References
ƒ Nagle, C. “Test automation frameworks”; 1999

ƒ Hayes, L. "Establishing an Automated Testing Framework"

Tutorial presented at STAR’99East Conference, Orlando,
Florida, May 11, 1999.

ƒ Nagle, C. “Automation Framework Support”, May 2005

ƒ Cem Kaner, http://www.kaner.com

ƒ Zambelich, K. Totally Data-Driven Automated Testing 1998

http://www.sqa-test.com/w_paper1.html
 Thank you!

For more information:

Brian Massey
Senior Product Manager
IBM/Rational Software
678-248-4523
jbmassey@us.ibm.com
www.ibm.com/rational