1

Experience Driven Test Automation

Mark Fewster mark@grove.co.uk www.grove.co.uk

Dorothy Graham info@dorothygraham.co.uk www.DorothyGraham.co.uk

Contents Introduction Management issues

manager support setting appropriate goals success on a large scale

Technical issues
testware architecture life cycle for automated tests standards quality of scripts test effectiveness
2

BCS SIGiST, September 2011

Grove Consultants

New book: Experiences of Test Automation

29 case studies + 14 anecdotes collected, commented on and edited by Dot Graham and Mark Fewster to be published in 2011 by Addison Wesley contributions from a dozen countries applications from finance to space, IT to embedded both traditional and agile development Model-Based Testing, cloud, exploratory automation, tool-supported manual testing, test monkeys learn from others experiences
3

Contents Introduction Management issues

manager support setting appropriate goals success on a large scale

Technical issues
testware architecture life cycle for automated tests standards quality of scripts test effectiveness
4

BCS SIGiST, September 2011

Grove Consultants

Manager support can make or break automation projects need to understand automation capability & limits
building good automation takes time and effort and pays the greatest dividends building poor automation can be quick and cheap but costs outweigh benefits longer term

need to manage automation projects

not every test can or should be automated only automate good tests recognise automation and testing as different skills
5

Interesting manager stories the unbelievable case of propaganda

already advertised 100% automated testing successful automation cancelled by stakeholders Think of it my way

the mysterious case of death by stakeholders

the classic case of inappropriate demarcation

Bo Roop: automating testing of automation tool technical challenges particularly in maintenance manager: you write tests, others run & maintain but others couldnt maintain, tests abandoned
6

BCS SIGiST, September 2011

Grove Consultants

Interesting manager stories the unusual case of too many tools

1st project success with non GUI application 2nd project new tool, GUI application, no dev. co-op. 3rd project different tool, framework/libraries 4th project another tool, Python framework 5th project new tool version, no to framework now automation deliverable for developers!

the unsurprising case of spare time work

5 people part time no goals, no time to coordinate, re-invention, not publicised lasted 6 months
7

Setting appropriate goals the refreshing case of a clear goal

problems after release, want improved reliability a few people, basic execution & check log automated tests found bugs but delayed the release to fix them management reaction?

and another
Lars Wahlberg: a senior manager set clear goals (this helped) provided adequate funds, monitored closely 50,000 new tests in 3 years, reusable for all projects
8

BCS SIGiST, September 2011

Grove Consultants

Automation objective: find lots of bugs? tests find bugs, not automation automation is a mechanism for running tests the bug-finding ability of a test is largely unaffected by the manner in which it is executed* this can be a dangerous objective
particularly for regression automation!
Manual Scripted 507 (24%) Exploratory Testing 1229 (58%) Automated Tests 197 (9%) Bug Fix Verification 178 (8%)

Newman and Allen, CRM system

9

Success on a large scale MBT @ ESA: Stefan Mohacsi, Armin Beer

home-grown tool interfaced to commercial tools Model-Based Testing and Test Case Generation layers of abstraction for maintainability define model before software is ready capture and assign GUI objects later developers build in testability ROI calculations invest 460 hours in automation infrastructure break-even after 4 test cycles
10

BCS SIGiST, September 2011

Grove Consultants

Database testing: Henri van de Scheur

tool developed in-house (now open source) agreed requirements up front 9 months, 4 developers in Java (right people) good architecture, started with quick wins
flexible config., good reporting, metrics used to improve

results: 2400 times more efficient from: 20 people, 40 tests, 6 platforms in 4 days to: 1 person, 200 tests, 10 platforms in 1 day quick dev tests, nightly regression, release tests life cycle of automated tests low maintenance, machine use 24x7, better quality
11

Contents Introduction Management issues

manager support setting appropriate goals success on a large scale

Technical issues
testware architecture life cycle for automated tests standards quality of scripts test effectiveness
12

BCS SIGiST, September 2011

Grove Consultants

Testware architecture
Testers

write tests (in DSTL)

testware architecture (test framework)

High Level Keywords

abstraction here = easier to write automated tests = widely used

structured testware
Structured Scripts

Test Automator(s)

abstraction here = easier to change tools and maintain = long life

Test Execution Tool

runs scripts
13

Implementing testware architecture tool will have its own way of organising tests
generic solution may not suit will lock you in to that tool

a better way (gives independence from tools)

organise your tests to suit you, e.g. use pre-processing to copy files to where the tool expects to find them use post-processing to copy result files to where you want them to be it is about making the testers environment ergonomic to work efficiently with minimal mistakes
14

BCS SIGiST, September 2011

Grove Consultants

Designing testware architecture anecdote

Molly Mahai, Test Manager set up good architecture? books not enough picked something to get started after a while, realised limitations too many projects, library cumbersome re-designed architecture, moved things around we didnt know what was needed until we experienced the problems for ourselves redesign quickly small impact learn on the fly versus pilot
15

Life cycle for automated tests case study

Henri van de Scheur, testing databases, big success 5 year span, in-house tool, now open source <http://kenai.com/projects/jet>

life cycle
Suspend

Development

Candidate

Active

Retired

Deleted

different types of test suite MATS, Daily, Weekly, Non-Functional

16

BCS SIGiST, September 2011

Grove Consultants

Standards case studies

Mecke, et al, major SAP application, >3 MLOC (auto) large government agency, 12+ automators, 100s developers/testers

coding standards
readable and modular to encourage reuse and minimise maintenance effort

document standards
automated test specifications and reports

process standards
e.g. automated test development, execution, and maintenance
17

Quality of scripts case study and anecdote

Mecke, et al, major SAP application, >3 MLOC (auto) Randy Rice, large organisation, big push to automate

reviews
review by automation expert useful (copied from dev.) daily technical reviews -> peer reviews -> forum

static analysis on automation code

in-house static analysis tool, routine checks e.g. reuse issues, attribute issues, naming conventions, zombie scripts continually gather new ideas for checks
18

BCS SIGiST, September 2011

Grove Consultants

10

Effectiveness of automated tests anecdotes

Julian Harty various experience reports

inherited API tests slow to run, days to update

poor structure, only thing tested for: run-time exceptions, not does it do what it should deleted half, re-wrote rest based on unit tests

networked tests all but one passed

it was the only one that actually connected to the server the only one to indicate the truth! others said pass if there was no error message

consider whether to improve or discard such tests

19

Summary: some key points management support and understanding is essential for long-term success
appropriate objectives and resource, measure ROI

a few technical issues

good testware architecture helps life cycle for automated tests & standards quality of scripts & test effectiveness

case studies provide many insights and ideas

take advantage of them and improve your own chances of success
20

BCS SIGiST, September 2011

Grove Consultants