Technical Test Automation Challenges PDF

TF
Half-‐day Tutorials
8:30 AM

Technical Test Automation Challenges:

Patterns and Solutions

Presented by:

Seretta Gamba

Freelancer

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐-‐ 268-‐-‐ 8770 ·∙·∙ 904-‐-‐ 278-‐-‐ 0524 -‐ info@techwell.com -‐ http://www.starwest.techwell.com/

Seretta Gamba

Seretta Gamba has forty years’ experience in development and fifteen in test
automation. After going through all the usual developer roles, in 2001 she was put
in charge of test automation for her current company. She developed a framework
that enabled her company to quickly get excellent results. After having talked about
the framework at a couple of conferences, she met Dorothy Graham and was invited
to write a chapter in the book Experiences of Test Automation. With Dorothy she
has been developing the Test Automation Patterns Wiki and is now writing a story
book about the patterns.

Technical Test Automation Challenges: Patterns and Solutions
Technical Test Automation

Challenges: Patterns and
Solutions
Prepared by
Dorothy Graham &
Seretta Gamba
Presented by
Seretta Gamba
srttgmb@yahoo.com
TestAutomationPatterns.org
1
© Seretta Gamba and Dorothy Graham 2018
Tutorial objectives
 understand some technical issues in test
automation
 be aware of the issues and patterns in the Test
Automation Patterns Wiki
 identify patterns that could help solve technical
issues
 exercise using personas and the wiki
 identify your own issues and patterns, share & discuss
 explore automation solutions (more than in wiki)
 be able to use the wiki for further solutions (after
the tutorial)
NOTE: system level automation, not unit level
NOTE: test automation patterns, not test patterns 2
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
About you
– show of hands
• job role (tester, test manager, automator, other)
• industry sector
– finance / government / embedded / software products / retail /
mobile apps / others
• automation experience
– none / < 6 months / 6 – 24 months / 2 – 5 yrs / > 5 yrs
• tools using / used (major ones)
– brought a laptop or tablet?
About you
– other people here today
• may have similar problems to you
• may use the same tools or tools you want to know about
• may have ideas to help you
• you may have knowledge that will help them
– introduce yourself to your neighbour / partner
• your name
• why are you here? your current problems / issues /
things you want to find out about
• company you work for
• your experience in (system-level) test automation
• tools you use(d)
4
Issues (problems)
 what problems affect you in test
automation?
 what causes the most trouble?
 what takes longer than it should?
 what is holding you back in your
automation?
 what are you struggling with?
 what would you most like to improve?
 (technical or management issues)
5
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 6
Origin of Test Automation Patterns
 Experiences book published 2012

 Seretta contributed Ch 21
 Automation through the back door,
by supporting manual testing
 she is a developer, used to
using patterns
 when she read the other chapters,
she thought: “Patterns! Patterns!”
 she wrote up test automation patterns
as a book & asked Dot to join her
 Word didn’t work very well,
so we made a wiki
7
Test automation patterns wiki
 purpose: share information, ideas and

experiences about test automation, using
issues and patterns
 mind-maps summarize issues and patterns
(not clickable as links)
 issues and patterns are classified into sub-
sections (e.g. management, process, etc.)
 wiki started 28 January 2013

 Changed host in 2018
Test Automation Issues

 an Issue is a problem that testers and/or
test automators encounter when trying to
do test automation
 they are things that take longer than they
should, that hold you back in what you
want to automate, or that are just too
much trouble
 an Issue can also be a task that you must
complete when doing test automation
 Issues are written in ITALICS CAPS
 ( but not in the wiki!) 9
Test Automation Issues (>50)

 Process Issues: problems that occur when the
test automation process has not yet reached the
necessary maturity
 Management Issues: problems that occur when
management has not given the necessary support
(budget, resources, team)
 Design Issues: problems that occur when an
efficient testware architecture and maintainability
are not built in from the very beginning
 Execution Issues: problems that occur when the
automated tests are run (unpredictable results etc)
10
Test Automation Patterns

 generally reusable solution to a commonly
occurring problem within a given context
 a way of solving an issue or problem in test
automation that has worked in practice for
many people
 a pattern is not prescriptive or a set of
steps, but rather ideas to consider to see if
they would apply in your situation or context
 Patterns are written in CAPS

11
Test Automation Patterns (>80)

 Process Patterns: how the test automation
process should be set up or how it can be
improved
 Management Patterns: how to manage test
automation both as an autonomous project or
integrated in the development process
 Design Patterns: how to design the test
automation testware so that it will be efficient and
easy to maintain
 Execution Patterns: how to take care that test
execution is easy and reliable
12
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 13

Purpose:
 share information, ideas and
experiences about test
automation, using issues and
patterns
 give a space to learn, give and
get feedback
testautomationpatterns.org 14
15
Exercise: Issues (1)

 in the Test Automation Patterns Wiki
go to Exercises
 open Scenario 5
 Work out Exercise 1
The following exercises are used on tutorials
and workshops. You are welcome to try these
as well, but solutions are only given in the
tutorial or workshop.
Scenario 1: Owen Optimistic

Scenario 2: Ivy Independent
Scenario 3: Ursula Unfocused
Scenario 4: Ivan Indispensible
Scenario 5: Nancy Naive
Scenario 6: Charlie Chainer
Scenario 7: Ted Toolswitcher
Scenario 8: Freddie Frustrated 16

 which issue(s) does Nancy have?
 technical or management issues?
17
Navigating the issues

 finding out which issue is troubling our test
automation isn’t easy
 finding the correct pattern to apply without
really knowing the issue is just as difficult
 the Wiki offers a way to identify your

specific issues quickly
 “Diagnostic”
18
Diagnostic? What's that?

 we want to identify the most
pressing automation
problems (issues) by asking
specific questions (just like a
doctor would do)
 if the issue is known, it can
be solved with the pattern
approach
 afterwards you can address
the next most pressing
problem 19
How to use the Diagnostic

1. in the Diagnostic page you will be shown some
general questions
2. select the answer that best describes the most
pressing problem in your test automation.
3. the system shows the next, more detailed question
or the issue that currently causes the most pain
4. repeat steps 2. and 3. until you reach the specific
issue that needs to be solved first; it will suggest
the patterns needed to solve it
5. select the pattern(s) that best fit your context and
apply it/them
6. if you still have problems, start again from step 1
20

 please go back to
Nancy Naive
21

 which issues did you find?
 did you find the same ones as before?

 was it easier or more difficult?
22
Issues found in exercises
• this exercise is based on a common

problem
– BRITTLE SCRIPTS
• we will look at patterns to address this
23
BRITTLE SCRIPTS
• small changes to the application cause
large changes in the automation
– high maintenance cost
• Patterns:
– TESTWARE ARCHITECTURE
– ABSTRACTION LEVELS
– MAINTAINABLE TESTWARE
– MANAGEMENT SUPPORT
– GOOD PROGRAMMING PRACTICES
– COMPARISON DESIGN
24
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 25
 Organisation of, and relationship
between, artefacts
 scripts, input, test data, test descriptions,
expected results, actual results, log files,
etc.
 When you create an automated test:
 do you know what already exists that you
can re-use?
 do you know where to put the artefacts?
(test materials)
 do you know what names to use?
26
 Organisation of, and relationship
between, artefacts
 scripts, input, test data, test descriptions,
expected results, actual results, log files,
etc.
 When you execute an automated test:
 do you know where to find all the test
results?
 and why they should be stored in a different
way to the test
 do you know how to analyse a test failure?
27
LOCALISED REGIMES
• “everyone will do the sensible thing”

– most will do something sensible, but different
• “use the tool however it best suits you”
– ignores cost of learning how best to automate
• problems include:
– effort wasted repeatedly solving the same
problems in different ways
– no re-use between teams
– multiple learning curves
28
Easy way: use tool’s architecture

• tool will have its own way of organizing tests
– where to put things (for the convenience of the
tool!)
– will “lock you in” to that tool – good for vendors!
• a better way (gives independence from
tools)
– organize your tests to suit you
– as part of pre-processing, copy files to where
the tool needs (expects) to find them
– as part of post-processing, copy back to where
you want things to live
29
ABSTRACTION LEVELS
• most critical factor for success
– worst: close ties between scripts, tool & tester
• separate testers’ view from technical aspects
– so testers don’t need tool knowledge
• for widespread use of automation
• scripting techniques address this
• separate tests from the tool – modular design
– likely changes confined to one / few module(s)
– re-use of automation functions
– for minimal maintenance and long-lived
automation 30
ABSTRACTION LEVELS
Data layer:
databases etc.
DB specialists
Technical layer: Environment layer:

scripts, etc. Browsers etc.
Test automators (Web) specialists 31
ABSTRACTION LEVELS
Testers:
Test cases
32
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 33
DATA-DRIVEN TESTING
• test data extracted from scripts
– placed into separate data files
• control scripts read data from data files
– one script implements several tests by reading
different data files (reduces script maintenance
per test)
• reduced build cost
– faster and easier to automate similar test
procedures
– many test variations using different data
• multiple control scripts required
– one for each type of test (with varying data)
34
Data-driven example
Control script
For each TESTCASE
Data file: TestCase1 OpenDataFile(TESTCASEn)
ReadDataFile(RECORD)
FILE ADD MOVE DELETE
For each record
countries
Sweden ReadDataFile(RECORD)
Data file: TestCase2 Case (Column(RECORD))
USA
FILE ADD MOVE DELETE FILE:
4,1 OpenFile(INPUTFILE)
Europe
Norway
France ADD:
2 AddItem(ITEM)
Germany
7 MOVE:
1,3
MoveItem(FROM, TO)
2,2
DELETE:
1 DeleteItem(ITEM)
5,3 …..
Next record
Next TESTCASE 35
Data-driven: example data file
Data file: TestCase1

FILE ADD MOVE DELETE
countries Headings indicate actions
- but may not be read by a
Sweden
control script (they could be
USA just “comments”)
4,1
Norway
2 Position represents data for the
7 action (data in the wrong
column will cause problems)
Tests are not identical – first one has 3 countries, 2 moves,

next test had 2 countries, 3 moves.
The test data drives the test for things it can deal with –
the script responds to data in a fixed position in the data file.
36
About keywords
• single control script (Interactive Test Environment)
– improvements to this benefit all tests (ROI)
– extracts high-level instructions from scripts
• ‘Test Definition’
– independent of tool scripting language
– a language tailored to testers’ requirements
Unit test: calculate one
• software design
interest payment
• application domain System test: summarise
• business processes interest for one customer
• more tests, fewer scripts Acceptance test: end of day

run, all interest payments
37
Comparison of data files
data-driven approach keyword approach

FILE ADD MOVE DELETE SAVE
ScribbleOpen Europe
Europe
France AddToList France Italy
Italy MoveItem 1 to 3
1,3 MoveItem 2 to 2
2,2
DeleteItem 1
1
5,2 MoveItem 5 to 2
Test2 SaveAs Test2
what happens when

which is easier to the test becomes this looks more like
read/understand? large and complex? a test
38
Exercise: Patterns (1)

go to Exercises
 open Scenario 4


 which Patterns would help Ivan?
 technical or management patterns?
40

 please go back to
Ivan Indispensable
41

 what is Ivan’s main Issue?
 which Pattern(s) did you find?
 did you find the same ones as before?

 was it easier or more difficult?
 which was most helpful, the Diagnostic or
just looking through the issues / patterns? 42
• the exercises are based on common

problems
– BRITTLE SCRIPTS
– INADEQUATE DOCUMENTATION
• we will look at patterns to address some of

these
43
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 44

• document the automation scripts and the test
data, so that they are:
– easily found
– understandable
– reusable
– traceable (e.g. to requirements, user stories)
• implementation
– SET STANDARDS
– descriptive names, naming conventions
– standard template (Test Definition)
– configuration management with SUT
– let “newbies” update documentation
45
advantages of a standard approach

 you can reuse scripts or data if you know
how they work, what they are and where
they are!
 it’s easier to maintain stuff when you know
how it works
 can automate lots of peripheral tasks
 new colleagues have an easier time
learning
This saves time and effort!

46
Test definition
(for every automated test)
• name of test • measures
• purpose of test – expected run time (of the
automated test)
• materials/artifacts used – EMTE (equivalent
– scripts (shared & unique) manual test effort)
– expected results – others?
– data files
• test selector tags
• results produced – to run tests or groups of
• set-up instructions* tests
• execution instructions* – examples: smoke tests,
short tests, bug fix tests.
• tear-down instructions* long tests, specific
• related tests environment tests
*lists of relevant keywords / test actions 47
General control script
For each test to be executed

Read test definition file
Verify test definition file
Check specified materials exist using relevant
Execute setup keywords, as
Execute test steps specified in test
Perform post-execution comparison definition file
Check specified results exist
Execute teardown
Report results
EndFor
48
Exercise
go to Exercises
 open Scenario 3
 Work out all three exercises

Exercise
 which Issue(s) did you find that
Ursula has?
 what search methods did you use?

 which was most useful?
 what Pattern(s) would help Ursula?
50
• the exercises are based on common

problems
– BRITTLE SCRIPTS
– INADEQUATE DOCUMENTATION
– UNFOCUSED AUTOMATION
• we will look at patterns to address some of
these
51
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 52
• tests that are worth automating

• not just regression tests
– set-up for tests
– data creation or manipulation
– repetitive tasks
• implementation
– AUTOMATE GOOD TESTS
– KNOW WHEN TO STOP
– SHARE INFORMATION
See Experiences section for useful tips from

Joachim Van Dorpe and Gaby Spengler 53
AUTOMATE GOOD TESTS

• good candidates for automation:
– smoke tests
– tests of critical areas
– regression tests
– same tests for different
environments/browsers/devices
– complex tests
– tests requiring high precision
– anything repetitive, time-consuming,
boring 54
Angie Jones’ evaluation worksheet

– gut feel (to compare later) vs:
• customer risk
– impact on customer & frequency of use
• test value
– distinctiveness of the test & likelihood a bug
will be fixed
• cost-efficiency
– time to script test & complexity of test
• history
– similarity to weak areas & frequency of broken
before
– calculate and compare to gut feel!
Thanks to Angie Jones: angiejones.tech
There is a spreadsheet for this – email Dot for a copy 55
Angie Jones’ evaluation worksheet

• each factor between 1 and 5, the two
multiplied to give a value between 1 and 25
for each of the four areas
• total number between 4 and 100
– 75 – 100: automate
– 26 – 74: possibly automate
– 4 – 25: don’t automate
• adapt to your situation
– different factors, different weights
Thanks to Angie Jones: angiejones.tech

There is a spreadsheet for this – email Dot for a copy 56
KNOW WHEN TO STOP

• not all tests should be automated
– better as manual tests:
• user interface / usability
• tests not run often
• tests for very unstable system
• tests that take too long to automate (not
worth it)
– prioritise the tests to be automated
• then whenever you stop, you have
automated the ones most worth automating
57
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 58
• make each automated test self-contained

– can be run on its own
– can be run in any order
– leave the system and tool ready for next test
– patterns include:
• FRESH SETUP (data etc needed for the test)
• ONE CLEAR PURPOSE
• PRIORITIZE TESTS
• PARALLELIZE TESTS
59
Automated tests vs manual tests

• manual tests
– optimised for human beings
– what the brain needs to remember
– don’t want to wait for long setup, better to group
tests
– longer tests, can work around or fix “on the fly”
• automated tests
– optimised for the tool / computer to execute
– no one to sort out problems the tool can’t deal
with
– a failing test blocks everything after that point
– shorter tests (and automated recoveries) give you
more testing
60
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 61
TOOL INDEPENDENCE
• Separate the technical implementation
of the tests from tool-specific elements
– modular scripts
– keep tool-specific parts as small as
possible
• and self-contained to minimise change
– patterns include:
• TEST AUTOMATION FRAMEWORK
• OBJECT MAP
• GOOD PROGRAMMING PRATICES
62
Tool independence – why bother?
• your current tool is ideal, right?

• over the next 2 – 5 – 10 years, changes in
– tool development / tool vendors
– your applications
– devices / platforms to test
– variations for different customers
• a bit of additional structuring now will
save you lots of time in the future
63
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 64
Comparison types compared
scribble1.scp dynamic comparison

Test script: Error
-test input message
log.txt
as expected?
-comparison
instructions Log
countries.dcm
Initial Scribble countries2.dcm

Document Edited
countries2.dcm
Document
Expected Compare diffs.txt
Output post-execution Differences

65
comparison
COMPARISON DESIGN
Test sensitivity
• the more data there is available:
– the easier it is to analyse faults and debug
• the more data that is compared:
– the more sensitive the test
• the more sensitive a test:
– the more likely it is to fail
– (this can be both a good and bad thing)
66
Sensitive vs specific (robust) test
test is supposed specific test

to change only verifies this
this field field only
test
outcome
unexpected sensitive test

change occurs verifies the
entire outcome
67
Too much sensitivity = redundancy
three tests, if all tests are

each changes specific, the
a different field unexpected
change is
missed
test
outcome
unexpected if all tests are

change occurs sensitive, they
for every test all show the
unexpected change 68
Using test sensitivity

• sensitive tests (SENSITIVE COMPARE):
– few, at high level
– breadth / sanity checking tests
– good for regression / maintenance
• specific/robust tests (SPECIFIC COMPARE):
– many, at detailed level
– focus on specific aspects A good test
automation strategy
– good for development will plan
a combination of
sensitive and
specific tests
69
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 70
Test status – pass or fail?

• tool cannot judge pass or fail
– only “match” or “no match”
– assumption: expected results are correct
• when a test fails (i.e. the software fails)
– need to analyse the failure
• true failure? write up bug report
• test fault? fix the test (e.g. expected result)
• known bug or failure affecting many automated
tests?
– this can eat a lot of time in automated testing
– solution: additional test statuses
71
Known bug
• minor bug in all test results known bug

• alternatives:
– add bug to expected results
• bug preservation
– ignore test results / don’t run
• untested - miss other things
– analyse every result
• waste of time
• a better way - new test
status: expected fail
72
Compare to No differences found Differences found

(true) expected outcome Pass Fail
expected fail outcome Expected Fail Unknown
don’t know / missing Unknown Unknown
• other possible additional test statuses

– test blocked
– environment problem (e.g. network down,
timeouts)
– set-up problems (files missing)
– test needs to be changed but not done yet
– different version of expected results
73
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TOOL INDEPENDENCE
COMPARISON DESIGN
Conclusion 74
Applying the patterns
• time to explore the wiki for any issues that

you have (covered today or not) and/or
patterns you want to look at
• work with a colleague or partner
• ask questions about the issues or patterns
• how can you use the wiki to help?
75
Discussion
• any [technical] issue you want to discuss
• any [technical] pattern you want to discuss
• anything from the game? ;-)
76
Summary
• we looked at technical issues
– different ways of getting into the wiki
• issues, patterns, diagnostics
• we looked at technical patterns
• and additional information
• I hope this is / will be useful for you
• please help to improve the wiki by
adding your own comments &
experiences
77
Finally
any more questions?
please email me!
srttgmb@yahoo.com
Thank you for coming today

All the best in your automation!
78

Technical Test Automation Challenges PDF

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Technical Test Automation Challenges PDF

Transféré par

Droits d'auteur :

Formats disponibles

Technical Test Automation Challenges:

Technical Test Automation

Origin of Test Automation Patterns

 Experiences book published 2012

Test automation patterns wiki

 purpose: share information, ideas and

 wiki started 28 January 2013

Test Automation Issues

Test Automation Issues (>50)

Test Automation Patterns

 Patterns are written in CAPS

Test Automation Patterns (>80)

Test automation patterns wiki

Test automation patterns wiki

Exercise: Issues (1)

Scenario 1: Owen Optimistic

Exercise: Issues (1)

 technical or management issues?

Navigating the issues

 the Wiki offers a way to identify your

Diagnostic? What's that?

How to use the Diagnostic

Exercise: Issues (2)

 Work out Exercise 2

Exercise: Issues (2)

 did you find the same ones as before?

Issues found in exercises

• this exercise is based on a common

• we will look at patterns to address this

• “everyone will do the sensible thing”

Easy way: use tool’s architecture

Technical layer: Environment layer:

Data-driven: example data file

Data file: TestCase1

Tests are not identical – first one has 3 countries, 2 moves,

• more tests, fewer scripts Acceptance test: end of day

Comparison of data files

data-driven approach keyword approach

what happens when

Exercise: Patterns (1)

Scenario 1: Owen Optimistic

Exercise: Patterns (1)

 technical or management patterns?

Exercise: Patterns (2)

 Work out Exercise 2

Exercise: Patterns (2)

 did you find the same ones as before?

Issues found in exercises

• the exercises are based on common

• we will look at patterns to address some of

DOCUMENT THE TESTWARE

DOCUMENT THE TESTWARE

advantages of a standard approach

This saves time and effort!

*lists of relevant keywords / test actions 47

General control script

For each test to be executed

Scenario 1: Owen Optimistic

 what search methods did you use?

Issues found in exercises

• the exercises are based on common

AUTOMATE WHAT’S NEEDED

• tests that are worth automating