TS 708

Automated Testing of
Object-Oriented
Components using
Intelligent Test Artifacts
Michael Silverstein
Lead Architect
SilverMark, Inc.
(www.javatesting.com)
1 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts, Copyright 2000 SilverMark, Inc.
Component Test
Frameworks
• Level 0: Self tests, usually in static
methods
• Level 1: Test cases implemented as
methods in harness classes
• Level 2: Basic execution framework
(J-Unit, for example)
– TestSuite, TestCase, TestResult
• Level 3: Execution framework includes
objects that encapsulate test specific
behavior, promotes task-configured
reusable test components
2 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,
Example Application

Test Design Questions
• How do I create a specifically configured

instance of a component under test?
• How do I manage the application of a
stimulus to the component under test?
• How do I organize stimuli into reusable
groups?

• How do I validate the state of the

component and system under test after
one or more stimuli?
– How do I manage reference objects?
– How do I compare references objects to
target objects, so that just the states that
are important are compared?
– How do I intelligently traverse the state of
my object under test?
• How do I validate that expected
exceptions are raised?
• How do I validate that expected

events are fired?
• How do I manage variations on
test inputs in order to drive different
paths with the same test stimuli?
• How do I map development
activities to the tests that validate
their correctness?

Pattern Responses
to Design Questions
• Test Component—Implement a
reusable component model to
encapsulate configurable aspects
of automated testing
• Test Sequence—Present a complete
or commonly implemented subset of
a test as a discrete, reusable unit
• Configured Instance—Provide an
instance of an object configured
to a particular state

Pattern Responses
to Design Questions
• Test Stimulus—Encapsulate a
stimulus/response validation operation
for a component under test for singly
or multiply threaded systems
• Nested State—Provide a mechanism
for specifying and traversing a chain
of object references to extract a
specific state

Pattern Responses
to Design Questions
• Aggregate State—Represent the state of a
component in terms of an aggregate set or
subset of the states of itself and its
subcomponents
• State Validation—Assert that a component
is in the expected state. Define this
assertion as a reusable component
• Object Variation—Organize configured
object variations

Test Component
• Intent: Implement a reusable component

model to represent aspects of automated
testing, of arbitrary granularity
• Context: Move common test code out of
test cases and into a framework in a way
that supports creation of reusable test
components

Test Component
• Strategy: Create a hierarchy of classes

called Artifacts, each of which embodies
specific test execution behavior, as well
as the properties required to configure
instances for specific applications of
the artifact
– A specification for an artifact describes a
specific configuration of an artifact instance

Test Artifact—Properties
• Name—Used to identify and locate

an artifact
• Description—More extensive descriptive
information about the artifact, either as
text or a reference to some external
document (URL)
• Classification of Intent—Specifies intent
of artifact for documentation, (use case
scenario, requirements, validation,
invariant, etc.)

• Expected exception—A reference

to an exception that is expected to
be signaled as a result of execution
of the artifact. Execution of the artifact
is said to have failed if the exception
is not raised
• Precondition action—Actions to set and/or
verify the state of the system under test
prior to further execution of the artifact

• Invariant action—A set of assertions that

express general consistency constraints that
apply to every class instance as a whole,
regardless of stimulus. If specified, this is
executed after execution of the artifact
• Synchronization—Strategy for determining
when a response to a stimulus has
completed
– Message completed/returned [synchronous]
– Event(s) fired [synchronous or asynchronous]
– Miscellaneous system state change [asynchronous]

What Do Test Artifacts
Look Like?
• An object implemented to provide a specific
test-oriented behavior, configured to act on
some aspect of the system under test
• In many cases, a test artifact is defined in terms
of other test artifact definitions or references to
other test artifacts
• Instantiate test artifact or artifact tree,
then execute it
“Test artifact #2”
Instantia
te eren ce
tifact ref
ar
Test
“Test artifact #1”
Execute

Where Do Artifacts
Come From?
• Hierarchy of test classes whose purpose
is to provide test artifacts and supporting
methods, directed toward aspects of the
system under test
– Structural: based on the structural
relationships of the classes under test
– Functional: based on the functional
relationships of classes, clusters and
subsystems of system under test

Providing Artifacts
Test hierarchy
Hierarchy under test

Test Artifact—Consequences
• Test artifacts provide semantics that

capture important strategic and tactical
decisions about tests
• Implementing typical testing patterns
as a palette of reusable, individually
configurable component types that
embody common testing behavior
enables test developers to focus
less on test implementation and more
on the problem of exercising the
system under test
• Artifacts offer arbitrary levels of test

organization beyond the traditional class,
method, or statement level
• Test artifacts support either a bottom-up
or top-down test development process

• Test artifacts benefit test tool writers

because artifacts present a façade for
complex testing behavior, and artifacts
may be combined to form a complete
representation of a test which may
then be presented for editing by a
tool user interface
• Test development speed increases
as more reusable components
become available

• Defining test artifacts in one and only one

place eases maintenance because
changes to test code due to changes in
the system under test are localized

Test Sequence
• Intent:
– Present a sequence of test actions of arbitrary
granularity as a discrete, reusable unit, where
test actions may themselves be references to
predefined reusable components

Test Sequence
• Context:
– A prerequisite for creating tests via
composition of components is a container
in which to hold them
– Fine-grained executable units enable
automated testing tools to provide better
definition UIs, better tracking of test execution
progress, and more detailed reports

Test Sequence
• Strategy:
– Create a composite test sequence artifact
composed of artifact specifications
– Execution behavior: execute each component
in order

Test Sequence
• Consequences:
– The test sequence artifact encourages
construction through composition
– Because composite artifacts are responsible
for sequencing through their components
during execution, they may also embody
general execution rules, such as ceasing
execution on failure

Test Sequence
• Example Test Sequences:

– Create new Bank
– Add new account to Bank
– Remove account from Bank
– Insert card
• Account found
• Account not found
– Withdraw cash <= balance
– Withdraw cash > balance (exception)

Test Sequence
• Maps nicely to Use Case

Scenario, requirements
– Supports “Uses” and “Extends”

Test Stimulus
• Intent:
– Encapsulate a stimulus/response validation operation
for a component under test for singly or multiply
threaded systems
• Context:
– The basic unit of execution of a test case is the
application of a stimulus
– The response to a stimulus should be validated in
terms of a change in the state of the component
under test, or some area of the system under test,
as well as an object returned directly by the stimulus
– In multithreaded systems, the response to a
stimulus may be temporally non-deterministic

Test Stimulus—Strategy
Create an artifact that embodies any or all of:
• A stimulus on some object, defined as one of:
– A reference to a target object and the action to perform on the
target object (message send), as well as any arguments.
References to configured instance artifacts serve well for
acquiring argument values
– The selector for a method within a supporting test class
– A reference to an artifact
– Other executable representation of the stimulus
(Java™ technology inner class, Smalltalk block)
• Precondition state validation
• Postcondition state validation
• Invariant validation
• A synchronization mechanism for suspending test
execution until an asynchronous process
has completed, or time limit is exceeded
Configured Instance
• Intent: Provide an instance of an

object configured to a particular state
• Context: Creating supporting objects is a
necessary, and often distracting, evil of
component testing. Supporting objects
are required as:
– Inputs to individual stimuli (message
parameters)
– Comparison targets (gold standard objects)
– Starting points for sequences of stimuli for
testing components in a specific state
Configured Instance
• Some Configured Instances for

ATM Example:
– New Account
– Account with $0.00/S/M/L balance
– Inactive account
– Debit Transaction for a particular
account and amount
– Credit Transaction for a particular
account and amount

Configured Instance
• Strategy: Create an artifact whose purpose

is to return an object in a particular state
– Approaches:
• Instantiation: Create a configured instance of
a class based on:
– The class to instantiate
– A constructor method and parameters
– Additional stimuli
• Materialization: Retrieve serialized object
from some table/database/file, then apply
additional stimuli as necessary

Configured Instance
• Strategy: Create an artifact whose purpose

is to return an object in a particular state
– A configured instance may be implemented as
a test sequence or test sequence subclass
• Subclass would have additional specification
for instantiation or retrieval of object, before
applying additional stimuli

Configured Instance
• Consequences: Over time, test developers

accumulate a library of supporting object
instances that they may readily apply to
test actions

Configured Instance Example
Test Class Artifact Class under test
com.mybank.atm.tests.Account provides instances

of ConfiguredInstance that are set up to return instances
of com.mybank.atm.Account in a particular state

Configured Instance
Example—Reusing Preexisting Artifact
Reference to
ConfiguredInstance
(not shown)

Configured Instance
Example—Reusing Preexisting Artifact
Reference to
ConfiguredInstance
(not shown)

Simple Unit Test Scenario
Instantiate a DebitTransaction setting a particular state, and execute…

Example Code—
Configured Instance
// Return a configured instance of a DebitTransaction with amount set to $123
public ConfiguredInstance smallDebitTransaction() {

ConfiguredInstance artifact =
new ConfiguredInstance(“Debit transaction for small amount");
// Set subject via constructor specification

artifact.setConstructorStep(
new Stimulus(“Default").setMessageSpec(
message(com.myBank.atm.DebitTransaction.class)));
// Apply stimulus to subject via Stimulus

artifact.addComponent(new Stimulus(“Set amount").setMessageSpec(
message(subject(),“setAmount",new int[]
{123}));
return artifact; }

Example Code—
Artifact Reference
/* Artifact that references an artifact */
public TestSequence requestDebitScenario() {
TestSequence artifact =
new TestSequence(“Debit transaction scenario");
ArtifactReference reference =
new ArtifactReference(“Debit transaction test subject”,
com.myBank.atm.tests.DebitTransaction,”smallDebitTransaction”);
artifact.setConstructorStep(reference);
// Apply stimulus to subject via Stimulus
artifact.addComponent(
new Stimulus(“Execute transaction").setMessageSpec(
message(subject(),“execute)));
return artifact; }

Example Test Tool View

Nested State
• Intent: Provide a mechanism for specifying

and traversing a recursive chain of object
references to extract a specific state
• Context: Complex objects and object
clusters, especially implementations of
the façade pattern, conceal recursively
held collaborating objects whose individual
states contribute to the whole, and whose
access is often required for state validation
during testing

Nested State
• Strategy:
– Implement methods on the root class to provide
nested state
• Adds code bloat to delivered code for the sake
of testability
– Implement code that provides a means to specify an
access scheme for a particular state of a
given object. For example,
return myObject.x().y().z();
– Configure a composite artifact to contain a sequence
of stimuli to be used to access successive states in
an object reference chain
• Less straightforward, but presents opportunity
to build off of a state returned by an artifact
reference
Nested State
• Consequences:
– A configured nested state artifact is a
permanent, reusable access strategy
for a particular sub-state within a component
– Maintenance is localized

Aggregate State
• Intent: Represent the state of a component

in terms of an aggregate of some subset of
the states of its subcomponents
• Context: The recursive state of an object is
the value set of instance variable values
obtained by recursively expanding each
instance variable value
– A component has one complete recursive
state, but may have many aggregate state
representations, or views, depending on
which states are of interest

Aggregate State
• Strategy:
– Populate a composite aggregate state artifact
with components that, when executed, extract
individual states from the object under
consideration
• Nested state artifacts are a prime candidate
for population
• Execution time behavior gathers and returns
a collection of resultant execution values of
each of its components
• Implement comparison operators that take
into account the complete set of states,
like equals()
Aggregate State
• A configured aggregate state artifact acts as a reusable

representation of a particular view of an object’s state
• Useful shorthand for comparing objects:
Typical aggregate state comparison

return customer.x().equals(referenceCustomer.x()) &
customer.x().y().equals(referenceCustomer.x().y()) &
customer.x().y().z().equals(referenceCustomer.x().y().z())
… etc. for each state
Artifact aggregate state comparison

(new MyArtifacts().getArtifactNamed(
“General Customer State”)).subject(customer).equals(
(new MyArtifacts().getArtifactNamed(
“General Customer State”)).subject(referenceCustomer));

State Validation
• Intent: Assert that a component is in an

expected state. Define this assertion as
a reusable component
• Context:
– State validation involves comparing an object’s
recursive state to some known, expected state,
encoded as:
• A comparison to one or more Gold Standard
reference objects
• Simple comparisons to primitives or literals
• A test for some known object’s inclusion or
exclusion within a collection type
• A comparison between object impressions
State Validation—Strategy
• A common strategy: implement an

assert(boolean) test API to log results
of executing an expression that logs the
success or failure of a contained
expression, during test execution
– Simple, flexible, but not very reusable

State Validation—Strategy
• Use common postcondition validation

action test artifact property:
– Close, structural relationship between stimulus
and response validation enhances understanding
of the test. Define a validation action as:
• An expression that evaluates to a boolean. This is similar
to the assert API discussed above
• A reference to an artifact that performs the validation
• Where the same validation is performed repeatedly
(invariants, for example) defining and then referencing
an artifact is advantageous, especially if many state
comparisons are involved
• Implement an aggregate state artifact as a
comparison wrapper, implementing equals()

Object Variation
• Intent: Organize variations of configured

object instances as a single,
reusable resource
• Context: Need to be able to pass input
variations of test objects to a test for the
purpose of driving execution of the
different paths (data driven tests)
• Variations are themselves reusable units
that may shared among test developers

Object Variation
• Strategy:
– Implement an artifact that presents an
enumeration interface for sequentially
accessing its elements, whereby execution
of a given test element is repeated over the
elements returned by the object variation
artifact
• Statically described references to configured
object instances
• Objects dynamically generated by rules
– For example: an Integer interval or
a calculated equivalence set

Object Variation
• Strategy:
– For multiple variation artifacts, implement
pluggable value mixing policy (one-by-one,
each-with-every, etc.).

Object Variation
• Consequences: Useful variations are

themselves reusable units that may shared
among test developers

Object Variation Example
“Account Variations Artifact”
Components Account Variations
Configured Instance
New Account
Artifact Reference
Configured Instance
Account
Balance=$0
Artifact Reference
Configured Instance Account
Balance=$1
Artifact Reference
Configured Instance Account
Balance=$1E6

Object Variation Example

Object Variations with Reuse
Previously defined
ConfiguredInstance artifact

Example Artifact Hierarchy

References and
Further Reading
[Beck 94] Kent Beck, Simple Smalltalk Testing. The Smalltalk Report 4(2):16-18,
October 1994
[Binder 99] Robert V. Binder, Testing Object-Oriented Systems: Models, Patterns,

and Tools. Addison-Wesley Object Technology Series. 1999
[Dwyer 99] Graham Dwyer and Graham Freeburn, Business Object Scenarios: a fifth-
generation approach to automated testing. In Software Test Automation : Effective Use of
Test Execution Tools. Edited by Mark Fewster and Dorothy Graham Addison-Wesley Pub
Co. 1999
Firesmith 96] Donald G. Firesmith, Pattern Language for Testing Object-Oriented

Software, Object Magazine, Vol. 5, No. 8, SIGS Publications Inc., New York, New York,
January 1996, pages 32-38.
[Gamma+ 95] Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design
Patterns : Elements of Reusable Object-Oriented Software. Addison-Wesley Professional
Computing. 1995
[Fowler 99] Martin Fowler, A UML Testing Framework. Software Development

Magazine. April, 1999

References and
Further Reading
[Fewster 99] Mark Fewster, Dorothy Graham Software Test Automation :
Effective Use of Test Execution Tools. Addison-Wesley Pub Co. 1999
[Hetzel 84] Bill Hetzel, Software Testing, QED Information Sciences, Inc. 1984
[Jeffries 99] Ronald E. Jeffries, Extreme Testing, Software Testing and Quality
Engineering 1(2):23-26 March/April 1999
[Jorgenson 94] Paul C. Jorgenson and Carl Erickson,. Object-Oriented Integration Testing.
CACM, 37(9):30-38 September 1994
[Kaner 93] Cen Kaner, Jack Falk, Hung Quoc, Testing Computer Software, International
Thompson Computer Press 1993
[Marick 95] Brian Marick, The Craft of Software Testing, Prentice Hall. 1995
[McGregor 94] John D. McGregor, Testing Object-Oriented Systems Tutorial Notes In The
Conference on Object-
Oriented Systems, Languages and Applications, October 1994
[McGregor 96a] John D. McGregor and Timothy Korson, Integrating Object-Oriented

Testing and Development Processes. CACM, 37(9):59-77 September 1994

References and
Further Reading
[McGregor 96b] John D. McGregor and A. Kare, PACT: An Architecture for Object-
Oriented Component Testing, in the Proceedings of The Ninth International Software
Quality Week in San Francisco, California, SR Institute, San Francisco, 22 May 1996.
[McGregor 99] John D. McGregor Test Patterns: Please Stand By, Journal of Object-
Oriented Programming, 12(3):14-19, June 1999
[Murphy 94] Gail C. Murphy, Paul Townsend, and Pok Sze Wong, Experiences With
Cluster and Class Testing. CACM, 37(9):39-47 September 1994
[Myers 79] Glenford Myers, The Art of Software Testing, John Wiley & Sons, Inc. 1979
[Poston 94] Robert M.Poston, Automated testing from Object Models. CACM, 37(9):48-58
September 1994.
[Siegel 96] Shel Siegel, Object Oriented Software Testing, a Hierarchical Approach. John
Wiley & Sons, Inc. 1996
[Siepmann 94] Ernst Siepmann, A. Richard Newton: TOBAC: A Test Case Browser for
Testing Object-Oriented Software.
ISSTA 1994: 154-168


TS 708

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

TS 708

Transféré par

Droits d'auteur :

Formats disponibles

Automated Testing of

3 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• How do I create a specifically configured

4 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• How do I validate the state of the

• How do I validate that expected

6 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

7 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

8 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

9 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Intent: Implement a reusable component

10 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Strategy: Create a hierarchy of classes

11 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Name—Used to identify and locate

12 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Expected exception—A reference

13 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Invariant action—A set of assertions that

14 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

15 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

16 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

Hierarchy under test

17 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Test artifacts provide semantics that

• Artifacts offer arbitrary levels of test

19 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Test artifacts benefit test tool writers

20 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Defining test artifacts in one and only one

21 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

22 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

23 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

24 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

25 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Example Test Sequences:

26 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Maps nicely to Use Case

27 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

28 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Intent: Provide an instance of an

• Some Configured Instances for

31 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Strategy: Create an artifact whose purpose

32 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Strategy: Create an artifact whose purpose

33 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Consequences: Over time, test developers

34 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

Test Class Artifact Class under test

com.mybank.atm.tests.Account provides instances

35 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

36 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

37 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

38 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

// Return a configured instance of a DebitTransaction with amount set to $123

public ConfiguredInstance smallDebitTransaction() {

// Set subject via constructor specification

// Apply stimulus to subject via Stimulus

39 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

40 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

41 708, Automated testing of Object-Oriented Components Using Intelligent Test Artifacts,

• Intent: Provide a mechanism for specifying