Code Coverage Analysis

This paper gives a complete description of code coverage analysis (test coverage
analysis), a software testing technique.
By Steve Cornett. Copyright Bullseye Testing Technology !!"#$%%&. 'll rights
reserved. (edistri)ution in whole or in part is prohi)ited without permission.
Contents
*ntroduction
Structural Testing and +unctional Testing
The ,remise
Basic -easures
Statement Coverage
.ecision Coverage
Condition Coverage
-ultiple Condition Coverage
Condition/.ecision Coverage
-odified Condition/.ecision Coverage
,ath Coverage
0ther -easures
+unction Coverage
Call Coverage
1inear Code Sequence and 2ump (1CS'2) Coverage
.ata +low Coverage
0)3ect Code Branch Coverage
1oop Coverage
(ace Coverage
(elational 0perator Coverage
4ea5 -utation Coverage
Ta)le Coverage
Comparing -easures
Coverage 6oal for (elease
*ntermediate Coverage 6oals
Summary
(eferences
6lossary
Introduction
Code coverage analysis is the process of7
+inding areas of a program not e8ercised )y a set of test cases,
Creating additional test cases to increase coverage, and
.etermining a quantitative measure of code coverage, which is an indirect
measure of quality.
'n optional aspect of code coverage analysis is7
*dentifying redundant test cases that do not increase coverage.
' code coverage analyzer automates this process.
9ou use coverage analysis to assure quality of your set of tests, not the quality of the
actual product. 9ou do not generally use a coverage analy:er when running your set of
tests through your release candidate. Coverage analysis requires access to test program
source code and often requires recompiling it with a special command.
This paper discusses the details you should consider when planning to add coverage
analysis to your test plan. Coverage analysis has certain strengths and wea5nesses. 9ou
must choose from a range of measurement methods. 9ou should esta)lish a minimum
percentage of coverage, to determine when to stop analy:ing coverage. Coverage analysis
is one of many testing techniques; you should not rely on it alone.
Code coverage analysis is sometimes called test coverage analysis. The two terms are
synonymous. The academic world more often uses the term <test coverage< while
practitioners more often use <code coverage<. 1i5ewise, a coverage analy:er is
sometimes called a coverage monitor. * prefer the practitioner terms.
Structural Testing and Functional Testing
Code coverage analysis is a structural testing technique ('=' glass )o8 testing and white
)o8 testing). Structural testing compares test program )ehavior against the apparent
intention of the source code. This contrasts with functional testing ('=' )lac5#)o8
testing), which compares test program )ehavior against a requirements specification.
Structural testing e8amines how the program wor5s, ta5ing into account possi)le pitfalls
in the structure and logic. +unctional testing e8amines what the program accomplishes,
without regard to how it wor5s internally.
Structural testing is also called path testing since you choose test cases that cause paths to
)e ta5en through the structure of the program. .o not confuse path testing with the path
coverage measure, e8plained later.
't first glance, structural testing seems unsafe. Structural testing cannot find errors of
omission. >owever, requirements specifications sometimes do not e8ist, and are rarely
complete. This is especially true near the end of the product development time line when
the requirements specification is updated less frequently and the product itself )egins to
ta5e over the role of the specification. The difference )etween functional and structural
testing )lurs near release time.
The Premise
The )asic assumptions )ehind coverage analysis tell us a)out the strengths and
limitations of this testing technique. Some fundamental assumptions are listed )elow.
+aults relate to control flow and you can e8pose faults )y varying the control flow
?Bei:er!!% p."%@. +or e8ample, a programmer wrote <if (c)< rather than <if
(!c)<.
9ou can loo5 for failures without 5nowing what failures might occur and all tests
are reliable, in that successful test runs imply program correctness ?-orell!!%@.
The tester understands what a correct version of the program would do and can
identify differences from the correct )ehavior.
0ther assumptions include achieva)le specifications, no faults of omission, and
no unreacha)le code.
Clearly, these assumptions do not always hold. Coverage analysis e8poses some plausi)le
faults )ut does not come close to e8posing all classes of faults. Coverage analysis
provides more )enefit when applied to an application that ma5es a lot of decisions rather
than data#centric applications, such as a data)ase application.
Basic Measures
' large variety of coverage measures e8ist. >ere is a description of some fundamental
measures and their strengths and wea5nesses.
Statement Coverage
This measure reports whether each e8ecuta)le statement is encountered.
'lso 5nown as7 line coverage, segment coverage ?Atafos!BB@, C ?Bei:er!!% p.C&@ and
)asic )loc5 coverage. Basic )loc5 coverage is the same as statement coverage e8cept the
unit of code measured is each sequence of non#)ranching statements.
* highly discourage using the undescriptive name C. ,eople sometimes incorrectly use
the name C to identify decision coverage. Therefore this term has )ecome am)iguous.
The chief advantage of this measure is that it can )e applied directly to o)3ect code and
does not require processing source code. ,erformance profilers commonly implement this
measure.
The chief disadvantage of statement coverage is that it is insensitive to some control
structures. +or e8ample, consider the following C/CDD code fragment7
int* p = NULL;
if (condition)
p = &variable;
*p = 123;
4ithout a test case that causes condition to evaluate false, statement coverage rates this
code fully covered. *n fact, if condition ever evaluates false, this code fails. This is the
most serious shortcoming of statement coverage. *f#statements are very common.
Statement coverage does not report whether loops reach their termination condition # only
whether the loop )ody was e8ecuted. 4ith C, CDD, and 2ava, this limitation affects loops
that contain break statements.
Since do#while loops always e8ecute at least once, statement coverage considers them
the same ran5 as non#)ranching statements.
Statement coverage is completely insensitive to the logical operators ( and &&).
Statement coverage cannot distinguish consecutive !witch la)els.
Test cases generally correlate more to decisions than to statements. 9ou pro)a)ly would
not have % separate test cases for a sequence of % non#)ranching statements; you would
have only one test case. +or e8ample, consider an if#else statement containing one
statement in the then#clause and !! statements in the else#clause. 'fter e8ercising one of
the two possi)le paths, statement coverage gives e8treme results7 either E or !!E
coverage. Basic )loc5 coverage eliminates this pro)lem.
0ne argument in favor of statement coverage over other measures is that faults are evenly
distri)uted through code; therefore the percentage of e8ecuta)le statements covered
reflects the percentage of faults discovered. >owever, one of our fundamental
assumptions is that faults are related to control flow, not computations. 'dditionally, we
could reasona)ly e8pect that programmers strive for a relatively constant ratio of
)ranches to statements.
*n summary, this measure is affected more )y computational statements than )y
decisions.
Decision Coverage
This measure reports whether )oolean e8pressions tested in control structures (such as the
if#statement and while#statement) evaluated to )oth true and false. The entire )oolean
e8pression is considered one true#or#false predicate regardless of whether it contains
logical#and or logical#or operators. 'dditionally, this measure includes coverage of
!witch#statement cases, e8ception handlers, and interrupt handlers.
'lso 5nown as7 )ranch coverage, all#edges coverage ?(oper!!F p.&B@, )asis path
coverage ?(oper!!F p.FB@, C$ ?Bei:er!!% p.C&@, decision#decision#path testing
?(oper!!F p.G!@. <Basis path< testing selects paths that achieve decision coverage.
* discourage using the undescriptive name C$ )ecause of the confusion with the term C.
This measure has the advantage of simplicity without the pro)lems of statement
coverage.
' disadvantage is that this measure ignores )ranches within )oolean e8pressions which
occur due to short#circuit operators. +or e8ample, consider the following C/CDD/2ava
code fragment7
if (condition1 && (condition2 f"nction1()))
!tate#ent1;
el!e
!tate#ent2;
This measure could consider the control structure completely e8ercised without a call to
f"nction1. The test e8pression is true when condition1 is true and condition2 is true,
and the test e8pression is false when condition1 is false. *n this instance, the short#
circuit operators preclude a call to f"nction1.
Condition Coverage
Condition coverage reports the true or false outcome of each )oolean su)#e8pression,
separated )y logical#and and logical#or if they occur. Condition coverage measures the
su)#e8pressions independently of each other.
This measure is similar to decision coverage )ut has )etter sensitivity to the control flow.
>owever, full condition coverage does not guarantee full decision coverage. +or
e8ample, consider the following CDD/2ava fragment.
bool f(bool e) $ ret"rn fal!e; %
bool a&2' = $ fal!e( fal!e %;
if (f(a && b)) )))
if (a&int(a && b)') )))
if ((a && b) * fal!e + fal!e) )))
'll three of the if#statements a)ove )ranch false regardless of the values of a and b.
>owever if you e8ercise this code with a and b having all possi)le com)inations of
values, condition coverage reports full coverage.
Multiple Condition Coverage
-ultiple condition coverage reports whether every possi)le com)ination of )oolean su)#
e8pressions occurs. 's with condition coverage, the su)#e8pressions are separated )y
logical#and and logical#or, when present. The test cases required for full multiple
condition coverage of a condition are given )y the logical operator truth ta)le for the
condition.
+or languages with short circuit operators such as C, CDD, and 2ava, an advantage of
multiple condition coverage is that it requires very thorough testing. +or these languages,
multiple condition coverage is very similar to condition coverage.
' disadvantage of this measure is that it can )e tedious to determine the minimum set of
test cases required, especially for very comple8 )oolean e8pressions. 'n additional
disadvantage of this measure is that the num)er of test cases required could vary
su)stantially among conditions that have similar comple8ity. +or e8ample, consider the
following two C/CDD/2ava conditions.
a && b && (c (d && e))
((a b) && (c d)) && e
To achieve full multiple condition coverage, the first condition requires " test cases while
the second requires . Both conditions have the same num)er of operands and operators.
The test cases are listed )elow.
a && b && (c (d && e))
1) , - - - -
2) . , - - -
3) . . , , -
/) . . , . ,
0) . . , . .
1) . . . - -
((a b) && (c d)) && e
1) , , - - -
2) , . , , -
3) , . , . ,
/) , . , . .
0) , . . - ,
1) , . . - .
2) . - , , -
3) . - , . ,
4) . - , . .
15) . - . - ,
11) . - . - .
's with condition coverage, multiple condition coverage does not include decision
coverage.
+or languages without short circuit operators such as Hisual Basic and ,ascal, multiple
condition coverage is effectively path coverage (descri)ed )elow) for logical e8pressions,
with the same advantages and disadvantages. Consider the following Hisual Basic code
fragment.
6f a 7nd b .hen
)))
-ultiple condition coverage requires four test cases, for each of the com)inations of a
and ) )oth true and false. 's with path coverage each additional logical operator dou)les
the num)er of test cases required.
Condition/Decision Coverage
Condition/.ecision Coverage is a hy)rid measure composed )y the union of condition
coverage and decision coverage.
*t has the advantage of simplicity )ut without the shortcomings of its component
measures.
BullseyeCoverage measures condition/decision coverage.
Modified Condition/Decision Coverage
'lso 5nown as -C/.C and -C.C.
This measure requires enough test cases to verify every condition can affect the result of
its encompassing decision ?Chilens5i!!F@. This measure was created at Boeing and is
required for aviation software )y (CT'/.0#CBB.
+or C, CDD and 2ava, this measure requires e8actly the same test cases as
condition/decision coverage. -odified condition/decision coverage was designed for
languages containing logical operators that do not short#circuit. The short circuit logical
operators in C, CDD and 2ava only evaluate conditions when their result can affect the
encompassing decision. The paper that defines this measure ?Chilens5i!!F@ section <G.G
I8tensions for short#circuit operators< says <?use of short#circuit operators@ forces
rela8ation of the requirement that all conditions are held fi8ed while the condition of
interest is varied.< The only programming language referenced )y this paper is 'da,
which has logical operators that do not short circuit as well as those that do.
Path Coverage
This measure reports whether each of the possi)le paths in each function have )een
followed. ' path is a unique sequence of )ranches from the function entry to the e8it.
'lso 5nown as predicate coverage. ,redicate coverage views paths as possi)le
com)inations of logical conditions ?Bei:er!!% p.!B@.
Since loops introduce an un)ounded num)er of paths, this measure considers only a
limited num)er of looping possi)ilities. ' large num)er of variations of this measure
e8ist to cope with loops. Boundary#interior path testing considers two possi)ilities for
loops7 :ero repetitions and more than :ero repetitions ?Atafos!BB@. +or do#while loops,
the two possi)ilities are one iteration and more than one iteration.
,ath coverage has the advantage of requiring very thorough testing. ,ath coverage has
two severe disadvantages. The first is that the num)er of paths is e8ponential to the
num)er of )ranches. +or e8ample, a function containing % if#statements has %$F paths
to test. 'dding 3ust one more if#statement dou)les the count to $%FB. The second
disadvantage is that many paths are impossi)le to e8ercise due to relationships of data.
+or e8ample, consider the following C/CDD code fragment7
if (!"cce!!)
!tate#ent1;
!tate#ent2;
if (!"cce!!)
!tate#ent3;
,ath coverage considers this fragment to contain F paths. *n fact, only two are feasi)le7
successJfalse and successJtrue.
(esearchers have invented many variations of path coverage to deal with the large
num)er of paths. +or e8ample, n#length su)#path coverage reports whether you e8ercised
each path of length n )ranches. 0thers variations include linear code sequence and 3ump
(1CS'2) coverage and data flow coverage.
ther Measures
>ere is a description of some variations of the fundamental measures and some less
commonly use measures.
Function Coverage
This measure reports whether you invo5ed each function or procedure. *t is useful during
preliminary testing to assure at least some coverage in all areas of the software. Broad,
shallow testing finds gross deficiencies in a test suite quic5ly.
BullseyeCoverage measures function coverage.
Call Coverage
This measure reports whether you e8ecuted each function call. The hypothesis is that
faults commonly occur in interfaces )etween modules.
'lso 5nown as call pair coverage.
!inear Code Se"uence and #ump $!CSA#% Coverage
This variation of path coverage considers only su)#paths that can easily )e represented in
the program source code, without requiring a flow graph ?4oodward!B%@. 'n 1CS'2 is
a sequence of source code lines e8ecuted in sequence. This <linear< sequence can contain
decisions as long as the control flow actually continues from one line to the ne8t at run#
time. Su)#paths are constructed )y concatenating 1CS'2s. (esearchers refer to the
coverage ratio of paths of length n 1CS'2s as the test effectiveness ratio (TI() nD$.
The advantage of this measure is that it is more thorough than decision coverage yet
avoids the e8ponential difficulty of path coverage. The disadvantage is that it does not
avoid infeasi)le paths.
Data Flo& Coverage
This variation of path coverage considers only the su)#paths from varia)le assignments to
su)sequent references of the varia)les.
The advantage of this measure is the paths reported have direct relevance to the way the
program handles data. 0ne disadvantage is that this measure does not include decision
coverage. 'nother disadvantage is comple8ity. (esearchers have proposed numerous
variations, all of which increase the comple8ity of this measure. +or e8ample, variations
distinguish )etween the use of a varia)le in a computation versus a use in a decision, and
)etween local and glo)al varia)les. 's with data flow analysis for code optimi:ation,
pointers also present pro)lems.
'(ect Code Branch Coverage
This measure reports whether each machine language conditional )ranch instruction )oth
too5 the )ranch and fell through.
This measure gives results that depend on the compiler rather than on the program
structure since compiler code generation and optimi:ation techniques can create o)3ect
code that )ears little similarity to the original source code structure.
Since )ranches disrupt the instruction pipeline, compilers sometimes avoid generating a
)ranch and instead generate an equivalent sequence of non#)ranching instructions.
Compilers often e8pand the )ody of a function inline to save the cost of a function call. *f
such functions contain )ranches, the num)er of machine language )ranches increases
dramatically relative to the original source code.
9ou are )etter off testing the original source code since it relates to program requirements
)etter than the o)3ect code.
!oop Coverage
This measure reports whether you e8ecuted each loop )ody :ero times, e8actly once, and
more than once (consecutively). +or do#while loops, loop coverage reports whether you
e8ecuted the )ody e8actly once, and more than once.
The valua)le aspect of this measure is determining whether while#loops and for#loops
e8ecute more than once, information not reported )y others measure.
's far as * 5now, only 6CT implements this measure.
)ace Coverage
This measure reports whether multiple threads e8ecute the same code at the same time. *t
helps detect failure to synchroni:e access to resources. *t is useful for testing multi#
threaded programs such as in an operating system.
's far as * 5now, only 6CT implements this measure.
)elational perator Coverage
This measure reports whether )oundary situations occur with relational operators (K, KJ,
L, LJ). The hypothesis is that )oundary test cases find off#)y#one errors and mista5en
uses of wrong relational operators such as K instead of KJ. +or e8ample, consider the
following C/CDD code fragment7
if (a 8 b)
!tate#ent;
(elational operator coverage reports whether the situation aJJ) occurs. *f aJJ) occurs
and the program )ehaves correctly, you can assume the relational operator is not suppose
to )e KJ.
's far as * 5now, only 6CT implements this measure.
*ea+ Mutation Coverage
This measure is similar to relational operator coverage )ut much more general
?>owden!B$@. *t reports whether test cases occur which would e8pose the use of wrong
operators and also wrong operands. *t wor5s )y reporting coverage of conditions derived
)y su)stituting (mutating) the programMs e8pressions with alternate operators, such as <#<
su)stituted for <D<, and with alternate varia)les su)stituted.
This measure interests the academic world mainly. Caveats are many; programs must
meet special requirements to ena)le measurement.
's far as * 5now, only 6CT implements this measure.
Ta'le Coverage
This measure indicates whether each entry in a particular array has )een referenced. This
is useful for programs that are controlled )y a finite state machine.
Comparing Measures
9ou can compare relative strengths when a stronger measure includes a wea5er measure.
.ecision coverage includes statement coverage since e8ercising every )ranch
must lead to e8ercising every statement.
Condition/decision coverage includes decision coverage and condition coverage
()y definition).
,ath coverage includes decision coverage.
,redicate coverage includes path coverage and multiple condition coverage, as
well as most other measures.
'cademia says the stronger measure subsumes the wea5er measure.
Coverage measures cannot )e compared quantitatively.
Coverage ,oal for )elease
Iach pro3ect must choose a minimum percent coverage for release criteria )ased on
availa)le testing resources and the importance of preventing post#release failures.
Clearly, safety#critical software should have a high goal. 9ou might set a higher coverage
goal for unit testing than for system testing since a failure in lower#level code may affect
multiple high#level callers.
Nsing statement coverage, decision coverage, or condition/decision coverage you
generally want to attain B%E#!%E coverage or more )efore releasing. Some people feel
that setting any goal less than %%E coverage does not assure quality. >owever, you
e8pend a lot of effort attaining coverage approaching %%E. The same effort might find
more faults in a different testing activity, such as formal technical review. 'void setting a
goal lower than B%E.
Intermediate Coverage ,oals
Choosing good intermediate coverage goals can greatly increase testing productivity.
9our highest level of testing productivity occurs when you find the most failures with the
least effort. Iffort is measured )y the time required to create test cases, add them to your
test suite and run them. *t follows that you should use a coverage analysis strategy that
increases coverage as fast as possi)le. This gives you the greatest pro)a)ility of finding
failures sooner rather than later. +igure illustrates the coverage rates for high and low
productivity. +igure $ shows the corresponding failure discovery rates.
+,(*H'TI <T9,IJ,*CT;'1TJ+igure and figure $<
Figure 1 and figure 2
0ne strategy that usually increases coverage quic5ly is to first attain some coverage
throughout the entire test program )efore striving for high coverage in any particular
area. By )riefly visiting each of the test program features, you are li5ely to find o)vious
or gross failures early. +or e8ample, suppose your application prints several types of
documents, and a fault e8ists which completely prevents printing one (and only one) of
the document types. *f you first try printing one document of each type, you pro)a)ly
find this fault sooner than if you thoroughly test each document type one at a time )y
printing many documents of that type )efore moving on to the ne8t type. The idea is to
first loo5 for failures that are easily found )y minimal testing.
The sequence of coverage goals listed )elow illustrates a possi)le implementation of this
strategy.
*nvo5e at least one function in !%E of the source files (or classes).
*nvo5e !%E of the functions.
'ttain !%E condition/decision coverage in each function.
'ttain %%E condition/decision coverage.
Aotice we do not require %%E coverage in any of the initial goals. This allows you to
defer testing the most difficult areas. This is crucial to maintaining high testing
productivity; achieve ma8imum results with minimum effort.
'void using a wea5er measure for an intermediate goal com)ined with a stronger
measure for your release goal. Iffectively, this allows the wea5nesses in the wea5er
measure to decide which test cases to defer. *nstead, use the stronger measure for all
goals and allow the difficulty of the individual test cases help you decide whether to defer
them.
Summary
Coverage analysis is a structural testing technique that helps eliminate gaps in a test suite.
*t helps most in the a)sence of a detailed, up#to#date requirements specification.
Condition/decision coverage is the )est general#purpose measure for C, CDD, and 2ava.
Setting an intermediate goal of %%E coverage (of any type) can impede testing
productivity. Before releasing, strive for B%E#!%E or more coverage of statements,
)ranches, or conditions.
)eferences
Bei-er.//0 Bei:er, Boris, <Software Testing Techniques<, $nd edition, Aew 9or57 Han
Aostrand (einhold, !!%
Chilens+i.//1 2ohn 2oseph Chilens5i and Steven ,. -iller, <'pplica)ility of -odified
Condition/.ecision Coverage to Software Testing<, Software Ingineering 2ournal,
Septem)er !!F, Hol. !, Ao. &, pp.!G#$%%.
)TCA/D2.34B, <Software Considerations in 'ir)orne Systems and Iquipment
Certification<, (CT', .ecem)er !!$, pp.G, CF.
5o&den./46 <4ea5 -utation Testing and Completeness of Test Sets<, IEEE Trans.
Software Eng., Hol.SI#B, Ao.F, 2uly !B$, pp.GC#GC!.
McCa'e./37 -cCa)e, Tom, <' Software Comple8ity -easure<, IEEE Trans. Software
Eng., Hol.$, Ao.", .ecem)er !C", pp.G%B#G$%.
Morell.//0 -orell, 1arry, <' Theory of +ault#Based Testing<, IEEE Trans. Software
Eng., Hol.", Ao.B, 'ugust !!%, pp.BFF#B&C.
8tafos./44 Atafos, Simeon,<' Comparison of Some Structural Testing Strategies<,
IEEE Trans. Software Eng., Hol.F, Ao.", 2une !BB, pp.B"B#BCF.
)oper.//1 (oper, -arc, <Software Testing<, 1ondon, -c6raw#>ill Boo5 Company,
!!F
*ood&ard./40 4oodward, -.(., >edley, .. and >ennell, -.'., <I8perience with
,ath 'nalysis and Testing of ,rograms<, *III Transactions on Software Ingineering,
Hol. SI#", Ao. G, pp. $CB#$B", -ay !B%.
,lossary
Fault # ' )ug. ' defect.
9rror # ' mista5e made )y a person that results in a fault.
Failure # The run#time manifestation of a fault.