Académique Documents
Professionnel Documents
Culture Documents
ABSTRACT
In this paper we present a new, non-pheromone-based test suite optimization
approach inspired by the behavior of biological bees. Our proposed approach
is based on ABC (Artificial Bee Colony Optimization) which is motivated by
the intelligent behavior of honey bees. In our proposed system, the sites are
the nodes in the Software under Test (SUT), the artificial bees modify the test
cases with time and the bees aim is to discover the places of nodes with higher
coverage and finally the one with the highest usage by the given test case. Since
ABC system combines local search methods carried out by employed bees
with global search methods managed by onlookers and scouts, we attain near
global optima. We investigate whether this new approach outperforms existing
test optimization approach based on Genetic Algorithms (GA) in the task of
software test optimization. Taking into account the results of our experiments,
we conclude that (i) the proposed approach uses fewer iterations to complete
the task; (ii) is more scalable, i.e., it requires less computation time to complete
the task, and finally (iii) our approach is best in achieving near global optimal
solution.
Keywords: Software under Test (SUT), Software Testing, Artificial Bee Colony
Optimization (ABC), Coverage, Software Test Optimization, Genetic Algorithm
(GA).
1- INTRODUCTION
Software testing process typically consumes at least 50% of the
total cost involved in software development [1]. This percentage might
even be higher if the software is safety critical [2]. Automation of the testing
process is desirable both to reduce the testing cost and also to improve the
quality of software. Many of the automation activities focus on the test script
level automation and execution of the test scripts in capture and playback model.
Most software development organizations automate test-related tasks. But, the
test automation process cannot be generalized. While testing tools available in
the market might not be applicable in all test environments, they may be useful
in some of them.
The automation of test data generation is still a research area since the
1
automated testing tools simply generate test data without any consideration of
amount of time spent for test data generation and selection of test cases based
on the test adequacy criteria. It is up to the tester to choose the test cases from
the generated test cases and they have to stop the testing process whenever
they feel exhaust or the specific test adequacy criterion is met. Hence, this
process involves a lot of human intervention and thus leads to lack of achieving
the expected goal.
As Phil McMinn briefed in his paper [6], exhaustive enumeration of a programs
input is infeasible for any reasonably-sized program, yet random methods are
unreliable and unlikely to exercise deeper features of software that are not
exercised by mere chance. Previous efforts have been limited by the size and
complexity of the software involved, and the basic fact that in general, test data
generation is an undecidable problem. The application of meta-heuristic search
techniques to test data generation is a possibility which offers much promise for
these problems. Meta-heuristic search techniques are high-level frameworks
which utilize heuristics in order to find solutions to combinatorial problems at a
reasonable computational cost. Such a problem may have been classified as
NP-complete or NP-hard, or be a problem for which a polynomial time algorithm
is known to exist but is not practical.
The application of artificial intelligence (AI) techniques in Software Engineering
(SE) is an emerging area of research that brings about the cross fertilization
of ideas across two domains [3,9]. A number of published works have begun
to examine the effective use of AI for SE related activities that are inherently
knowledge intensive and human-centered.
Recent research and development of ABC based systems are focusing mostly
on applications such as financial decision making systems, transportation,
manufacturing, aerospace, military and so on [19]. In our approach, we extended
the functionality of the bee to do the testing and monitoring activity so that,
it reduces the manual work and improves the confidence on the software by
testing it with the coverage of the given software. In our approach we used state
coverage, code coverage, branch coverage and path coverage as coverage
based test adequacy criteria. As per Horgaon, these coverage based metrics
are used to reveal errors in a powerful way [4].
There are several coverage based test adequacy criterion available (Beizer,
1990). In our approach, we used, state coverage, code coverage, branch
coverage and path coverage as test adequacy criteria.
State Coverage - Unlike other coverage-based criteria, state coverage measures
the extent of checks of program behavior. Let Ni be the set of output-defining
nodes of SUT subject to test ti, and Vi be the set of covered output defining
nodes of SUT subject to test ti. The optimistic state coverage of test suite T is
| Vi|
| Ni|
(5)
Code Coverage-In code testing, criteria based on coverage of the building blocks
of programs can be used to determine the adequacy of tests. Code coverage is
a measure used in software testing. It describes the degree to which the source
code of a program has been tested. It is a form of testing that inspects the code
directly and is therefore a form of white box testing.
Branch Coverage - Branch coverage adequacy criterion verifies whether each
control structure (such as an if statement) is evaluated both to true and false
conditions.
Branch Coverage % = No. of Branches covered / Total No. of branches
Path Coverage based Test Adequacy Criterion - A test T is considered adequate
if it tests all paths. In case the program contains a loop, then it is adequate to
traverse the loop body zero time or once.
Test Optimization To maximize the profit of coverage and minimize the total
number of test cases needed.
2- LITERATURE REVIEW
2-1 EXISTING WORK ON SOFTWARE TEST OPTIMIZATION
The literature review shows that:
the research on software testing problems has centered mostly on software
test optimization;
intelligent software testing techniques are extensively proposed to solve
software test optimization problems;
several meta-heuristic search techniques such as applying rules, metaheuristics (like Genetic Algorithm (GA), Ant Colony Optimization (ACO), Tabu
Search, Simulated Annealing, Bacteriologic Algorithm (BA), etc.), Fuzzy
logic and Neural Networks (NN), Hybrid Genetic Algorithm (HGA) and other
approximation methods [13-18] have been proposed for some specific types
of problems in test optimization; and
there is still a need for a more effective solution approach to more general
problems such as test suite optimization and there is a need for developing
5
system (FBS). In the case of FBS, the agents (artificial bees) use approximate
reasoning and rules of fuzzy logic in their communication and acting. In this
way, the FBS is capable to solve deterministic combinatorial problems, as
well as combinatorial problems characterized by uncertainty [24].
Li-Pei Wong et.al, presented an improved Bee Colony Optimization algorithm
with Big Valley landscape exploitation as a biologically inspired approach to
solve the Job Shop Scheduling problem. They compared the experimental
results of the proposed algorithm with Shifting Bottleneck Heuristic, Tabu
Search Algorithm and Bee Colony Algorithm with Neighborhood Search on
Taillard JSSP benchmark and they showed that it is comparable to these
approaches [23].
Duan TEODOROVI et.al, proposed a bee colony optimization approach to
complex transportation problems [22].
Adil Baykaso lu et.al, applied Artificial Bee Colony (ABC) Algorithm to
Generalized Assignment Problem in the book on Swarm Intelligence: Focus
on Ant and Particle Swarm Optimization [25].
Alok Singh applied Artificial Bee Colony (ABC) Algorithm to the leafconstrained minimum spanning tree problem. Given an undirected,
connected, weighted graph, the leaf-constrained minimum spanning tree
(LCMST) problem seeks on this graph a spanning tree of minimum weight
among all the spanning trees of the graph that have at least l leaves. In this
paper, they proposed an artificial bee colony (ABC) algorithm for the LCMST
problem. They compared the performance of ABC approach against the
existing best approaches. They demonstrated the superiority of the new ABC
approach over all the other approaches based on computational results. The
new approach obtained better quality solutions in shorter time [26].
R. Srinivasa Rao et.al, applied Artificial Bee Colony Algorithm for Optimization
of Distribution Network Configuration for Loss Reduction [27].
Nurhan Karaboga proposed a new design method based on artificial bee
colony algorithm for digital IIR filters [28].
Mohammad et.al, have applied honey-bee mating optimization algorithm on
clustering in data mining. They applied honey-bee mating technique to avoid
local optima in cluster analysis. They also compared the efficiency of the
approach with other approaches such as GA, SA, TS, and ACO, on several
well known data sets. Their finding shows that the proposed algorithm works
well than the best one [29].
3- A
RTIFICIAL BEE COLONY OPTIMIZATION (ABC) AN
INTRODUCTION
Artificial Bee Colony (ABC) is one of the most recently defined algorithms by
Dervis Karaboga in 2005, motivated by the intelligent behavior of honey bees.
It is as simple as Particle Swarm Optimization (PSO) and Differential Evolution
(DE) algorithms, and uses only common control parameters such as colony size
and maximum cycle number [19].
ABC as an optimization tool provides a population-based search procedure in
which individuals called foods positions are modified by the artificial bees with
time and the bees aim is to discover the places of food sources with high nectar
amount and finally the one with the highest nectar [19].
In ABC model, the colony consists of three groups of bees: employed, onlookers
and scouts. In this system, the artificial bees fly around in a multidimensional
search space and some employed and onlooker bees choose food sources
depending on the experience of themselves and their nest mates, and adjust
their positions. Some scouts bees fly and choose the food sources randomly
without using experience. If the nectar amount of a new source is higher than
that of the previous one in their memory, they memorize the new position and
forget the previous one [19].
Employed bees goes to their food source and come back to hive and dance on
this area. The employed bee whose food source has been abandoned becomes
a scout and starts to search for finding a new food source. Onlookers watch the
dances of employed bees and choose food sources depending on dances.
10
SUT
Search Bee
Selector Bee
Replace Bee
Optimized Test
Suite Repository
Step 6
If a test path is not complete, then repeat steps 3 to 6.
Otherwise, the nodes and test cases from the temporary-node-list and
temporary-test case-list are selected for the next test path generation by
the employed bees.
Step 7 - If the onlooker bee/selector bee finds that the selected test cases are
not efficient, then the scout bee/replace bee generates a new population of test
cases and replaces the test cases in the temporary-test case-list with new test
cases.
Until the specified termination criterion is met. (Maximum number of cycles
(MCN) is reached or the specified coverage criterion is met (above 95%))
(8)
Fitness of each node is decided by the happiness value (hv) associated with
each node based on constraint satisfaction.
(i)
For a>b, ci(n) =(b-a) and if ci(n) < 0, hv(i) = MIN, otherwise H (ci
(n))=MAX or 0
(9)
(ii)
For a>=b, ci(n) = (b-a) and if ci(n) <= 0, hv(i) = MIN, otherwise H (ci
(n))=MAX or 0
(10)
(iii)
For a<b, ci(n) = (b-a) and if ci(n) > 0, hv(i) = MIN, otherwise H (ci
(n))=MAX or 0
(11)
(iv)
For a<=b, ci(n) = (b-a) and if ci(n) >= 0, hv(i) = MIN, otherwise H (ci
(n))=MAX or 0
(12)
(v)
For a==b, ci(n) = (b-a) and if ci(n) = 0, hv(i) = MIN, otherwise H (ci
(n))=MAX or 0
(13)
(vi)
For a!=b, ci(n) = (b-a) and if ci(n) != 0, hv(i) = MIN, otherwise H (ci
(n))=MAX or 0
(14)
(vii)
For a OR b, hv(i) = hv(ci(a))+hv(ci(b)
(15)
(viii) For a AND b hv(i) = MIN (hv(ci(a), hv(ci (b))
(16)
Pi values are normalized into [0,1]
8. Produce new test cases vi for the onlookers from the test cases xi,
selected depending on Pi and evaluate them.
9. Apply the greedy selection process for the onlookers between xi and
vi.
10. Determine the abandoned test case, if exists and replaces it with a new
randomly produced test case xi for the scout using the equation :
xij = minj+rand(0,1)*(maxjminj)
(17)
The scout waggle dances to indicate the new test case generation.
11. Memorize the best test case achieved so far using the fitness value.
12. cycle = cycle+1
13. Until cycle=Max. Cycle Number (MCN)
15
16
17
4 else if (b>c)
6 else if(a>c)
8}
Path
1.
1-2-3
1,1,2
2.
1-2-4-5
1,1,1
3.
1-2-4-6-7
1,1,1
4.
1-2-4-6-8
1,1,1
Where i=1 to 3 for variables a,b and c and j=1 to cyclomatic complexity
that indicates the total number of paths in the program.
xij indicates a test case i related to path j here it is a triple a,b,c. i=1
means it is for a, 2 for b and 3 for c.
18
v21=x21+q21(x21-x31)=1+(-1)(1-2) = 2
v31=x31+q31(x31-x11)=1+(0)(2-1) = 1
19
S.No.
Test Sequence
Probability value
based on Fitness
Value
Coverage % (Test
sequence)
1.
1-2-3
1,1,2
0%
1,2,1
0%
2,1,1
100%
1,1,1
0%
1,2,1
100%
1,1,1
0%
1,2,1
0%
2,2,1
100%
1,1,1
100%
2.
3.
4.
1-2-4-5
1-2-4-6-7
1-2-4-6-8
be integers greater than 0 and less than or equal to 100. The program output
may be one of the following: [Scalene Triangle; Isosceles Triangle; Equilateral
Triangle, Not a triangle]. This problem is chosen as the bench mark problem
because it is the most famous problem in software testing.
// Triangle Classification Problem
main()
{ int a,b,c;
boolean isatriangle;
1.Print(Triangle Classification Problem);
Print(Enter three integers which are sides of a triangle);
Read(a,b,c);
2. If(a<(b+c)) && (b<(a+c)) && (c<(a+b))
3.isatriangle=true;
4. Else isatriangle=false;
5. If (isatriangle)
{ 6.If(a==b) xor (a==c) xor (b==c) && !(a==b) && (a==c)
7.Print(Isosceles Triangle);
9. Print(Equilateral Triangle);
12. }
13. Else print(Not a triangle);
14. }
Figure 4 Sample code for Triangle Classification Problem
For the code in Fig.4, the Cyclomatic Complexity value is 5. There are five
independent test paths available in the given SUT and are listed in table 3. The
blocks of executable statements are given in Fig. 4.
21
Path
1.
1-2-4-13-14
2.
1-2-3-5-6-7-8-10-12-14
3.
1-2-3-5-6-8-9-10-12-14
4.
1-2-3-5-6-8-10-12-14
5.
1-2-3-5-6-8-10-11-12-14
Table 4 Sample Test Cases with expected output and path coverage
details
Test
Case
Expected Output
Path Covered
Not a Triangle
1-2-4-13-14
Not a Triangle
1-2-4-13-14
Not a Triangle
1-2-4-13-14
Equilateral
1-2-3-5-6-8-9-10-12-14
Isosceles
1-2-3-5-6-7-8-10-12-14
Isosceles
1-2-3-5-6-7-8-10-12-14
Isosceles
1-2-3-5-6-7-8-10-12-14
Scalene
1-2-3-5-6-8-10-11-12-14
9.
Scalene
1-2-3-5-6-8-10-11-12-14
10.
Scalene
1-2-3-5-6-8-10-11-12-14
The initial test cases are generated as a random population generation. Let
Xij be the initial set of test cases.
Xij = {2,2,3} where j=2 (path =2) , i=1 to 3 for parameters a, b and c. And the
values are a=2,b=2 and c=3.
The search bee searches the node corresponds to it and it identified that, the
test case is suited for node 2 in path2.
Hence, the fitness value is high for node 2, which makes the selector bee
to select this node as the best node to traverse. Then the neighbor nodes
are explored. And the test case is selected for node6 which is meant for
Isosceles Triangle. This exploration is continued until it reaches a node that
has no more frontiers to explore.
The test case is selected for path j=2.
22
Then the next generation of test cases is generated using the formula 1:
o V13= 2+1 * (2-2) = 2
o V23=2+ (1) * (2-3) = 3
o V33=3+(1) * (3-2) = 4
Vij = {2,3,4} which is fit for node 2 and so the test case is selected and
exploration is done by the selector bee.
While exploration, the search bee identifies test case is fit for node10 which
is meant for Scalene Triangle, and now the process continues till no more
frontiers to explore.
The test case is selected for path j=3.
Since, there are no more test cases available for selection, the scout
generates a new test case using the formula 4:
Xij = minj + rand(0,1)*(maxj-minj)
Xij = {2,2,3} + (-1) * {2,2,4} {2,2,3} = {2,2,2}
Hence the test case has the highest fitness value for node 2 and then selector
bee selects it and explores the neighbor nodes. Now the node 8 is selected
as the best node to explore and finally the test path 3.
The process is continued by all the bees in parallel to cover the entire SUT.
Again, the next test case is generated by scout bee using the formula 4:
Vij = minj + rand(0,1) * (maxj minj)
Vij = {2,2,2} + (1) * {2,2,4} {2,2,2}
This is repeated till all the test paths have been covered or a required path
coverage percentage has been achieved.
Hence the entire test case generation is done with coverage based analysis by
three different types of bees working in parallel.
In our case studies, we showed the coverage results of variables of type integer.
For other data types like float and double, the value range of q should be
determined in such a way that, it will produce the test cases in floating point and
double based representations.
For char data type, the ASCII equivalent of values is used for generation of test
cases. Similar to that for Boolean data types, either true or false values should
be generated for the variables which are achieved by toggling the values of q
during each cycle. Hence, the proposed approach is well suited for any type of
problem that involves different data types.
23
ABC Based
GA Based
Type of algorithm
Population Based
Population Based
Fitness Function
No. of Cycles
Termination Criteria
Pheromone / Non
Pheromone based
Non-Pheromone Based
Non-Pheromone Based
Communication about
selection
Status flag
Path Coverage %
For ABC based test optimization framework, the parameters mentioned in table
3 were applied. The initialization of test cases is done as a random process.
Then the subsequent generations were carried out by the three bees using the
formulae 6 to 17. The selection of test cases was done based on the heuristic
happiness value that shows the coverage of the SUT. For example, the selection
of test cases as x [1] =2, x [2] =1, x [3] =1 yields the result as coverage of
sequence 1-2-3. The number of test runs is increased to get more accurate
values.
25
The test objects have been tested using GA and the proposed ABC tester
algorithm. The results were gathered in terms of their path coverage, test runs
and time taken for the entire test case optimization process and is shown in
tables 8 and 9.
Table 6 Academic Problems
S.NO.
Test Object No
[1] 1.
Timetable Generation
ATP1
[2] 2.
ATP2
[3] 3.
ATP3
[4] 4.
ATP4
[5] 5.
ATP5
Test Object No
[6] 1.
ITP1
[7] 2.
Multi-point software
ITP2
[8] 3.
ITP3
[9] 4.
ITP4
The results indicated that, when compared to GA, ABC based test optimization
produced high coverage of test cases within minimal number of test runs.
Table 8 Results of Academic test problems
GA Based Optimization
S.No.
Test
Problem
1.
Path
Coverage %
Generations
Time
Path
Coverage %
Cycles
Time
ATP1
70%
100
20
95%
50
10
2.
ATP2
65%
200
45
98%
80
14
3.
ATP3
75%
70
18
97%
35
4.
ATP4
90%
300
52
99%
50
10
5.
ATP5
85%
150
29
96%
70
12
The results gathered from academic problems indicated that, even after we
increased the number of cycles in test case generation, the path coverage
percentage is low in GA when compared to ABC.
26
Test
Problem
1.
GA Based Optimization
Path
Coverage %
ITP1
2.
ABCBased Optimization
Generations
Time
(sec)
Path
Coverage %
Cycles
Time
70%
500
40
92%
150
22
ITP2
75%
300
35
98%
75
20
3.
ITP3
80%
400
52
97%
125
25
4.
ITP4
68%
180
20
98%
60
10
Similarly, for industrial test problems; we identified that, for complex systems
which involve multiple data types the performance of ABC is superior and takes
only less time for test case generation process.
8-CONCLUSION
In this paper, we demonstrated the application of ABC in software test suite
optimization and showed the superiority of the proposed approach over the
existing GA based approach. Problems with GA include no memorization, non
linear optimization, risk of suboptimal solution and delayed convergence. There
is no guarantee for global optimal solution even when it may be reached. But,
ABC model based test suite optimization generates near global optimal results
and it converges within less number of test runs. As a future work, different
versions of ABC have to be applied for test suite optimization and conduct an
analytical study in finding the best ABC version in near global optimal solution
generation for software testing. Also, the performance of ABC can be compared
with Ant Colony Optimization (ACO) for efficiency evaluation.
REFERENCES
[1] Rudolf Ramler, Klaus Wolfmaier, Economic perspectives in test automation:
balancing automated and manual testing with opportunity cost, Proceedings
of the international workshop on Automation of Software Test, pp.15-23,
2006.
[2] Michael Grottke, Kishor S. Trivedi, Fighting Bugs: Remove, Retry, Replicate,
and Rejuvenate, IEEE Computer, Vol.40, No.2, pp. 107-109, 2007.
[3] Briand, L. C., On the many ways Software Engineering can benefit from
Knowledge Engineering, Proc. 14th SEKE, Italy, pp. 3-6, 2002.
[4] Horgan, J., London, S., and Lyu, M., Achieving Software Quality with Testing
Coverage Measures, IEEE Computer, Vol. 27 No.9 pp. 60-69, 1994.
[5] Li, H., Lam, C.P., Optimization of State-based Test Suites for Software
Systems: An Evolutionary Approach, International Journal of Computer &
27
29
APPENDIX A
Sample Code
A sample code for binary search tree construction is given below for which the
path coverage metric has been calculated.
//Binary Tree Node Construction
public class BinaryTreeNode {
private BinaryTreeNode left;
private BinaryTreeNode right;
private Comparable obj;
public BinaryTreeNode(Comparable x) {
obj = x;
left = null;
right = null;
}
public void insert(BinaryTreeNode z) {
if (z.obj.compareTo(obj) == 0)
return;
if (z.obj.compareTo(obj) < 0)
if (left == null)
left = z;
else
left.insert(z);
else
if (right == null)
right = z;
else
right.insert(z);
}
public BinaryTreeNode search(Comparable x)
{
if (x.compareTo(obj) == 0)
return this;
if (x.compareTo(obj) < 0)
if (left == null)
return null;
else
return left.search(x);
else
if (right == null)
return null;
else
return right.search(x);
}}
//Binary Tree Construction
public class BinaryTree {
30
tg3=tg3+6;
return tg3;
}
BinaryTreeNode n = root.search(x);
tg3=tg3+7;
if (n != null)
{tg3=tg3+8;}
else
{tg3=tg3+9;}
return tg3;
}}
Figure A-2 - Path Coverage Criterion Evaluation
for Binary Search Program
Appendix B
32
Figure B-2 Happiness value calculation and path coverage % of initial set
of test cases
Figure B-3 Selected test cases along with their Path coverage %
33