Vous êtes sur la page 1sur 13

TSINGHUA SCIENCE AND TECHNOLOGY

ISSNll1007-0214ll02/13llpp429-441
Volume 19, Number 5, October 2014

Worst-Input Mutation Approach to Web Services Vulnerability Testing


Based on SOAP Messages

Jinfu Chen , Huanhuan Wang, Dave Towey, Chengying Mao, Rubing Huang, and Yongzhao Zhan

Abstract: The growing popularity and application of Web services have led to increased attention regarding the
vulnerability of software based on these services. Vulnerability testing examines the trustworthiness and reduces
the security risks of software systems. This paper proposes a worst-input mutation approach for testing Web
service vulnerability based on Simple Object Access Protocol (SOAP) messages. Based on characteristics of
SOAP messages, the proposed approach uses the farthest neighbor concept to guide generation of the test suite.
The corresponding automatic test case generation algorithm, namely, the Test Case generation based on the
Farthest Neighbor (TCFN), is also presented. The method involves partitioning the input domain into sub-domains
according to the number and type of SOAP message parameters in the TCFN, selecting the candidate test case
whose distance is the farthest from all executed test cases, and applying it to test the Web service. We also
implement and describe a prototype Web service vulnerability testing tool. The tool was applied to the testing of
Web services on the Internet. The experimental results show that the proposed approach can find more vulnerability
faults than other related approaches.

Key words: security testing; Web service vulnerability; SOAP message; test case generation; mutation operator

increasing. Although Web services are the typical


1 Introduction SOA form and have been the focus of widespread
attention and application, problems of quality and
Due to the rapid development and wide application of reliability represent significant obstacles to their further
the Internet, use of Service-Oriented Architecture development. Furthermore, due to some Web service
(SOA) for distributed Web systems has been characteristics, traditional software testing approaches
 Jinfu Chen, Huanhuan Wang, Rubing Huang, and Yongzhao
are not easily applied. Some factors that contribute
Zhan are with the School of Computer Science and to the difficulty of application include (1) different
Communication Engineering, Jiangsu University, Zhenjiang development and application environments (which
212013, China. E-mail: jinfuchen@ujs.edu.cn; whhyjs@gmail. increases the difficulty of testing before the Web
com; rbhuang@ujs.edu.cn; yzzhan@ujs.edu.cn. services are deployed); (2) the characteristics of Web
 Dave Towey is with the School of Computer Science, The service distribution, discovery, and dynamic bindings
University of Nottingham Ningbo China, Ningbo 315100,
as well as the uncertain and invisible processes; and (3)
China. E-mail: dave.towey@nottingham.edu.cn.
 Chengying Mao is with the School of Software and
the need for a service interface for Web service design
Communication Engineering, Jiangxi University of Finance and implementation when applying automatic testing
and Economics, Nanchang 330013, China. E-mail: maochy@ methods and techniques.
yeah.net. Although the testing of Web service robustness has
 To whom correspondence should be addressed. already been examined[1-4] and a number of tools
Manuscript received: 2014-04-01; revised: 2014-07-14; proposed, several difficulties and shortcomings remain,
accepted: 2014-08-18
including (1) the need for significant intervention in
430 Tsinghua Science and Technology, October 2014, 19(5): 429-441

the testing process; (2) the fact that only simple message parameters.
performance and access testing have been performed;  We implement the proposed approach in a
and (3) the fact that the approaches used in Simple Web Service Vulnerability Testing System
Object Access Protocol (SOAP) message mutations (WSVTS) tool, which we further evaluate through
are not optimal, with most studies to date being comparison with other Web service testing
based on Web Services Definition Language (WSDL) approaches. The results show that, in most cases,
specifications and Extensible Markup Language (XML) the proposed approach can detect more faults than
documents and few using SOAP messages. A Web other approaches.
service, whose structure and source codes are not
visible to the client, is located on the service 2 Related Work
provider’s site, making research into its vulnerability
challenging. Web service vulnerability refers to flaws Currently, research on Web service vulnerability testing
in the service that threaten the security of the computer remains limited, with studies focusing mainly on
system, for example, memory leaks, buffer overflows, functionality testing[2, 5, 6] , reliability analysis[3] , data
and cross-boundary access (where memory variables perturbation[7-9] , and Web service rule mutation[10-12] .
access areas outside their defined scope). Some types of Takase and Tajima[2] proposed an approach to the
Web service vulnerability faults might not be effectively functional testing of Web services by first extracting
revealed by traditional testing approaches, including the SOAP message using the WSDL converter and
memory security faults, which are often triggered by then exchanging messages using the SOAP message
illegal parameter values; and arithmetic security faults, binding framework. A disadvantage of this approach,
which are often caused by parameter interaction such as however, is that it only bundles some of the input
dividing by zero and out-of-range operand values. parameters to obtain the return value for a single
To address the issue of testing Web service message, rather than bundling multiple interdependent
vulnerability, we propose an approach based functions. If the combined services could be processed
on SOAP message mutation and the worst-input on the physical machine at the same time, then the
technique. The worst-input mutation method, which process could be more efficient. Sun et al.[5, 6] have
uses characteristics of SOAP messages, is presented proposed a metamorphic relations-based approach to
in detail in this paper. The corresponding automatic testing Web services in the context of SOA without
test case generation algorithm, namely, the Test Case the need for oracles. An alternative approach, based
generation based on the Farthest Neighbor (TCFN), on fault injection, was proposed by Wu et al.[3] , but
is also discussed. The method involves partitioning the working mode of SOAP documents could not
the input domain into sub-domains according to the be tested, multiple mistakes could not be injected at
number and type of SOAP message parameters in the network layer, and the fault injection messages
the TCFN and then selecting the candidate test case could not be authenticated. An approach based on data
whose distance is the farthest from all executed test communication perturbation, in which the perturbation
cases and applying it to test the Web service. Finally, operators were designed according to characteristics
a prototype Web service vulnerability testing tool is of the SOAP message, was proposed by Almeida and
implemented and applied to a number of real Web Vergilio[7] . Experiments were conducted using their
services. Experimental results show that the proposed proposed mutation operators and SMAT-WS[7] tools,
approach is both effective and practical. but it was found that the designed mutation operators
The main contributions of this paper are as follows: were not sufficient for comprehensive testing. Fuzzy
 We propose a set of mutation operators that can approaches to generating perturbation test cases have
automatically mutate Web service SOAP messages also been studied[8, 9] , but, to date, an appropriately
based on security rules and message parameter feasible test case generation algorithm has not yet been
types. presented.
 Using the farthest neighbor concept, we propose a Web service data value perturbation and rule
worst-input mutation method to test Web service mutation are the focus of this current paper. An
vulnerability and present test case generation approach to test case generation based on data
algorithms based on the number and type of SOAP value perturbation was proposed by Offutt and
Jinfu Chen et al.: Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages 431

Xu[10] ; in their approach, request messages were the RTG to an eRTG (extended RTG), which is a 6-
modified by mutation operations that resulted from tuple <E; N; DT; P; A; ns >, where E is a finite set
data value perturbation, Remote Procedure Call (RPC) of elements; N is a finite set of non-terminals; DT
communication perturbation, and data communication is a finite set of data types defined as fint, string,
perturbation. However, only a few special values bool, numerical, char, objectg; P is a finite set of
(such as maximum and minimum and valid decimal) production rules; ns is the starting non-terminal; and
were considered in the mutation process. Their data A is a 2-tuple <n; type> with n as the number of
value and communication perturbation approach[10] was parameters and “type” as the parameter type, which
modified by Melo and Silveira[11] , who also extended may be one of frec, cir, curg, where “rec” is the
the mutations[12] introduced previously[1, 7] by using an rectangular input domain, “cir” is the circular input
invalid test case value in the data value perturbation domain, and “cur” is the curved input domain. Given
and by introducing two strategies (al l and choi ce) a set of all element instances N , a mutation operator is
and four mutation operators for RPC communication in r=f .n1 ; n2 ;   ; ni /, where f is a function, i > 1, each
the data communication perturbation. The test coverage n1 ; n2 ;   ; ni 2 N and has an arbitrary data type, and r
for the RPC and document communication was also outputs the mutated n1 ;   ; ni with the same data type
increased, but the overall mutation testing approach as the input n1 ;   ; ni .
was not completely comprehensive, nor was a test case Although a set of interference operators has
generation algorithm proposed. been previously introduced[15, 16] , the uncertainty and
We previously proposed a combinatorial mutation randomness of an initial object led to data redundancy
approach for testing the interactive faults of Web and low efficiency after mutation. We have therefore
services[13] . That approach defines the corresponding designed a total of 15 mutation operators for SOAP
combinatorial strategies based on SOAP message parameter types combined with Web service features,
mutation and combinatorial testing, allowing multiple as shown in Table 1.
mutants to be injected at one time to help uncover We defined a security rule for testing the vulnerability
interactive faults. However, if the tested Web services of Web services based on the proposed mutation
have only one service method or one method parameter, operators as follows: the vulnerability of Web services
then the combinatorial mutation approach cannot is VWS=G.r/, where r=f .n1 ; n2 ;   ; ni / is the
offer its full potential advantage. In order to test mutation operator for the tested Web service, G.r/
different kinds of Web services, we now propose a represents the vulnerability that is triggered by r, and
worst-input mutation method based on the farthest ni 2N are the Web service input parameters. When
neighbor concept, which, as a complementary approach the tested Web services accept the input parameters, if
to combinatorial mutation, can also enhance the any exceptions are triggered by the mutation operators,
effectiveness of Web service vulnerability detection. then the tested Web service is deemed to have some
vulnerability flaws.
3 Mutation Operators and Security Rules It is usual to encapsulate data in a SOAP protocol
The appropriate design of mutation operators is critical format, and a SOAP message can be expressed as two
for mutation testing based on SOAP messages and, parts: input parameters and security control rules. Based
for it to be successful, the object and purpose of the on the SOAP message input parameters, a worst-input
mutation should be explicitly clear. SOAP is a message mutation approach to SOAP message mutation testing
protocol based on an XML document, which forms is proposed and presented in the following section.
the basis of the mutation object. A formal description
4 Worst-Input Mutation Approach
for the XML modeling of a SOAP message was
given by Novak and Zamulin[14] . Offutt and Xu[10] With regular mutation[7] , the mutant can be obtained
extended the Regular Tree Grammar (RTG) model to through a small modification of the legitimate
<E; N; D; P; A; ns >, but no specific parameter type input. Taking the opposite perspective, we identify
information or classification was provided regarding the the farthest neighbor sequence from the legitimate
general characteristics of the XML document. Based input and use that as the test data to generate test
on these models, we have improved and extended cases according to SOAP message types. Effective
432 Tsinghua Science and Technology, October 2014, 19(5): 429-441

Table 1 Mutation operators of Web service vulnerability testing based on SOAP messages.
ID Operator Brief description Cases/Examples
01 SVB Set the value of n to be blank Change value n to “ ”
02 SVN Set the value of n to be “null” Change value n to “null”
Insert parameter operator into
03 IPO Insert absolute value symbol into the value assigned to node n
the value assigned to node n
Delete node n and its child
04 DNS Delete root nodes and child nodes from the SOAP message
nodes from the SOAP message
05 FVS Format the value of string “%n%n ......(256)”, “%s% s(1024)” etc.
06 IIV Integer irregular value 0,+/-(1,28 -1,28 ,28 +1,216 ,216 +1,216 -1,232 ,232 +1,232 -1,264 , 264 +1)
0, 1, -1, +/-(the max float point +/-1),
07 FIV Float irregular value
+/-(the min float point +/-1),5E-324,1.7E+308,pi,e
08 CIV Char irregular value ‘A’, ‘Z’, Null, ‘a’, ‘z’, ‘ ’, ‘../ ’, ‘f’, ‘(’, ‘[’, ‘nn’, ‘n0’, ‘ns’, ‘nd’
Exchange the order of values
09 EOV Exchange the order of the values assigned to n1 , n2
assigned to nodes
10 EON Exchange the order of nodes Exchange the order of n1 , n2
Escape character string “ne nn nr nd nx ns”,
11 RSV Random string value
“nxff nxfe nx00 nx01 nx42 nxb5 nnnnn nh9cc...”
Generate String(int n) such as:
12 LSV Long string value
“AAA......(256)”, “AAA......(1024)”,“AAA......(15000)”
URL and the value of file “http://dddddddeeeeerrttttt”, “//sytem32//Notepad.exe”,
13 UVF
directory string “H:nABCnkillvirus.exe”, “D:nAA.exeexe”
14 SSI SQL string injection “a or 1=1”, “delete”, “drop table users”, “sql attempt5– –”
15 PFB Parameter flip bit Use ReverseBit() to flip the value assigned to node n

test cases should have the greatest possible test Ming distance[20] ; and a multi-dimensional variation
coverage, typical representation for triggering faults, inverse probability distribution. BRA or ResStr are
and low redundancy. The farthest neighbor idea is used when the SOAP message has only one parameter;
similar to the concept of Adaptive Random Testing NFDT or CFDT are used when there are two; and
(ART)[17-19] , which is based on various empirical the weighted Ming distance or inverse probability
observations that show that many program faults result distribution algorithms are used when there are more
in failures manifesting in contiguous areas of the than two. As can be seen in the TCFN flow chart
input domain. Therefore, suggesting that if previously (Fig. 1), the SOAP message is obtained by parsing the
executed test cases have not revealed a failure, new WSDL file of the Web services being tested. Using an
test cases should be as far from the already executed XML analysis technique, the number and type of SOAP
non-failure test cases as possible. Intuitively speaking, message parameters are extracted and are the basis on
the farthest test cases have a higher probability of which different algorithms are then called to generate
detecting Web service security exceptions. Hence, we test cases.
investigate some farthest neighbor algorithms to detect The input region is divided into sub-regions based on
the security exceptions of Web services based on related the number and type of message parameters; then, the
ART algorithms and mutation. appropriate algorithm is selected to generate test cases
The input domain is partitioned into sub-domains for testing the Web service. The main sub-algorithms of
according to the number and type of SOAP message the TCFN algorithm are as follows.
parameters. A corresponding test case generation (1) BRA algorithm
algorithm is then selected and test cases conforming When the input parameter data type is Integer (int),
to the requirements of each sub-domain are randomly the BRA algorithm and related mutation operators are
generated. The candidate test case whose distance is used to generate the farthest test cases. The BRA
farthest from all executed test cases is then selected algorithm flips all bits (from 0 to 1, and 1 to 0).
and applied to test the Web service. Here, we propose (2) ResStr algorithm
the TCFN algorithm (Algorithm 1), which is based The ResStr algorithm calculates the length of the
on the presented eRTG model. The TCFN algorithm string, reverses it, and uses the Char Irregular Value
consists of six sub-algorithms: BRA (bit reversal); (CIV) mutation operator to increase or decrease the
ResStr (string reversal); NFDT (next furthest distance length of the reversed string. The Web service’s SOAP
test); CFDT (circle furthest distance test); a weighted message can be
Jinfu Chen et al.: Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages 433

mutated using the reversed string, after which the


Algorithm 1 TCFN
response information of the client is examined to
Input: the input domain D.Xmin ; Ymin /.Xmax ; Ymax / of the
SOAP message parameter determine vulnerability.
Output: the set of test cases S D fe1 ; e2 ;    ; en g (3) NFDT algorithm
1: if (n DD 1) then The NFDT algorithm is based on the Adaptive
2: f Random Testing (ART) family of algorithms[21] . The
3: if ( DT is numerical ) then test cases are divided into sets E (executed) and C
4: call BRA algorithm and related mutation operators
(candidate), both of which are initially empty; however,
5: end if
6: if ( DT is string ) then as testing progresses, E contains n executed test cases
7: call ResStr algorithm and related mutation operators fe1 ; e2 ; e3 ;   ; en g, and C contains k random candidate
8: end if test cases fc1 ; c2 ; c3 ;   ; ck g. ART research suggests
9: g that changes in the candidate set size have little impact
10: else if (n DD 2) then
on the speed of detecting the first failure when k > 10;
11: f
so, as with previous studies, we set k to 10 in this
12: divide the type of the input region according to the
parameter’s value experiment[21] . At the start of testing, when E is
13: if (type==rec) then empty, a test case, e, is generated randomly, executed,
14: call the NFDT algorithm and appended to E. The next test case, cj , can be
15: else if (type==cir) then selected from C by calculating the distance between
16: call the CFDT algorithm each element of C and the executed test case e and
17: else if (type==cur) then
then selecting the element (cj ) that has the greatest
18: generate the max-value and min-value of the same
interval of the function according to the input region distance. The NFDT algorithm is shown in Algorithm 2.
distribution function and related mutation operators The original binary search algorithm[22] is improved
19: end if in Step 13 to increase search efficiency and verify its
20: g effectiveness. Since the input region is a finite set, as
21: else if (n > 3) then
the number of test cases grows, so too does their density
22: call the inverse probability distribution or weighted Ming
in the corresponding input region — the distance
distance algorithms based on parameter features
23: end if
between a new test case and the nearest executed test
24: return the set of test cases S D fe1 ; e2 ;    ; en g case becomes much smaller. The candidate test cases
can be considered when the distance between test cases
(d ) is relatively large. A ratio parameter is then defined
on the basis of the binary search algorithm as follows:
an array[N ] is an ordered integer array whose values
range from small to large, the sub-array from array[L]
to array[H ] is one sub-array of the ordered array, and
the element array[mid] is the value nearest to target
value x. “mid” is then selected. Hence, the ratio
x arrayŒL
parameter formula is R D . The
arrayŒH  arrayŒL
x arrayŒL mid L
formula D can then be
arrayŒH  arrayŒL H L
deduced, and “mid” can be obtained using mid D L C
R.H L/.
The difference between the NFDT algorithm and
the typical Fixed Size Candidates Set (FSCS) ART
algorithm is that the next test case is determined based
on the position of test cases previously executed by the
NFDT algorithm. The input domain is divided into two
Fig. 1 Flow chart of test case generation using the farthest areas based on previously executed test cases, thereby
neighbor algorithm. reducing the search space and number of distance
434 Tsinghua Science and Technology, October 2014, 19(5): 429-441

calculations. The improved binary search algorithm


Algorithm 2 NFDT can help to identify the candidate test case closest
Input: the input domain D.Xmin ; Ymin /.Xmax ; Ymax / of the to previously executed test cases. According to the
SOAP message parameter distance between the closest and executed test cases,
Output: the set of test cases S D fe1 ; e2 ;    ; en g a decision is made as to whether or not distance
1: input region D of the soap message
calculations will be made for all candidate test cases,
f.Xmin ; Ymin /.Xmax ; Ymax /g;
2: set E D fg; C D fg
thus potentially reducing the total number of distance
3: randomly generate the first test case e.x; y/ by using related calculations performed, which is similar to the filtering
mutation strategies and operators, and divide D into T and technique used by Chan et al.[23]
L by e’s x-value (4) CFDT algorithm
4: select T f.i; j /; .s; t /g from D, (e … T ), D D T The CFDT algorithm uses the restricted adaptive
5: while (D ¤ NULL) do
random testing technique[24] to select the next test case,
6: if (T ¤ L) then
7: if .x i / > .s x/ then using an exclusion region radius. Generally speaking,
8: the next test case is generated from T f.i; j /; .x; t/g, the selected test cases have better detection capability
then D=D [ f.i; j /; .x; t /g for finding the security exceptions of Web services
9: else than the general test cases. There are two reasons for
10: the next test case is generated from Lf.x; j /; .s; t/g, this. First, the selected test cases are always away from
then D=D [ f.x; j /; .s; t /g
previously executed test cases that have been generated
11: end if
12: else outside the exclusion region: more distant test cases
13: select field T or L randomly can more easily find security exceptions than normal
14: select a big field T 0 2 D, and randomly generate k test cases[16] . Second, the selected test cases have been
test cases fc1 ; c2 ;    ; ck g by using related mutation mutated based on mutation operators designed to detect
strategies and operators, C D C [ fc1 ; c2 ;    ; ck g special security exceptions.
15: sort the set of x value from small to large
Two parameters, A and P , are defined to measure
16: find a test case e 2 E, find Cj by using improved binary
search method, whose x-axis the SOAP input domain when it is shaped as a circle
p is nearer to e
17: calculate the distance d D x 2 C y 2 or ellipse. A and P represent the area and perimeter
18: calculate all the distances xi between e and all the
p an ellipse, A D  ab and P D  .3.a C b/=2
of
test cases behind Cj ab/, respectively[25, 26] (a and b are the radii of the
19: end if ellipse; when a D b, the ellipse is a circle). S is
20: for each Ci from Cj to Ck do
the set of test cases to be tested; C is the set of test
21: if (d > xi ) then
22: d D dnew cases randomly generated; and N is the number of test
23: else cases in S. The first test case is randomly generated
24: stop
p calculating according to d < xi < and subsequent ones are generated p using an iterative
x 2 C y 2 [24]
approach . Parameter R D A=.2n / is used to
25: end if determine the size of the exclusion region. Each test
26: end for
case in S is set as the center of a region, with R as
27: for each Ci from C1 to Cj do
28: if (d > xi ) then the radius of the circular exclusion region. The first
29: d D dnew generated test case not falling in an excluded region
30: else is then selected as the next test case. An adjustment
31: stop
p calculating according to d < xi < parameter, r, is introduced to compensate for the
x 2 C y 2 effects of overlapping zones and portions
32: end if p of zones lying
outside the input domain. R is set as Ar=.2n /. The
33: end for
34: search the max value d corresponding to test case Cj as CFDT algorithm is shown in Algorithm 3.
the next test case, Cj ! e and E D E [ e, the two fields (5) Weighted Ming distance
divided by Cj are joined in D If the number of SOAP parameters (n) is three or
35: D D T more, then the inputs are regarded as the n-tuple data
36: end while
set (T ), with each t D .x1 ; x2 ;    ; xn /, t 2 T
37: return the set of test cases S D fe1 ; e2 ;    ; en g
being a single input from T . When a test case (e)
is generated randomly, a new coordinate system is
Jinfu Chen et al.: Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages 435

Algorithm 3 CFDT space and the inverse probability distribution function


Input: the circle center e1 .x; y/ and radius R of the SOAP for the input space can be obtained, then it can be used
message input region to guide the generation of some unconventional test
Output: the set of test cases S D fe1 ; e2 ;    ; en g
cases to detect security exceptions. Generally speaking,
1: set S D fg; C D fg; n D 0; r D 1
2: randomly generate e1 by using related mutation strategies
unconventional inputs can effectively trigger security
and operators and S D S [ e1 ; exceptions for Web services. The input distribution
3: while ( R ¤ 0 ) do function is usually a probability density function whose
4: find an exclusion circle (ei , (i D 1; 2; 3;    ), R=
p output ranges from 0 to 1, where 0 means that it is
A=.2n /), randomly generate k test cases impossible to select inputs from the input domain and 1
fc1 ;p
c2 ;    ; ck g and then fc1 ; c2 ;    ; ck g … (ei ,
means that the inputs from the input domain are 100%
R= A=.2n /) C D C [ fc1 ; c2 ;    ; ck g
5: sort the k test cases according to x-value from small to available. The main steps needed to get the inverse
large, calculate all distances di , and then find the test case probability distribution function are as follows[28] .
ei whose distance is the largest and S D S [ ei , n=n+1 Step 1: Describe the probability of each input (an
6: set r to adjust the exclusion region ordered n-tuple) as a value in the (n C 1)-th dimension.
7: end while
Step 2: Determine the hyper-plane that is defined by
8: return the set of test cases S D fe1 ; e2 ;    ; en g
setting the n C 1 dimension value to a constant, 1/K,
where K is the cardinality of the input space.
defined based on it, with each previously executed Step 3: Reflect the input distribution through this
point (test case) translated appropriately. Without loss hyper plane.
of generality, the following explanation of this method Step 4: If any of the resulting values in the (n C 1)-th
is in two-dimensional space, but the method applies to dimension are negative, translate the graph by a vector
higher dimensions: Lines L1 and L2 are perpendicular of magnitude C so that all values in the nC1 dimension
axes through point e, dividing the area that includes are non-negative.
all points within the neighborhood of e into four sub- Step 5: Normalize the resulting graph in n C 1 space,
areas: M; N; S; and O: The four sub-areas are marked dividing each value by the total volume. At the end of
as the neighborhood areas of point e. Lines L1 and this step, the value in the n C 1 space associated with
L2 are also seen as the boundaries between the four each n-tuple is the probability of selection in the inverse
areas, with the corners formed by the lines being called probability distribution function.
neighborhood angles. A diagonal is formed across from The SOAP message is obtained by parsing the
each neighborhood angle, enclosing the neighborhood WSDL file of the Web services being tested and
area. Any points in the neighborhood area should is then transformed into a DOM tree. Based on
be filtered using related algorithms[20] . Based on the the number and type of SOAP parameters, the
neighborhood areas and some rules[20, 27] , the weighted appropriate TCFN algorithm is called to generate
Ming distance (WD) between points t and e is defined test cases. The complexity of the TCFN algorithm
n
!1=2 n is mainly determined by the BRA, ResStr, NFDT,
X X
2
as WD = j xi yi j wi = wi , where CFDT, weighted Ming distance, and inverse probability
i D1 i D1 distribution algorithms. In the BRA algorithm, flipping
wi represents the corresponding weight for every all bits (from 0 to 1, and 1 to 0) is time consuming. If
input parameter to define the contribution of different the bit length of the integer is n, then the complexity of
parameters. The formula can measure the distance the BRA algorithm is O.n/. In the ResStr algorithm,
between different inputs. Given a current test case (e), traversing the entire string is time consuming. If the
the Furthest Neighbor (FN) formula is used to select the length of the string is n, then the complexity of the
next test case and is defined as FN.e/ D fr 2 T j 8t 2 ResStr algorithm is O.n/. In the NFDT algorithm, a
T: WD.e; r/ > WD.t; r/g. The formula guarantees that set of test case candidates randomly complexity of
the distance between the current and next test cases is the ResStr algorithm is O.n/. In the NFDT algorithm,
always greater than or equal to the distance between the a set of test case candidates randomly generated in
next test case and any test case of T . the input domain is maintained. Each time a new
(6) Inverse probability distribution test case is required, the candidate test case that is
If the n-tuple parameters are from a continuous input farthest from all previously executed test cases is
436 Tsinghua Science and Technology, October 2014, 19(5): 429-441

selected. The runtime of the NFDT algorithm when and the Web service address information. The output is
generating n test cases is in the order of O.n2 /. The a Web service SOAP message.
main time cost of the CFDT algorithm is the large (b) SOAP message mutation generator
number of distance calculations that are performed Based on mutation operators designed for different
when new test cases are selected. The runtime of fault types, the mutation module mutates the SOAP
the CFDT algorithm when generating n test cases is message parameter type and value. The parameter type
in the order of O.n2 log n/. The time complexity of and number are obtained from the SOAP message
both the weighted Ming distance algorithm and the generator and the test cases are obtained from the test
inverse probability distribution algorithm is O.n2 /. The case generator.
total time complexity of the TCFN algorithm is (c) Test case generator
therefore O.n/ C O.n/ C O.n2 / C O.n2 / C O.n2 / C The test case generator provides a convenient
O.n2 log n/ D O.n2 log n/. interface for the tester to input test cases and can also
use different algorithms based on the SOAP message
5 Experiment and Analysis parameter number, as analyzed by the SOAP message
generator.
5.1 Experimental implementation
(d) Vulnerability analyzer
To investigate and evaluate the proposed TCFN The vulnerability analyzer generates a vulnerability
algorithm, a Web Service Vulnerability Testing System report after testing Web services. It analyzes Web
(WSVTS) was implemented. The WSVTS framework service vulnerability based on security specifications
is shown in Fig. 2; it obtains interface information by and reports on the number of security exceptions and
parsing the Uniform Resource Locator (URL) of the faults found.
Web service and gets the SOAP message by parsing the As can be seen in the WSVTS flow chart (Fig. 3),
WSDL document. the SOAP message is obtained by parsing the WSDL
The WSVTS was implemented in Visual C] on file of the Web services being tested. Then, using
the Microsoft .NET platform and contains four main an XML analysis technique, the number and type of
function modules: (a) the SOAP message generator; SOAP message parameters are extracted; based on this
(b) the SOAP message mutation generator; (c) the test information, the appropriate TCFN algorithm is called
case generator; and (d) the Web service vulnerability to generate test cases. Web services are tested based
analyzer. The details of these major modules are on the testing controller and client driver, using the
presented below. generated test cases. Finally, the vulnerability testing
(a) SOAP message generator report is obtained based on observations of the response
The input to the SOAP message generator is a WSDL messages received from the client of the Web services
file of the Web service being tested and consists of the being tested.
response message data type, the transmission protocol, In the experiments, some specifically written

Fig. 2 The WSVTS framework.


Jinfu Chen et al.: Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages 437

During the experimental process, the function of


the IPO mutation operator was merged with that of
either the IIV or PFB mutation operator to generate test
cases, according to specific circumstances and SOAP
message types. Different mutation operators may find
the same error for the same Web service, in which
case, the error was counted only once. Similarly, the
same fault found by different test cases generated by
the same perturbation operator was also only counted
once. Operator Efficiency (OE) defines the efficiency of
an operator in terms of finding faults and is calculated as
OE = EF / TC, where EF is the number of faults found
and TC is the total number of test cases generated by
the operators. The efficiency of the mutation operators
is shown in Fig. 4. Different mutation operators have
different efficiencies; the FVS operator has the highest
Fig. 3 Flow chart of Web services vulnerability testing. (36.52%).
5.2 Experimental results and analysis
services were analyzed in addition to several open
Web service vulnerabilities were found by the
Web services. The list of tested Web services is
proposed approaches. Although the proposed mutation
shown in Table 2. The columns titled “Number
operators are applicable to related approaches, test case
of service methods” and “Number of method
generation rules may differ. Also, continuous types of
parameters,” respectively, represent the number of
test case generation are more complex than discrete
total public service methods and the number of total
types, and test cases for continuous types can be adapted
method parameters of Web services. The column
to discrete types but not vice versa. We next compare
titled “Description” describes the use of the Web
our proposed approach with two others: SOAPUI[29]
services. The column “Faults seeded” records the
and SMAT-WS[7] .
total vulnerability faults we manually seeded into the
5.2.1 Comparison of WSVTS and SOAPUI
code. The number of total faults seeded to all subjects
A total of 20 kinds of specially designed Web services
is 408.
Table 2 The tested Web services.
Number of Number of
No. Service name Description Faults seeded
service methods method parameters
WS1 Stock 8 23 Searching stock information 16
WS2 Weatherforecast 7 19 Weather forecast service 12
WS3 E-Banking 9 25 Online banking service 35
WS4 Bookfinding 6 15 Searching book information 19
WS5 Domainfinding 5 13 Searching domain and IP address 29
WS6 Petinformation 7 16 Searching pet information 18
WS7 Traintime 7 14 Searching train timetable 13
WS8 Planetime 5 12 Searching aircraft flight information 15
WS9 QQcheckonline 7 13 Searching QQ online information 16
WS10 Queryresults 9 22 Searching student achievement information 31
WS11 Producedorder 8 16 Searching production order information 22
WS12 Calculator 7 15 Arithmetic calculating service 25
WS13 Maxdivisor 5 10 Finding the greatest common divisor 16
WS14 Mod 4 8 Finding the remainder of two numbers 21
WS15 Reversestring 8 14 Reversing the string 15
WS16 Stringcopy 6 12 Copying the string 20
WS17 Stringlength 4 8 Obtaining the length of the string 13
WS18 Login 5 8 User login 24
WS19 Vote 5 16 Getting the vote results 23
WS20 Echoinformation 6 13 Echoing personal information 25
438 Tsinghua Science and Technology, October 2014, 19(5): 429-441

Table 4 Test results of the WSVTS tool.


Number of test
Mutation operators Faults found
cases generated
DNS 42 10
SVN 118 19
EON 113 11
EOV 97 11
SVB 123 38
SSI 33 2
LSV 164 28
IIV 126 45
FVS 115 42
CIV 150 38
Fig. 4 Efficiency of the mutation operators. RSV 99 31
FIV 88 28
UVF 35 11
were investigated using these two approaches, based
PFB 118 23
on SOAP message parameter type. The SOAPUI[29]
Total 1421 337
is an open-source Web service testing tool, and
WSVTS is a testing tool based on the approach
proposed in this paper. Table 3 shows the experimental
results for open-source tool SOAPUI, in which the
test cases are manually entered according to SOAP
message parameter type. Table 4 shows the results for
WSVTS. Based on these results, the OVerall Efficiency
(OVE) of the mutation operators generated by the two
approaches are calculated to be approximately 21.1%
and 23.7%, respectively, confirming the feasibility of
our proposed approach and the validity of the test cases
generated.
Figure 5 gives a comparison of the efficiency of the
two approaches and shows that, for most operators, the

Table 3 Test results of the SOAPUI tool. Fig. 5 Comparison of the WSVTS and SOAPUI efficiencies.

Number of test
Mutation operators Faults found number of faults found by the WSVTS approach is
cases generated
DNS 42 8 higher than that found by the SOAPUI tool (exceptions
SVN 124 19 being the EON, FVS, RSV, and PFB operators). The
EON 113 11 UVF operator appears particularly efficient. The faults
EOV 97 9 that were found consist of some common vulnerability
SVB 130 38 faults, such as memory leak, buffer overflow, cross-
SSI 40 2 boundary access, and arithmetic security faults —
LSV 211 28 including dividing by zero and out-of-range operand
IIV 151 45
values. Thus, the designed operators and our approach
FVS 115 42
are confirmed to be very effective.
CIV 197 36
RSV 98 31
5.2.2 Comparison of SMAT-WS, WSVTS, and
FIV 102 28 SOAPUI
UVF 41 7 Research on SOAP message mutation testing is still
PFB 118 30 uncommon. The experimental results of SOAP message
Total 1579 334 perturbation reported by Almeida and Vergilio[7] is
reproduced here in Table 5. Their proposed mutation
Jinfu Chen et al.: Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages 439

operators are different from ours because of the


different Web services studied; therefore, we compare
the approaches based on the overall efficiency of the
mutation operators. The overall effectiveness of the
test cases generated by the SMAT-WS testing tool is
15.7%. A comparison of the overall efficiency for all
three methods is shown in Fig. 6. The experimental
results in Fig. 6 show that the overall efficiency of
WSVTS is the highest and SMAT-WS has the smallest
overall efficiency of the three methods.
Fig. 7 Comparison of the SOAPUI and WSVTS tools.
In addition, a comparison of the SOAPUI and
WSVTS tools is also shown in Fig. 7. The experimental
proposed mutation operators are different because the
results in Fig. 7 show that the fault-finding abilities
Web services tested with SMAT-WS are different from
of the WSVTS and SOAPUI approaches are similar
those tested by SOAPUI and WSVTS. In general, the
at the earlier stages of testing but that the rate of
number of test cases generated is different because of
faults found by WSVTS increases faster, supporting
the different mutation operators applied to different
the validity of this approach. Although the three
situations as well as the different number of faults.
approaches (SMAT-WS, SOAPUI, and WSVTS) are all
Compared with the other methods, the advantages
based on SOAP message mutation, the corresponding
of the WSVTS tool include the fact that the mutation
Table 5 SMAT-WS test results. Mutation operators: operators expand according to the characteristics of the
Incomplete (I), Null (N), Boundary Extension (BE), Inversion SOAP message in the experiment — in other words,
(IN), Value Inversion (VI), Space (S), Unauthorized (U), the testing is more comprehensive — and the algorithm
Mod Len (ML), Boundary (B). is automatically called to generate test cases according
Number of test to the number of parameters and the SOAP message
Mutation operators Faults found
cases generated type. The targeted faults consist of buffer overflow
I 54 16 faults, cross-boundary access faults, and arithmetic
N 54 21 security faults.
BE 363 24
IN 43 2 6 Conclusions
VI 45 8
Research on Web service vulnerability testing remains
S 54 19
B 162 27 limited, partly due to the services’ cross-platforms
U 54 16 and differing characteristics. In this paper, we have
ML 108 15 presented mutation operators designed for SOAP
Total 937 148 messages and a mutation testing algorithm for the
automated generation of test cases.
By designing appropriate SOAP message mutation
operators, the security of Web services can be tested
from the client side and vulnerability faults can
be identified from the user’s perspective. In most
cases, compared with the classic farthest neighbor
algorithm, the proposed TCFN algorithm reduces the
number of distance calculations. Compared with pure
random testing, the proposed TCFN algorithm can
detect more faults with fewer test cases. Because
specifically tailored test cases can be generated, the
efficiency and quality of test case generation can be
Fig. 6 Comparison of the overall efficiency for all three improved. Furthermore, the test cases can also be
methods. generated automatically, using legal and illegal input
440 Tsinghua Science and Technology, October 2014, 19(5): 429-441

parameters and mutation operators. The effectiveness Journal of Systems Architecture, vol. 57, no. 3, pp. 259-
of the proposed approach has been shown to be higher 268, 2011.
[9] S. Bekrar, C. Bekrar, R. Groz, and L. Mounier, Finding
than that of other available approaches. The efficiency
software vulnerabilities by smart fuzzing, in Proceedings
of the proposed mutation operators is higher than of the Fourth IEEE International Conference on Software
other approaches, such as SMAT-WS. In addition, the Testing, Verification and Validation, Berlin, Germany,
approach can detect more vulnerability faults than other 2011, pp. 427-430.
[10] J. Offutt and W. Xu, Generating test cases for web
approaches with the same test cases.
services using data perturbation, ACM SIGSOFT Software
In the future, we would like to continue research Engineering Notes, vol. 29, no. 5, pp. 1-10, 2004.
in the following areas. First, we will do more [11] A. C. V. de Melo and P. Silveira, Improving data
experiments to verify the reliability of the proposed perturbation testing techniques for web services,
approaches. Second, we will research how to further Information Science, vol. 181, no. 3, pp. 600-619, 2011.
[12] P. Silveira and A. C. V. de Melo, Exploring XML
reduce redundant test cases after mutating. Third, perturbation techniques for web services testing, Lecture
the automatic process of test case generation and Notes in Computer Science, vol. 5648, pp. 355-369, 2009.
mutation need to be further improved to enhance testing [13] J. F. Chen, Q. Li, C. Y. Mao, D. Towey, Y. Z. Zhan, and
efficiency. H. H. Wang, A web services vulnerability testing approach
based on combinatorial mutation and SOAP message
Acknowledgements mutation, Service Oriented Computing and Applications,
This work was partly supported by the National Natural vol. 8, no. 1, pp. 1-13, 2014.
[14] L. Novak and A. Zamulin, A formal model for
Science Foundation of China (Nos. 61202110 and XML schema, in Proceedings of the 21st International
61063013) and the Natural Science Foundation of Jiangsu Conference on Data Engineering Workshops, Tokyo,
Province (No. BK2012284). Japan, pp. 1283-1293, 2005.
[15] W. Xu, J. Offutt, and J. Luo, Testing web services
References by XML perturbation, in Proceedings of the 16th
IEEE International Symposium on Software Reliability
[1] S. Hanna and M. Munro, An approach for wsdl-based
Engineering, Chicago, USA, pp. 257-266, 2005.
automated robustness testing of web services, presented at [16] J. F. Chen, Y. S. Lu, and X. D. Xie, Component security
the 16th International Conference on Information Systems testing approach by using interface fault injection, Journal
Development, Nanchang, China, 2009, pp. 1093-1104. of Chinese Computer System, vol. 31, no. 6, pp. 1090-1096,
[2] T. Takase and K. Tajima, Efficient web service message
2010.
exchange by SOAP bounding framework, in the 11th IEEE [17] S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen,
International Enterprise Distributed Object Computing, W. Grieskamp, M. Harman, M. J. Harrold, and P. McMinn,
Annapolis, MD, USA, 2007, pp. 63-72. An orchestrated survey of methodologies for automated
[3] L. Wu, X. K. Li, and H. Wang, Research on the reliability software test case generation, Journal of Systems and
testing of web service based on fault injection technology, Software, vol. 86, no. 8, pp. 1978-2001, 2013.
Journal of Chinese Computer System, vol. 28, no. 1, pp. [18] T. Y. Chen, F. C. Kuo, H. Liu, and W. E. Wong, Code
127-131, 2007. coverage of adaptive random testing, IEEE Transactions
[4] M. Palacios, J. Garcia-Fanjul, and J. Tuya, Testing in on Reliability, vol. 62 no. 1, pp. 226-237, 2013.
service oriented architectures with dynamic binding: A [19] A. Shahbazi, A. Tappenden, and J. Miller, Centroidal
mapping study, Information and Software Technology, vol. voronoi tessellationsCa new approach to random testing,
53, no. 3, pp. 171-189, 2011. IEEE Transactions on Software Engineering, vol. 39, no.
[5] C. A. Sun, G. Wang, B. H. Mu, H. Liu, Z. S. Wang, and T. 2, pp. 163-183, 2013.
Y. Chen, A metamorphic relation-based approach to testing [20] C. Bohm, S. Berchtold, and D. A. Keim, Searching in high
web services without oracles, International Journal of Web dimensional spaces: Index structures for improving the
Services Research, vol. 9, no. 1, pp. 51-73, 2012. performance of multimedia databases, ACM Computing
[6] C. A. Sun, G. Wang, B. H. Mu, H. Liu, Z. S. Wang, Surveys, vol. 33, no. 3, pp. 322-373, 2001.
and T. Y. Chen, Metamorphic testing for web services: [21] T. Y. Chen, F. C. Kuo, R. G. Merkel, and T. H. Tse,
Framework and a case study, presented at the IEEE Adaptive random testing: The ART of test case diversity,
International Conference on Web Services, Washington Journal of Systems and Software, vol. 83, no. 1, pp. 60-66,
DC, USA, 2011, pp. 283-290. 2010.
[7] L. F. de Almeida and S. R. Vergilio, Exploring perturbation [22] M. H. Alsuwaiyel, Algorithms: Design Techniques and
based testing for web services, presented at the IEEE Analysis. World Scientific Pub Co Inc, November 1998.
[23] K. P. Chan, T. Y. Chen, and D. Towey, Adaptive random
International Conference on Web Services, Chicago, USA,
testing with filtering: An overhead reduction technique,
2006, pp. 717-726. presented at the 17th International Conference on Software
[8] H. C. Kim, Y. H. Choi, and D. H. Lee, Efficient file Engineering and Knowledge Engineering, Taipei, China,
fuzz testing using automated analysis of binary file format, pp. 292-299, 2005.
Jinfu Chen et al.: Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages 441

[24] K. P. Chan, T. Y. Chen, and D. Towey, Restricted [27] B. H. Li and Z. X. Hao, Efficient filtration and query
random testing: Adaptive random testing by exclusion, algorithm of reverse furthest neighbor, Journal of Chinese
International Journal of Software Engineering and Computer Systems, vol 30, no. 10, pp. 1948-1951, 2009.
Knowledge Engineering, vol. 16, no. 4, pp. 553-584, 2006. [28] J. M. Voas and K. W. Miller, Predicting software’s
[25] T. Y. Chen, F. C. Kuo, and C. A. Sun, Impact of the minimum-time-to-hazard andmean-time-to-hazard for rare
compactness of failure regions on the performance of input events, presented at the 6th International Symposium
adaptive random testing, Journal of Software, vol. 17, no. on Software Reliability Engineering, Toulouse, France,
12, pp. 2438-2449, 2006. 1995, pp. 229-238.
[26] I. N. Bronshtein, K. A. Semendyayev, G. Musiol, and H. [29] SoapUI, SmartBear software, http://www.soapui.org,
Mhlig, Handbook of Mathematics. Springer, 2007. 2012.

Jinfu Chen received the BEng degree from Chengying Mao received the BS degree
Nanchang Hangkong University, China, from Central South University, China, in
in 2004, and PhD degree from Huazhong 2001, and the PhD degree in computer
University of Science and Technology, software and theory from Huazhong
China, in 2009, both in computer science. University of Science and Technology,
He is currently an associate professor China, in 2006. He worked as a post-doc in
in the School of Computer Science and the College of Management of Huazhong
Communication Engineering of Jiangsu University of Science and Technology
University. His major research interests include software from July 2006 to September 2008. He is an associate professor
engineering, services computing, and information security. of the School of Software and Communication Engineering
He is a member of the ACM, IEEE CS, and China Computer in Jiangxi University of Finance and Economics, China.
Federation. His current research interests include service computing and
software engineering. He is a member of the ACM, IEEE, and
IEEE CS.
Huanhuan Wang is a software testing
engineer with ZTE Corporation. She
Rubing Huang is an assistant professor in
received her BEng degree from Qufu
the Department of Software Engineering,
Normal University, China, in 2009, and
School of Computer Science and
MS degree from Jiangsu University, China,
Communication Engineering, Jiangsu
in 2012, both in computer science. Her
University, China. He received his
research interests include software testing
PhD degree in computer science and
and service computing.
technology from Huazhong University
of Science and Technology, China, in
2013. His current research interests include software testing
Dave Towey is an assistant professor at and software maintenance, especially combinatorial interaction
the School of Computer Science, The testing, random testing, adaptive random testing, and test case
University Nottingham Ningbo China, prioritization. He is a member of the IEEE, the ACM, the IEICE,
prior to which he was with Beijing Normal and the IEEE Communications Society.
University–Hong Kong Baptist University:
United International College, China. He
received his BA and MA degrees from the Yongzhao Zhan is a professor at
University of Dublin, Trinity College in the School of Computer Science and
1997 and 2000, respectively, and a PhD degree in computer Communication Engineering, Jiangsu
science from The University of Hong Kong in 2006. His University. He received his PhD degree
research interests include software testing, software design, and from Nanjing University in 2000. He does
technology in education. He is a member of both the IEEE and research on distributed systems, image,
the ACM. and video retrieval. He is a member of the
ACM, IEEE, and IEEE CS.

Vous aimerez peut-être aussi