Vous êtes sur la page 1sur 12

This article has been accepted for inclusion in a future issue of this journal.

Content is final as presented, with the exception of pagination.

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 1

Automatic Pavement Crack Detection by


Multi-Scale Image Fusion
Haifeng Li , Dezhen Song , Senior Member, IEEE, Yu Liu, and Binbin Li

Abstract— Pavement crack detection from images is a challeng- significant transportation system failure (e.g. airport run-
ing problem due to intensity inhomogeneity, topology complexity, ways or bridge decks), and need to be detected accurately
low contrast, and noisy texture background. Traditional learning- and timely. Therefore, it is necessary to develop an automatic
based approaches have difficulties in obtaining representative
training samples. We propose a new unsupervised multi-scale crack detection method for pavement inspection. However,
fusion crack detection (MFCD) algorithm that does not require automatic crack detection is challenging due to intensity inho-
training data. First, we develop a windowed minimal intensity mogeneity, topology complexity, low contrast, and noisy back-
path-based method to extract the candidate cracks in the image grounds [4], [5]. Since a pavement crack is usually thin and
at each scale. Second, we find the crack correspondences across long, the most distinguishable characteristic is its continuous
different scales. Finally, we develop a crack evaluation model
based on a multivariate statistical hypothesis test. Our approach property. As a result, local image-processing methods, such as
successfully combines strengths from both the large-scale detec- intensity thresholding [5], [6], and edge detection based meth-
tion (robust but poor in localization) and the small-scale detec- ods [7], [8] can only obtain a set of disjoint crack fragments
tion (detail-preserving but sensitive to clutter). We analyze and with a high false positive rate. Machine learning approaches
experimentally test the computational complexity of our MFCD have been proposed, but the selection of parameters depends
algorithm. We have implemented the algorithm and have it
extensively tested on three public data sets, including two public on pavement appearance, crack variations, and image quality.
pavement data sets and an airport runway data set. Compared Moreover, the results from these learning methods depend on
with six existing methods, experimental results show that our the quality of manually-labeled training data sets. It is labor-
method outperforms all counterparts. Specifically, it increases expensive and not easy to obtain the representative training
the precision, recall, and F1-measure over the state-of-the-art by data, especially in the real applications, because crack images
22%, 12%, and 19%, respectively, on one public data set.
have large variations due to different lighting conditions, sur-
Index Terms— Crack detection, multi-scale image fusion, face types, and background texture. Morphological methods,
pavement inspection, robotic airport runway inspection. such as the seed-based approaches [9], [10] and minimal path
selection methods [11]–[13], have been successfully used in
I. I NTRODUCTION pavement crack detection by exploiting the connectivity among
crack pixels. However, their performance usually depends
C RACKS are common pavement surface distresses that
affect road performance and periodical road surveys
are necessary to assess pavement surface conditions [1]–[3].
on the parameter choice which requires manual extensive
parameter tuning for each dataset.
In fact, crack images exhibit different characteristics at
Traditional manual crack detection methods are very time-
different scales: at a large scale, crack detection is reliable, but
consuming, labor-intensive, with low-accuracy, and error
its localization is poor and may miss thin cracks; at a small
prone. Even thin cracks may be early warning signs of
scale, details are preserved, but detection suffers greatly from
Manuscript received November 8, 2017; revised May 14, 2018; accepted clutters in background texture. However, the key challenge is
July 12, 2018. This work was supported in part by the National Sci- how to effectively combine the strengths of different scales
ence Foundation under Grant NRI-1426752, Grant NRI-1526200, and Grant to improve crack detection performance. To deal with the
NRI-1748161, in part by the National Science Foundation of China under
Grant 61305107, in part by the Industrial Robot Application of Fujian challenges, we propose a new unsupervised multi-scale fusion
University Engineering Research Center, Minjiang University, under Grant based crack detection (MFCD) algorithm that does not require
MJUKF-IRA201803 and Grant MJUKF-IRA201807, in part by the Open Fund manually-labeled training data (See Fig. 1). MFCD computes
Project of Fujian Provincial Key Laboratory of Information Processing and
Intelligent Control, Minjiang University, under Grant MJUKF201732, in part the maximum average score of cracks at different scales.
by the Fundamental Research Funds for the Central Universities under Grant By extracting crack features for every probable target scale
3122016B006, and in part by the Chinese Scholarship Council. The Associate and evaluating the cracks jointly across scales, MFCD fuses
Editor for this paper was H. G. Jung. (Corresponding author: Haifeng Li.)
H. Li and Y. Liu are with the CS Department, Civil Aviation University of and filters cracks at all scales.
China, Tianjin 300300, China, and also with the Fujian Provincial Key Labo- We have implemented our MFCD algorithm and extensively
ratory of Information Processing and Intelligent Control, Minjiang University, tested it on three datasets in comparison to six existing meth-
Fuzhou 350108, China (e-mail: hfli@cauc.edu.cn).
D. Song and B. Li are with the CSE Department, Texas A&M Uni- ods. Experimental results show that our method consistently
versity, College Station, TX 77843 USA (e-mail: dzsong@cse.tamu.edu; outperforms the counterparts. More specifically, our MFCD
binbinli@tamu.edu). algorithm increases Precision, Recall and F1-measure over the
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org. state-of-the-art by 22%, 12% and 19%, respectively, on one
Digital Object Identifier 10.1109/TITS.2018.2856928 public dataset.
1524-9050 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

2 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Machine learning techniques [14]–[16] have become more


popular in recent years which include methods that build on
techniques such as support vector machines [4], [17], [18],
random forest [19], random structured forest [20], and neural
networks [21], [22]. In these learning methods, a pavement
image is often divided into a number of sub-images, each of
which is represented by a vector of features extracted from this
sub-image. These sub-images are then used for training and
classification for crack detection. However, since the training
and classification are conducted at each sub-image and as
local methods, they cannot exactly segment out crack curves
Fig. 1. An illustration of our MFCD algorithm outline. Given an original over the whole image because it often only provides a label
image, we apply Gaussian blur to generate multi-scale images with different
standard deviations σ1 < σ2 < . . . < σn representing different scales, then
to the sub-image as its output. Furthermore, a supervised
detect cracks at each scale, and finally fuse and filter cracks to obtain results. training stage is needed which requires accurately labeled data,
(a) Original image. (b) Generated multi-scale images. (c) Detected cracks at a difficult requirement for applications with large lighting and
each scale. (d) Cracks from multi-scale fusion.
scene variations.
Morphological methods [12], [23], [24] exploit the connec-
tivity among crack pixels and have been successfully used
The novelties and contributions of our paper are the
in pavement crack detection research. However, their perfor-
following:
mance are usually dependent upon the parameter choices [25]
• We propose a multi-scale crack fusion algorithm, MFCD,
which require manual extensive parameter tuning for each data
that successfully combines the strengths from both the
set. Our method is a variation of morphological methods but
large-scale detection (robust but poor in localization) and
focusing on removing arbitrary threshold selection and being
the small-scale detection (detail-preserving but sensitive
self-adaptive to different images because we employ statistic
to clutter), without any labeled training data.
parameters in the threshold setting.
• The statistical parameters and self-adaptive thresholds,
More specifically, our method builds on the existing
instead of arbitrary thresholds, are adopted in our method.
minimal intensity path based techniques which find the
The statistical parameters can guarantee the probability
best paths between pairs of endpoints of potential cracks.
to obtain the correct results, and the adaptive thresh-
Gavilán et al. [9] propose a seed-based approach by com-
olds make our method feasible in a wide range of
bining multiple directional non-minimum suppression with a
applications.
symmetry deck, where seeds are linked by computing paths
• The proposed method has shown its effectiveness and
with the lowest mean pixel intensities that meet the symmetry
superior performance in a wide range of experimental
restrictions. Kaul et al. [26] propose a method to detect the
setup and test data sets, as evident in the experimental
same types of contour-like image structures with less prior
results. No existing methods have been tested at this scale
knowledge about both the topology and the endpoints of
and variety.
the desired curves. Nguyen et al. propose a method, named
The rest of paper is organized as follows: we summarize
FFA, which takes into account brightness and connectivity
the related work in Section II before we introduce our crack
for crack detection simultaneously by introducing Free-Form
detection problem in Section III. We detail our algorithm
Anisotropy [13], [27]. FFA assumes that each crack is orga-
design in Section IV and analyze algorithm performance in V.
nized along a preferred direction, so that it can be tracked for
We test our algorithm in experiments in Section VI and
all possible pairs of endpoints. To avoid false detections caused
conclude our paper in Section VII.
by low intensity loops and remove the restriction on the shape
of the cracks, Amhaz et al. [11], [12] propose a two-stage
II. R ELATED W ORK minimal intensity path selection algorithm, named MPS, which
Popular image-based crack detection methods can be classi- first selects endpoints at the local scale and then selects min-
fied as four types: intensity thresholding methods, edge detec- imal intensity paths at the global scale. MPS method tries to
tion methods, machine learning techniques, and morphological find the path consisting of a series of neighboring pixels whose
methods. score is the sum of their intensities. Crack seeds are firstly
Intensity thresholding methods [5], [6] have been widely detected in grid cells by thresholding the pixel intensities,
studied due to their simplicity. These methods are sensitive to then paths are generated between each pair of adjacent crack
noise, leading to unreliable crack detection results especially seeds by searching in the whole image. Thereafter, two post-
for field images with significant visual clutters under poor processing steps, including elimination of artifacts and width
lighting conditions. Furthermore, selecting the appropriate detection, are introduced in MPS to improve the quality of the
threshold value is challenging. detection. Built on this approach and to improve the accuracy
Edge detection methods [7], [8] are also widely adopted. and computation speed, our new method employs statistical
However, the main drawback is that edge detection methods parameter selection in replacement of intensity parameters
can only detect a set of disjoint crack fragments and often fail to improve the robustness, adopts crack seeds clustering and
in low-contrast and high-clutter images. removal step to reduce the searching region in the following
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

LI et al.: AUTOMATIC PAVEMENT CRACK DETECTION BY MULTI-SCALE IMAGE FUSION 3

Fig. 2. An illustration of our windowed minimal intensity path based crack detection method. (a) Original pavement image. (b) Crack seed extraction results.
(c) Crack seed clustering and filtering illustration. (d) WMIP generation result. (e) Path verification results. (f) Crack region detection by path growing.

path generation, and designs the windowed path generation algorithm as illustrated in Fig. 1. First, we generate the multi-
algorithm to decease the computing time. scale images by Gaussian blurring with different kernel sizes.
Our MFCD algorithm is also inspired by multi-scale analy- Second, we find the candidate cracks at each scale by utilizing
sis methods which capture the intrinsic geometrical structure a windowed minimal intensity path selection based method.
that is key in human visual perception. Existing techniques, Finally, we determine the cracks correspondence across dif-
such as wavelet transforms [28], particle filters [29], beamlet ferent scales and propose a statistical crack evaluation model
transform [30], are essential different types of multi-scale to compute the average score of each crack at different scales
methods. The main challenge of these multi-scale analysis for crack fusion.
methods is to select the right scale for identifying useful
features or how to combine the detections at different scales
A. Multi-Scale Images Generation
to form output. Most approaches take a simplistic cue com-
bination: they either accept results (after thresholding) at all Using Gaussian blurring technique in computer vision [31],
scales, or accept results that appear at the coarsest scale. Our the image at the s-th scale level, I s , can be produced from the
method fuses results from different scales using a statistical convolution of a variable-scale Gaussian, G(u, v, σs ), with an
hypothesis test. input image I 0 .
1 −(u 2 +v 2 )/2σs2
III. P ROBLEM F ORMULATION G(u, v, σs ) = e . (1)
2πσs2
A. Assumption
Thus, a set of images with different scales are obtained with
We assume a crack has a lower intensity value than that different σs , which serve as the input to the following steps.
of the background image. This assumption can be accepted
in most general cases since cracks always absorb more light
than other areas and often appear as dark curves or tapes in B. Candidate Crack Extraction at Each Scale
the image. We also assume that pixels belonging to the same As shown in Fig. 2, we propose a Windowed Minimal
crack form a continuous path with an arbitrary shape. Intensity Path (WMIP) method to find candidate cracks at each
scale.
B. Notations 1) Crack Seed Extraction (Fig. 2(b)): Let g(x) be the
intensity value of pixel x, and Te be a threshold. We divide
Common notations are defined as follows,
the whole image I s into grid cells, denoted as Yis , i =
• I s , s = 0, 1, . . . , n s − 1, the digital image at the s-th
1, 2, . . . , n y . Each cell is a set of m × m pixels. In each grid
scale with I 0 denoting the original image. All images are
cell Yis , we select a pixel xis as a crack seed when the following
in grayscale.
two conditions are satisfied: 1) xis is the darkest pixel in Yis ,
• xis = [u, v]T , the i -th pixel in I s , with (u, v) being image
and 2) xis is within the top Te percent darkest pixels in I s .
coordinates.
• C s , the pixel set for the candidate cracks in I s . xis = argmin g(x),
• C ∗ , the detected crack pixel set as the algorithm output. x
s.t. x ∈ Yis , g(x) < ge , (2)
C. Problem Definition where ge is the pixel intensity value of the top Te percent
Our ultimate goal is to extract all cracks from an input pave- darkest pixels in I s .
ment image by multi-scale crack fusion, as shown in Fig. 1. As the output of the step, let us denote E s as the set of
Thus, our crack detection problem is defined as follows, crack seeds.
Definition 1 (Crack Detection): Given I 0 , generate multi- 2) Crack Seed Clustering and Filtering (see Fig. 2(c)):
scale images I s , s = 0, 1, . . . , n s − 1, extract the candidate With E s obtained, we use DBSCAN algorithm [32] to group
cracks C s from each I s , then fuse C s to obtain C ∗ . crack seeds into clusters, meanwhile, find the isolated crack
seeds that need to be removed. DBSCAN is a density-based
IV. M ULTI -S CALE PAVEMENT C RACK D ETECTION clustering algorithm, which can group together points that are
Pavement cracks in an image tend to have one or more closely packed and identify isolated outliers in low-density
particular salient scales. To combine the strengths of multiple regions. The advantages of DBSCAN include: 1) it does not
scales, we propose a three-step multi-scale crack detection require users to specify the number of clusters in the data
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

We also define Pi,s j := {Pi,s j |Pi,s j ⊆ Wi,s j } as a set of


all possible paths in the window set Wi,s j where Wi,s j is
the rectangular window with xis and xsj as the two opposite
vertices. The rectangle is iso-oriented and has it sides parallel
to coordinate axes.
For each pair of crack seeds xis and xsj ∈ Dis , the WMIP is
the path minimizing the sum of the intensities of pixels along
path (see Fig. 3(b)). Let Pi,s j be the WMIP, then

Pi,s j = argmin g(xas ), (5)
P∈Pi,s j xs ∈P
a
Fig. 3. WMIP generation. (a) Quadrant definition in crack seed cluster.
(b) WMIP generation. where xas represents a pixel in a candidate path P. We only
search the path within the window instead of the whole
image, because the windowed path searching can provide an
a priori; 2) it is robust to outliers; and 3) it only requires two approximate solution but reduce the computing time greatly.
parameters and is insensitive to the ordering of the points. In fact, as indicated in the tests, our windowed path searching
To find a cluster, DBSCAN starts with any crack seed, significantly reduces computing time while results remain
cis ∈ E s , and retrieves all crack seeds density-reachable similar to the algorithm performing path searching in the
from cis with regard to two parameters, ε and Mi n Pts. The whole image.
ε-neighborhood of csj is defined as, We apply Dijkstra algorithm [33] to solve the optimization
   problem in (5). Finally, we obtain the set of WMIPs, denoted

Nε (cis ) = csj cis − csj  ≤ ε, ∀csj ∈ E s , (3) as P s , for all (i, j ) pairs.
4) Path Verification (Fig. 2(e)): Not all WMIPs in P s
where Nε (cis ) contains at least Mi n Pts of crack seeds. In our
represent real cracks. To remove false WMIPs from P s , a ver-
experiments, we set ε = 16 and Mi n Pts = 4.
ification step is performed by calculating the mean intensity,
Denote the generated clusters as G n , n = 1, 2, . . . , n c . This
m(Pi,s j ), of each Pi,s j ∈ P s ,
clustering step can help us reduce the searching region while
finding the WMIP between crack seeds in the following step. 
g(xas )
3) WMIP Generation (Fig. 2(d)): For each crack seed xis , xas ∈Pi,s j
we define Dis be the set of crack seeds which need to be m(Pi,s j ) = , (6)
|Pi,s j |
connected with xis . We first find Dis and then employ a WMIP
based method to connect xis with each crack seed in Dis . where set cardinality operator | · | counts the number of pixels.
To determine Dis , we define the local coordinate system of If the mean intensity of a WMIP is bigger than a given
xi at first, with xis as the origin and two axes parallel to the
s
threshold, gm , we consider the case as false-positive detection
two axes of I s , respectively, as shown in Fig. 3(a). For crack and discard the WMIP. Threshold gm is set as the intensity
seed xis , we connect it with another crack seed xsj if a) they value of the top Tv percent darkest pixels in I s .
are in the same cluster, b) their geometric distance is smaller 5) Crack Region Detection by Path Growing (Fig. 2(f)):
than the threshold T p , and c) xsj is the nearest crack pixel to The previous steps may miss some outside pixels due to
xis in its own quadrant, which means each crack seed can be focusing on connectivity. This step is to absorb neighboring
connected with 4 other crack seeds at most. An example is dark pixels to grow paths. The neighboring relationship is not
shown in Fig. 3(a), where xis is connected with three crack limited to adjacent pixels, two pixels with distance smaller
seeds: xsj , xas and xbs . Thus, Dis = {xsj , xas , xbs }. We do not than a threshold are considered as neighbors. Define L(xis ) :=
connect xis and xcs because their distance is bigger than T p . {xsj |g(xsj ) < g(xis )} be the set of pixels whose intensities are
For xis and each crack seed xsj ∈ Dis , we use WMIP less than g(xis ), κ be the total amount of pixels in I s , Cis be
to connect them. We consider an image as a bidirectional the i -th pixel set for the candidate crack in I s . Initially, we set
weighted graph of pixels. Pixels are vertices. If and only if Cis = Pi,s j . Denote Gis as the new found crack pixel set which
two pixels xas and xbs are adjacent in the image, we build a is generated from Cis . Gis can be computed as
bidirectional edge between them, denoted as (xas , xbs ). Recall   
s |L(xa )|
s
the pixel intensity value of x is g(x). The edge weight from s s s s s
Gi := xa xa − xb  < r, xb ∈ Ci , < Tr , (7)
vertex xas to xbs is g(xbs ), while edge weight from xbs to xas is κ
g(xas ). g(xas ) and g(xbs ) are often not the same value. where r is the distance tolerance for path growing, Tr is a
We denote Pi,s j as a path connecting xis and xsj in the form threshold. We set r = 4 in practice. After obtaining Gis from
of a sequence of pixels, Cis , we add Gis into Cis .
Pi,s j := {xis , xks , xk+1
s s
, . . . , xk+n , xsj }, (4) Cis = Cis ∪ Gis . (8)
such that ||xis − xks || = ||xk+n
s
− xsj || = ||xk+m
s
− xk+m+1
s
|| = 1, This aggregation process is repeatedly performed, until there
for m = 0, 1, . . . , n − 1 due to adjacency. are no further pixels to add.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

LI et al.: AUTOMATIC PAVEMENT CRACK DETECTION BY MULTI-SCALE IMAGE FUSION 5

6) Crack Region Grouping: One real crack may correspond


to several crack regions in the image due to noise. We want to
group the detected crack regions according to their geometric
distances between each other. The distance between two
candidate crack pixel sets, Cis and C sj , can be computed as
di, j = min xas − xbs , s.t. xas ∈ Cis , xbs ∈ C sj . (9)
For a given threshold Tg , if di, j < Tg is satisfied, this
pair of crack regions, Cis and C sj , are labeled as the same
group. All crack regions may be divided into several groups. Fig. 4. An example output of score computation (Best viewed in color).
The grouped cracks serve as the inputs of the following crack (a) Original image. (b) Score.
matching step. For brevity, we use “one crack” to represent a
crack group in the rest of this paper.
After detecting candidate cracks at each scale, we are ready of each crack pixel to be a weighted Gaussian function. The
for multi-scale fusion. score of a pixel, xas which is defined as s(xas ),
(u a −u b )2 +(v a −v b )2
 1 − 2σ 2
C. Crack Correspondences Matching s(xas ) = wb × e p , (11)
2πσ p2
Now let us find the crack correspondences across different xbs ∈Ne (xas )
scales. If two cracks at different scales correspond to the same where σ p is the standard deviation of the Gaussian distri-
real crack, they are called as a pair of crack correspondence. bution, Ne (xas ) is the neighboring window with the size of
The problem of crack matching is defined as follows. (3σ p + 1) × (3σ p + 1) centering as xas , and wb is a weight.
Definition 2 (Crack Matching): Given C si and C s j , for We adopt the intensity weight in the score computation.
s s s
each Cisi ⊆ C si and C j j ⊆ C s j , find Cmsi ⊆ Cisi and Cn j ⊆ C j j , A pixel’s intensity weight is inversely proportional to its
s
such that Cmsi corresponds to Cn j . intensity value, i.e., a pixel with a lower intensity is given
si s
Then, for each Ci ⊆ C i and C j j ⊆ C s j , we find the
s
a greater intensity weight,
overlapping region with a distance tolerance as the matched
s |L(xbs )|
segments between Cisi and C j j . Let us define the distance from wb = 1 − . (12)
sj s κ
the pixel xb to crack Cisi as d(xbj , Cisi ),
Fig. 4 illustrates an example of score distribution where we
s s
d(xbj , Cisi ) = min xbj − xasi , s.t. xasi ∈ Cisi . (10) can see that the crack pixel has a larger score at this scale.
s With the score of pixel defined, we can compute the average
We want to find the overlapping region of Cisi and C j j score of crack Cis as follows,
within a distance tolerance as their correspondence. Initially, 
s s s
we set Cmsi = Cn j = ∅. Then, for each pixel xbj ∈ C j j , if the s(xas )
sj si sj s xas ∈C is
criteria, d(xb , Ci ) < Td , is satisfied, we add xb into Cn j , ρ(Cis ) = . (13)
where Td is a threshold. Similarly, for each point xa ∈ Cisi , si |Cis |
s s s s s
if d(xai , C j j ) < Td is satisfied, add xai into Cmi . When Cmi and With crack’s average score defined, we perform statis-
sj si s tical hypothesis test to remove false-positive cracks. From
Cn are unchanged, we obtain the correspondence Cm and Cn j .
The steps above are applied for each pair of cracks at (11) and (13), we know that ρ(Cis ) ∼ N(μs , σs2 ) is a random
different scales until all crack correspondences are found. variable following Gaussian distribution with mean μs and
In the rest of the paper, we renumber all cracks at each variance σs2 . Hence (ρ(Cis ) − μs )/σs ∼ N(0, 1) is a random
scale so that cracks with the same subscript are a group of variable following the normal distribution with zero mean
correspondences. and unit variance. Define Ci be the union pixel set of the
i -th matched cracks across all scales, thus
s −1
n
D. Crack Selection and Verification by Multi-Scale Fusion
Ci = Cis . (14)
The key issue here is how to optimally integrate all can- s=0
didate cracks C s to obtain C ∗ . First, we design the model
of average score to evaluate the probability of each detected We define function f (Ci ) to evaluate the probability of Ci
crack being from a true crack. Then, we develop statistical to be a real crack or not.
hypothesis test to remove false-positive cracks. s −1
n
ρ(Cis ) − μs 2
We propose a metric/scoring mechanism to evaluate a f (Ci ) = ( ) . (15)
σs
pixel’s probability to be a crack pixel. A dark pixel or a pixel s=0
with many dark neighboring pixels has higher probability to For each candidate crack Ci , we set up two hypotheses:
be a crack pixel than otherwise. Thus, the score for each
pixel is defined as the aggregation between itself and from H0 : Ci is a crack.
all neighboring pixels. We model the probability distribution H1 : Ci is not a crack.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

6 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Since f (Ci ) is the sum of squares of multiple normal Algorithm 1 WMIP Based Crack Detection
distributions according to (15), f (Ci ) follows χ 2 distribution input : I s
with n s degrees of freedom. Define F(x, n s ) be the cumulative output: C s
distribution function of χ 2 distribution with n s degrees of
1 Detect all crack seeds from I s to generate E s ; O(κ)
freedom, the probability of a value from χ 2 distribution larger
than x is 2 Cluster and filter E s using DBSCAN; O(κ log κ)
T 2 log T p
P{ f (Ci ) ≥ x} = 1 − F(x, n s ). (16) 3 foreach xis ∈ E s and xsj ∈ E s do O( p m 4 κ 2 )
4 if x j ∈ Di then
s s
By setting the significance level as α, we can obtain
5 Compute Pi,s j by solving (5) using Dijkstra
P{ f (Ci ) ≥ x} = α. (17) algorithm; O(T p2 log T p )
Since the function F is continuous and strictly monotoni- 6 Compute mean intensity m(Pi,s j ) using (6); O(T p2 )
cally increasing, combining (16) and (17), we can obtain
7 if m(Pi,s j ) < gm then
x = F −1 (1 − α, n s ), (18) 8 Discard Pi,s j ; O(1)
where F −1 (·) is the inverse function of F(·). Thus, we reject
9 foreach Pi,s j ∈ P s do O(κ log κ)
H0 if
10 repeat O(κ log κ)
f (Ci ) ≤ F −1 (1 − α, n s ). (19) 11 Compute Gis using (7); O(n i )
12 Cis = Cis ∪ Gis ; O(1)
After the statistical hypothesis test, we remove the false-
13 until Cis is unchanged;
positive cracks. Combine all the cracks remained to obtain C ∗ .
14 foreach Cis and C sj do O(|Cis ||C sj |)
V. A LGORITHM A NALYSIS 15 Compute di, j using (9); O(|Cis ||C sj |)
A. Complexity Analysis 16 if di, j < Tg then
We summarize the proposed WMIP based candidate crack 17 Merge Cis and C sj into the same group;
detection method in Algorithm 1 to facilitate our analysis. O(|Cis | + |C sj |)
Recall κ is the total amount of pixels in I s . For crack seed
18 foreach Cis ∈ C s do O(|Cis |)
extraction, since at most κ/m 2 cells are obtained, with each
19 if |Cis | < Ts then
cell containing m 2 pixels, the crack seeds can be detected in
20 Remove Cis from C s ; O(|Cis |)
O(κ/m 2 × m 2 ) = O(κ) time. Crack seed clustering takes
O(|E s | log |E s |) [32]. Obviously, |E s | ≤ κ/m 2 ≤ κ, thus, 21 return C s ; O(1)
crack seed cluster has a time complexity O(κ log κ). Since
we only find the WMIP within a rectangular window Wi,s j
whose diagonal length is smaller than T p , the maximum size
be considered as a constant. Thus, crack matching across two
of Wi,s j is 12 T p2 , thus, when proceeding the Dijkstra algorithm, s
scales has a time complexity O(|Cisi ||C j j |). Computing all
the number of vertex is smaller than 12 T p2 , and the number crack correspondences across each pair of adjacent scales takes
of edges is no more than 4T p2 . So the time complexity of s s
O(n s |Ci i ||C j j |) time. When computing the average score of
each WMIP generation is O(T p2 log T p ). There are at most any crack Cis , the scores of |Cis | pixels in Cis need to be cal-
κ 2 /m 4 paths in all because |E s | ≤ κ/m 2 . Thus, the time culated firstly, as shown in Eq. (13). The neighboring window
T 2 log T p
complexity for all WMIPs generation is p m 4 κ 2 . Since the for each pixel is (3σ p + 1) × (3σ p + 1). Thus, the overall
length of Pi,s j is no more than the size of Wi,s j , we have computation of average score of Cis takes O((3σ p + 1)2 |Cis |)
|Pi,s j | ≤ 12 T p2 . Therefore, we can compute m(Pi,s j ) in O(T p2 ) time. Since σ p is a constant, computing ρ(Cis ) has a time
time. Crack region detection by path growing takes O(κ log κ) complexity O(|Cis |). Since the total number of crack pixels
time. Computing di, j between two sets with size |Cis | and |C sj | in each I s is smaller than κ, merging cracks across n s scales
needs O(|Cis ||C sj |) time. Usually, the total number of crack takes in O(n s κ) time. Define the total number of Ci be n.
regions is very small and can be considered as constant. Thus, Considering |Ci | ≤ κ, the time complexity of hypothesis test
the time complexity of crack grouping is also O(|Cis ||C sj |). for all merged cracks is O(nκ). Thus, the crack selection and
s s
verification for each two scales takes in O(max(n i i , n j j )) time.
To summarize, since |Cis ||C sj | < κ 2 , the most computationally
expensive step in Algorithm 1 is WMIP generation using Since both |Cis | and n are much smaller than κ, the computa-
T p2 log T p 2
Dijkstra algorithm. Thus, the computational complexity of our tional complexity of our MFCD algorithm is O(n s m4
κ ).
T 2 log T p
WMIP based crack detection algorithm is O( p m 4 κ 2 ). Theorem 2: Our MFCD algorithm runs in
T p2 log T p 2
O(n s m 4 κ )
Theorem 1: The computational complexity of the proposed time.
T p2 log T p
WMIP based crack detection algorithm is O( m 4 κ 2 ).
We present our MFCD algorithm in Algorithm 2, and B. Parameter Determination
facilitate the computational complexity analysis as follows. One important characteristic of our method is that parame-
s s
Matching two cracks Cisi and C j j takes in O(|Cisi ||C j j |) time. ters or thresholds have clear statistical meanings which make
s
Generally, the total number of cracks in I is small and can them easy to be identified for different application scenarios.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

LI et al.: AUTOMATIC PAVEMENT CRACK DETECTION BY MULTI-SCALE IMAGE FUSION 7

Algorithm 2 MFCD Algorithm scales exceeds 3, the detected cracks are almost unchanged.
input : I s , s = 0, 1, . . . , n s − 1 However, the cost of computation increases with this number.
output: C ∗ Therefore, we use 3 scale images. Then, by fixing k = 0, 1, 2,
T p2 log T p 2
we experimentally find σ = 0.8 to be the best choice. Thus,
1 foreach I s do O(n s m4
κ ) the 3 scale images are the original image, σs = 0.8, and
2 Extract C s using WMIP algorithm;
T p2 log T p 2
O( m 4 κ ) σs = 1.6.
s
The choice of m determines the total number of grid cells.
3 foreach C si and C s j do O(n s |Cisi ||C j j |) A smaller m leads to more seeds which increase computation
s s s s
4 foreach Ci i ⊆ C si and C j j ⊆ C s j do O(|Ci i ||C j j |) load in the subsequent steps. On the other hand, increasing
si sj si sj the value of m may miss true crack seeds. In practice,
5 Match Ci and C j ; O(|Ci ||C j |)
m significantly impacts the computation time but its effects
6 foreach C s do O(n s |Cis |) on results do not change much after reaching a sufficient
7 foreach Cis ⊆ C s do O(|Cis |) large number. Thus, we set m = 8 based on the existing
8 Compute ρ(Cis ) using (13); O(|Cis |) literature [34].
Te , Tv and Tr are statistical parameters independent of
9 Merge matched cracks using (14); O(n s κ) image size. They are all based on pixel intensity. Te allows
10 foreach Ci do O(nκ) to retain only the darkest crack seeds. Tv retains the paths
11 Compute f (Ci ) using (15); O(n s ) with the lowest mean intensities. Tr controls the merging
12 if f (Ci ) ≥ F −1 (1 − α, n s ) then of dark pixels neighboring to the currently retained paths.
13 Add Ci into C ∗ ; O(|Ci |) Naturally, we would want maintain Te < Tv < Tr due to
14 return C ∗ ; O(1) their meanings. We choose a dark (i.e. small in intensity)
Te to avoid spending much time on generating many false-
positive paths. A small value of Tv can help us to remove
false-positive paths. However, compared with Te , we prefer a
bigger Tv because there are often pixels which are not as dark
as crack seeds on a path. Meanwhile, a verified path constitutes
only the skeleton of a crack, so a bigger Tr can help to absorb
neighboring dark pixels to grow this skeleton. Thus, we choose
their values jointly as Tr = k1 Tv = K 2 Te , k2 > k1 > 1 to
guarantee Te < Tv < Tr . Tr is determined statistically as
follows: we find the first trough of the image histogram and
count the pixels whose intensities are lower than that of the
first trough, denoting this amount as n, then we determine Tr as
Tr = n/κ, where κ is the total amount of pixels in the image.
For the choice of k1 and k2 , bigger values lead to only discard
few crack seeds and minimal intensity paths which causes
Fig. 5. Pavement inspection robot developed by Guimu Robot Co. Ltd.
significant computation demand in the subsequent steps, while
smaller values increase the risk of removing real cracks. Thus,
Parameters σ p in (11) and α in (17) do not have significant we have to find a trade-off between computing time and
influence on the detection performance. σ p is only used in completeness. In practice, we have found that k1 = 2 and
score computation, and different values only lead to different k2 = 10 work well.
absolute scores of cracks, but still keep the relative differences T p , Tg and Td are self-adaptive parameters which are chosen
between cracks. α is the significance level, indicating the according to the image size. T p determines the maximum
probability of rejecting the null hypothesis when it is true. distance between two adjacent crack seeds to be connected.
A significance level of 0.05 indicates a 5% risk of conclud- If we choose a large T p , we have to spend too much time to
ing that a difference exists when in fact there is no actual generate false-positive paths using Dijkstra algorithm. It does
difference. Thus, in practice, our proposed MFCD algorithm not improve the algorithm results much because most of these
requires the tuning of eight parameters: the standard devia- paths are discarded in the following path verification step.
tion of Gaussian kernel σs for multi-scale image generation, However, if we choose a small T p , some true crack paths
the size of grid cell m, the thresholds Te , Tv , Tr , T p , Tg and Td . may be missed. Considering the concerns above, we choose a
It is important to determine the values of σs in multi- relative bigger value for T p as T p = min(n v , n v )/10, where
scale image generation in Section IV-A. We have to find a n u and n v denote the resolution of the image. Tg and Td are
trade-off between efficiency and completeness. Define σs = only used for the crack matching. A very small value may
kσ, k ∈ N. First, we fix σ = 1, and can adjust the number increases false negative rate, while a very big value may lead
of scales from 1 to 5. By setting k = 0, 1, . . . , n, we obtain to a high false positive rate. In practice, we select at least
n + 1 images with different scales. We select 10 representative 10 representative images for each dataset and we find that
images for the experiments. We find that when the number of Tg = min(n v , n v )/6 and Td = min(n v , n v )/50 work well.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 8. F1-measure values of six algorithms on AigleRN dataset.

Fig. 6. Computation speed test results of our MFCD algorithm. Each data
point in this figure is an average of the results from ten pavement images.

Fig. 9. Averaged values of Precision, Recall, and F1-measure for all the
images in AigleRN. The values of GC, FFA, MPS are obtained according
to the results provided by [12]. The values of CrackIT and CrackForest are
obtained by running the source codes provided by their respective authors.
Here, WMIP indicates WMIP based crack detection method on I 0 .

A. Datasets, Counterparts, and Metrics


Three datasets are tested in our experiments.
• AigleRN dataset [12]. This public dataset contains
38 French pavement images with ground truth. The reso-
Fig. 7. Representative sample results of crack detection using our proposed
method. The first row lists the original images, rows 2-4 are detection results lution is 991 × 462 pixels. They have been pre-processed
from scale 1-3, the last row is the final detection results. to mitigate the influence of non-uniform lighting
conditions.
• CFD dataset [20]. This public dataset is composed
VI. E XPERIMENTS of 118 images with a resolution of 480×320 pixels, which
We have implemented our algorithm using MATLAB under can generally represent urban road surface conditions
a PC with an operating system of Windows 8, which has in Beijing, China. Each image has hand labeled ground
an Intel(R) CoreTM 2 i5-4200U CPU and 4 GB memory. truth. The images contain noises such as shadows, oil
We evaluate the performance of our MFCD method on three spots, and water stains.
datasets, including two public datasets and one self-captured • APR dataset. This self-captured dataset contains
dataset, and compare it with six state-of-the-art algorithms. 33 images of an airport runway, which are captured
We set the parameters in our MFCD algorithm according to from the Shuangliu International Airport, Chengdu,
the method presented in the section of V-B. China. APR dataset can be decomposed into categories
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

LI et al.: AUTOMATIC PAVEMENT CRACK DETECTION BY MULTI-SCALE IMAGE FUSION 9

Fig. 10. Example results of different algorithms on AiGleRN (from left to right: original image, ground truth, CrackIT, GC, CrackForest, FFA, MPS, WMIP
based method on I 0 , and MFCD).

according to their resolutions: 19 images with a •CrackForest [20]. CrackForest is a road crack detection
resolution of 1200 × 900 pixels, and 14 images with framework based on random structured forests, by learn-
a resolution of 2048 × 2048 pixels. We capture these ing the inherent structured information of cracks.
images with a Genie Nano M1920 Mono camera fixed For FFA, an important parameter is the distance of oriented
on a pavement inspection robot developed by Guimu segments used to compute features. The authors indicate that
Robot Co. Ltd., as shown in Fig. 5. The acquisition are “The distance must be higher than granulate size to obtain
made perpendicular to the pavement, which means that an efficient filtering. If distance is high enough, there is no
the optical axis of the sensor is perpendicular to the noise detection.” In our experiments, we choose the distance
airport runway. The ground truth of cracks are labeled to be 30 when testing on AigleRN and CFD datasets. For
manually by two different experts. Since we have to other methods mentioned above, we use the parameters as
capture these images during the night when the airport is recommended by their authors.
closed, the artificial lighting is used in our system. Due To evaluate the performance of different crack detection
to the special texture, the thin width of cracks and poor algorithms quantitatively, three metrics, including Precision,
lighting conditions, this dataset is quite challenging. Recall and F1-measure, are employed. These three metrics can
The six existing methods which we compare our algorithm be computed based on true positive (TP), false negative (FN),
to are: and false positive (FP),
• GC [35]. GC is a geodesic contour method with automatic TP
selection of points of interest based on auto-correlation. Precision = (20)
TP+FP
• FFA [13]. Free Form Anisotropy (FFA) is based on the TP
estimation of minimal intensity paths at each pixel in four Recall = (21)
TP+FN
directions, and the pixel is recognized as a crack if the Precision × Recall
path cost greatly varies with the direction. F1-measure = 2 × (22)
Precision + Recall
• MPS [12]. Minimal Path Selection (MPS) is a crack
detection algorithm based on the original minimal inten- Since acquiring a high quality ground truth is difficult
sity path selection. for real images, we allow a tolerance margin in measuring
• CrackIT [15]. CrackIT is a Matlab toolbox for road crack the coincidence between the detected cracks and the ground
detection and characterization. truth. As comparisons, according to the experiment settings
• CrackTree [10]. CrackTree is an automatic pavement in [12] and [20] where AigleRN and CFD datasets are
crack detection method which adopts minimum spanning proposed, we assume that TP pixels are included within a
trees to search cracks from a crack seed graph. 2 to 5-pixel vicinity of the ground truth on AigleRN and
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

CFD, respectively. These settings are the same for all tested
methods. For APR dataset, considering that the generation of
the ground truth remains a difficult task in the high resolution
images, we employ a 5-pixel tolerance distance between the
detection and the labeled ground truth for the calculation of
the TP rate.

B. Computation Speed Test


To validate the computation complexity analysis in theo-
rem 2, we perform a speed test on our MFCD algorithm.
According to theorem 2, the computation complexity of
MFCD algorithm is related to four parameters: n s , κ, m
and T p . We test the speed of the MFCD algorithm with
different settings of the four parameters. Each time, we fix
three parameters of them and change the remaining one to
verify the trend of computation time. The results are shown
in Fig. 6, where for each setting, 10 pavement images from
CFD dataset are carried out for averaged performance. Fig. 6
shows that computation time results are consistent with the
theoretical analysis.

C. Results on AigleRN Dataset


Fig. 7 illustrates two typical sample detections on AigleRN.
The final detection results (row 5) are obtained by fusing the
results from different scales (row 2-4). It is clear that multi-
scale fusion improves the overall performance.
The F1-measure values of the six algorithms on AigleRN
dataset are shown in Fig. 8. The summary statistics for
comparisons are presented in Fig. 9, and representative sample
results are shown in Fig. 10. It is clear that our MFCD method
outperforms the counterparts. CrackIT produces a higher FN
rate that leads to a lower recall value. GC can only find a
small part of cracks. CrackForest is sensitive to the background
texture. Note that we did not re-train the model in CrackForest
using AigleRN dataset. FFA outputs the results as a thicker
line, implying high FP rate. As for MPS, for the image with
serious noise, the values of FP and FN become an issue.
Furthermore, to validate whether the multi-scale fusion can
improve crack detection, we compare the WMIP based crack
detection result on I 0 with MFCD method. As shown in Fig. 9
and Fig. 10, it is clear that the multi-scale fusion indeed
improves the performance of single-scale detections.

D. Results on CFD Dataset Fig. 11. Some experimental results using our MFCD algorithm on CFD
dataset (from left to right: original image, ground truth, and results from
Our algorithm also achieves superior performance on CFD MFCD).
dataset. Fig. 11 presents representative images and their detec-
tion results using our MFCD algorithm, where we can see TABLE I
that the images in CFD are quite noisy. The performance of C RACK D ETECTION R ESULTS E VALUATION ON CFD D ATASET
our MFCD algorithm degrades when the contrast between
cracks and backgrounds is low (as shown in the 2nd row
in Fig. 11), or there are dark regions, e.g. oil spots, in the
pavement images (as shown in the 5th and 9th rows in Fig. 11).
The crack detection results on the CFD dataset are summarized
in Tab. I. Again, our algorithm outperforms all counterparts.

E. Results on APR Dataset Fig. 12 presents representative images and their detection
We have tested our algorithm on APR dataset, and com- results using CrackForest, CrackIT and our MFCD algorithms,
pared the results with CrackForest and CrackIT methods. where we can see that the images in APR are very challenging.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

LI et al.: AUTOMATIC PAVEMENT CRACK DETECTION BY MULTI-SCALE IMAGE FUSION 11

ACKNOWLEDGMENT
The authors would like to thank Guimu Robot Co. Ltd. for
their feedback. They are also grateful to C. Chou, H. Cheng,
S. Yeh, A. Kingery, A. Angert, T. Sun, D. Wang, Y. Sun, Y. Yu,
J. Gong, and M. Momin for their input and contributions to
the Networked Robots Laboratory at Texas A&M University.

R EFERENCES
[1] H. M. La et al., “Mechatronic systems design for an autonomous
robotic system for high-efficiency bridge deck inspection and evalua-
tion,” IEEE/ASME Trans. Mechatronics, vol. 18, no. 6, pp. 1655–1664,
Dec. 2013.
[2] H. M. La et al., “Autonomous robotic system for high-efficiency non-
destructive bridge deck inspection and evaluation,” in Proc. IEEE Int.
Fig. 12. Some experimental results on APR dataset (from left to right: Conf. Automat. Sci. Eng. (CASE), Aug. 2013, pp. 1053–1058.
original image, ground truth, results from CrackForest, CrackIT and MFCD). [3] H. M. La, N. Gucunski, S.-H. Kee, and L. Van Nguyen, “Data analysis
and visualization for the bridge deck inspection and evaluation robotic
system,” Vis. Eng., vol. 3, Dec. 2015, Art. no. 6.
TABLE II [4] Y. Hu, C.-X. Zhao, and H.-N. Wang, “Automatic pavement crack
C RACK D ETECTION R ESULTS E VALUATION ON APR D ATASET detection using texture and shape descriptors,” IETE Tech. Rev., vol. 27,
no. 5, pp. 398–405, 2010.
[5] J. Tang and Y. Gu, “Automatic crack detection and segmentation using
a hybrid algorithm for road distress analysis,” in Proc. IEEE Int. Conf.
Syst., Man, Cybern. (SMC), Oct. 2013, pp. 3026–3030.
[6] Y. Hu and C.-X. Zhao, “A local binary pattern based methods for
pavement crack detection,” J. Pattern Recognit. Res., vol. 5, no. 1,
pp. 140–147, 2010.
[7] R. S. Lim, H. M. La, Z. Shan, and W. Sheng, “Developing a crack
inspection robot for bridge maintenance,” in Proc. IEEE Int. Conf. Robot.
Automat. (ICRA), May 2011, pp. 6288–6293.
The crack detection results on the APR dataset are summarized [8] R. S. Lim, H. M. La, and W. Sheng, “A robotic crack inspection and
in Tab. II where the results of crackIT and CrackForest are mapping system for bridge deck maintenance,” IEEE Trans. Autom. Sci.
Eng., vol. 11, no. 2, pp. 367–378, Feb. 2014.
obtained using their respective open source codes. Note that [9] M. Gavilán et al., “Adaptive road crack detection system by pavement
we did not re-train the model in CrackForest with the images classification,” Sensors, vol. 11, no. 10, pp. 9628–9657, 2011.
of APR dataset. It can be seen that CrackForest produces a [10] Q. Zou, Y. Cao, Q. Li, Q. Mao, and S. Wang, “CrackTree: Automatic
crack detection from pavement images,” Pattern Recognit. Lett., vol. 33,
much higher FP rate. The value of FN rate of CrackIT is a little no. 3, pp. 227–238, 2012.
high that leads to lower recall value. Our MFCD algorithm [11] R. Amhaz, S. Chambon, J. Idier, and V. Baltazart, “A new minimal path
succeeds to detect the cracks in APR dataset and obtain a selection algorithm for automatic crack detection on pavement images,”
in Proc. IEEE Int. Conf. Image Process. (ICIP), Oct. 2014, pp. 788–792.
satisfied overall performance. [12] R. Amhaz, S. Chambon, J. Idier, and V. Baltazart, “Automatic crack
The original images and detection results of the three detection on two-dimensional pavement images: An algorithm based
datasets mentioned above are available on the following web on minimal path selection,” IEEE Trans. Intell. Transp. Syst., vol. 17,
no. 10, pp. 2718–2729, Oct. 2016.
page: http://telerobot.cs.tamu.edu/bridge/Datasets.html. [13] T. S. Nguyen, S. Begot, F. Duculty, and M. Avila, “Free-form anisotropy:
A new method for crack detection on pavement surface images,” in Proc.
VII. C ONCLUSION AND F UTURE W ORK 18th IEEE Int. Conf. Image Process. (ICIP), Sep. 2011, pp. 1069–1072.
[14] H. Oliveira and P. L. Correia, “Automatic road crack detection and
We developed a new MFCD method for detecting the cracks characterization,” IEEE Trans. Intell. Transp. Syst., vol. 14, no. 1,
from pavement images. First, we proposed a WMIP based pp. 155–168, Mar. 2013.
[15] H. Oliveira and P. L. Correia, “CrackIT—An image processing toolbox
method to detect the cracks at individual scales. Second, for crack detection and characterization,” in Proc. IEEE Int. Conf. Image
we matched the corresponding cracks across different scales. Process. (ICIP), Oct. 2014, pp. 798–802.
Finally, a crack evaluation model was built, and the crack was [16] R. Medina, J. Llamas, E. Zalama, and J. Gómez-García-Bermejo,
“Enhanced automatic detection of road surface cracks by combining
selected as the detected crack if it passed statistical hypothesis 2D/3D image processing techniques,” in Proc. IEEE Int. Conf. Image
T 2 log T p
test. Our MFCD algorithm takes in O(n s p m 4 κ 2 ) time. Process. (ICIP), Oct. 2014, pp. 778–782.
[17] M. R. Jahanshahi, S. F. Masri, C. W. Padgett, and G. S. Sukhatme,
We have implemented the proposed algorithm and experi- “An innovative methodology for detection and quantification of cracks
mental results are consistent with our complexity analysis. through incorporation of depth perception,” Mach. Vis. Appl., vol. 24,
We tested MFCD algorithm on three representative datasets. no. 2, pp. 227–241, 2013.
[18] K. Fernandes and L. Ciobanu, “Pavement pathologies classification using
Experimental results showed that 1) The proposed MFCD graph-based features,” in Proc. IEEE Int. Conf. Image Process. (ICIP),
algorithm improves the crack detection performance compared Oct. 2014, pp. 793–797.
with the single scale WMIP based method. 2) Our MFCD [19] P. Prasanna et al., “Automated crack detection on concrete bridges,”
IEEE Trans. Autom. Sci. Eng., vol. 13, no. 2, pp. 591–599, Apr. 2016.
algorithm outperforms six existing methods. [20] Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, “Automatic road crack
In the future, we will consider fusing the camera inputs detection using random structured forests,” IEEE Trans. Intell. Transp.
with ground penetration radar and laser ranger finder inputs. Syst., vol. 17, no. 12, pp. 3434–3445, Dec. 2016.
[21] A. Cord and S. Chambon, “Automatic road defect detection by tex-
We will study how to perform crack detection using multi- tural pattern recognition based on AdaBoost,” Comput.-Aided Civil
modal inputs. Infrastruct. Eng., vol. 27, no. 4, pp. 244–259, 2012.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

12 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

[22] B. Y. Lee, Y. Y. Kim, S.-T. Yi, and J.-K. Kim, “Automated image Dezhen Song (S’02–M’04–SM’09) received the
processing technique for detecting and analysing concrete surface Ph.D. degree in industrial engineering from Univer-
cracks,” Struct. Infrastruct. Eng., vol. 9, no. 6, pp. 567–577, 2013. sity of California at Berkeley, Berkeley, CA, US,
[23] H. M. La, N. Gucunski, K. Dana, and S.-H. Kee, “Development of an in 2004.
autonomous bridge deck inspection robotic system,” J. Field Robot., He is currently a Professor with the Depart-
vol. 34, no. 8, pp. 1489–1504, 2017. ment of Computer Science and Engineering, Texas
[24] D. Zhang, Q. Li, Y. Chen, M. Cao, L. He, and B. Zhang, “An A&M University, College Station, TX, USA. His
efficient and reliable coarse-to-fine approach for asphalt pavement crack research interests include networked robotics, dis-
detection,” Image Vis. Comput., vol. 57, pp. 130–146, Jun. 2017. tributed sensing, computer vision, surveillance, and
[25] S. Chambon and J.-M. Moliard, “Automatic road pavement assessment stochastic modeling.
with image processing: Review and comparison,” Int. J. Geophys., Dr. Song received the Kayamori Best Paper Award
vol. 2011, Jun. 2011, Art. no. 989354. of the 2005 IEEE International Conference on Robotics and Automation (with
[26] V. Kaul, A. Yezzi, and Y. C. Tsai, “Detecting curves with unknown J. Yi and S. Ding). He received the NSF Faculty Early Career Develop-
endpoints and arbitrary topology using minimal paths,” IEEE Trans. ment (CAREER) Award in 2007. From 2008 to 2012, he was an Associate
Pattern Anal. Mach. Intell., vol. 34, no. 10, pp. 1952–1965, Oct. 2012. Editor of the IEEE T RANSACTIONS ON ROBOTICS . From 2010 to 2014,
[27] M. Avila, S. Begot, F. Duculty, and T. S. Nguyen, “2D image based road he was an Associate Editor of IEEE T RANSACTIONS ON AUTOMATION
pavement crack detection by calculating minimal paths and dynamic pro- S CIENCE AND E NGINEERING. He is currently a Senior Editor for IEEE
gramming,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Oct. 2014, ROBOTICS AND AUTOMATION L ETTERS .
pp. 783–787.
[28] Y.-J. Jeon, J. P. Yun, D.-C. Choi, and S. W. Kim, “Defect detection algo-
rithm for corner cracks in steel billet using discrete wavelet transform,”
in Proc. ICCAS-SICE, Aug. 2009, pp. 2769–2773.
[29] R. G. Lins and S. N. Givigi, “Automatic crack detection and measure-
ment based on image analysis,” IEEE Trans. Instrum. Meas., vol. 65,
no. 3, pp. 583–590, Mar. 2016.
[30] L. Ying and E. Salari, “Beamlet transform-based technique for pavement Yu Liu received the B.S. degree from the
crack detection and classification,” Comput.-Aided Civil Infrastruct. Department of Computer Science and Technology,
Eng., vol. 25, no. 8, pp. 572–580, 2010. Taiyuan University of Technology, Taiyuan, China,
[31] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed. in 2015. He is currently pursuing the master’s
London, U.K.: Pearson, 2017. degree in computer science and technology with the
[32] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm Civil Aviation University of China, Tianjin, China.
for discovering clusters in large spatial databases with noise,” in Proc. His current research interests include the
KDD, vol. 96. 1996, pp. 226–231. areas of computer vision, image processing, and
[33] E. W. Dijkstra, “A note on two problems in connexion with graphs,” understanding.
Numer. Math., vol. 1, no. 1, pp. 269–271, Dec. 1959.
[34] Y. Huang and B. Xu, “Automatic inspection of pavement cracking
distress,” J. Electron. Imag., vol. 15, no. 1, p. 013017, 2006.
[35] S. Chambon, “Detection of points of interest for geodesic contours:
Application on road images for crack detection,” in Proc. Int. Joint
Conf. Comput. Vis. Theory Appl. (VISAPP), 2011, pp. 1–4.

Haifeng Li received the Ph.D. degree in control the- Binbin Li received the B.S. degree from the
ory and control engineering from Nankai University, Department of Electrical Engineering and Automa-
Tianjin, China, in 2012. tion, Harbin Institute of Technology, Harbin, China,
He is currently an Associate Professor with the in 2012. He is currently pursuing the Ph.D. degree in
Department of Computer Science and Technology, computer engineering with Texas A&M University,
Civil Aviation University of China, Tianjin, China. College Station, TX, USA.
He has authored or co-authored over 30 techni- His current research interests include the areas of
cal articles. His research interests include computer robot vision, visual tracking, and recognition.
vision, image processing, robotic sensing, multisen-
sor fusion, robot localization, and navigation.