Available online at www.sciencedirect.

com

ScienceDirect
Cognitive Systems Research 45 (2017) 109–123
www.elsevier.com/locate/cogsys

Active and semi-supervised learning for object detection with

imperfect data
Action editor: Simona Doboli
Phill Kyu Rhee ⇑, Enkhbayar Erdenee, Shin Dong Kyun, Minhaz Uddin Ahmed,
Songguo Jin
Computer and Information Engineering Department, Inha University, 100 Inha-ro, Nam-gu 22212, Incheon, Republic of Korea

Received 24 June 2016; received in revised form 21 March 2017; accepted 18 May 2017
Available online 26 May 2017

Abstract

In this paper, we address the combination of the active learning (AL) and semi-supervised (SSL) learnings, called ASSL, to leverage
the strong points of the both learning paradigms for improving the performance of object detection. Considering the pros and cons of the
AL and SSL learning methods, ASSL where SSL method provides the incremental improvement of semi-supervised detection perfor-
mance by combining the concept of diversity imported from AL methods. The proposed method demonstrates outstanding performance
compared with state-of-art methods on the challenging Caltech pedestrian detection dataset, reducing the miss rate to 12.2%, which is
significantly smaller than current state-of-art. In addition, extensive experiments have been carried out using ILSVRC detection dataset
and online evaluation for activity recognition.
Ó 2017 Elsevier B.V. All rights reserved.

Keywords: Active Learning (AL); Semi-Supervised Learning (SSL); Convolutional Neural Network (CNN); Deep learning; Object detection

1. Introduction data distribution. However, the assumption may not be

Object detection is a key problem for robotics, automo-
tive applications, and surveillance systems. Despite the
great advances in object detection technology, which cate-
gorizes and localizes visual objects in a scene, the imbal-
anced training data poses a very challenging problem
since it very often causes biased results and performance
degradations. Supervised detection methods (Girshick,
2015; Ren, He, Girshick, & Sun, 2015) have shown great
promise for the automatic object detection with the
assumption that the training and testing sets have the same
⇑ Corresponding author.
E-mail addresses: pkrhee@inha.ac.kr (P.K. Rhee), enkhbayar@inha.
edu (E. Erdenee), tisgb_1@naver.com (S.D. Kyun), minhaz@inha.edu
(M.U. Ahmed), sgkim735@inha.edu (S. Jin).
valid in real-world, and the underlying distributions
between training and testing data are very often substan-
tially imbalanced. Sample selection bias can cause a signif-
icant performance degradation the object detection based
on a supervised learning method.
Recently, convolutional neural network (CNN) (Bengio,
Courville, & Vincent, 2013; Hinton, 2006; LeCun et al.,
1989) becomes dominant in object detection area since
Krizhevsky, Sutskever, and Hinton (2012) broke through
the performance barrier of large-scale object detection in
ILSVRC 2012 (Russakovsky et al., 2015). High dimen-
sional deep feature space such as fast RCNN (Girshick,
2015), SPP (He, Zhang, Ren, & Sun, 2015) causes perfor-
mance degradations due to not only its relatively small
quantity but also insufficient quality of training samples
since most previous object detection approaches rely on

http://dx.doi.org/10.1016/j.cogsys.2017.05.006
1389-0417/Ó 2017 Elsevier B.V. All rights reserved.
110 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
that training data samples being selected randomly from
distributions. However,
imbalanced sample selection is very likely to occur in
object detector learning, where the available training sam-
ples are relatively small comparing to the feature space size
and huge computation powers are necessary. In real world
applications, many researchers and developers also very
often encounter the problem of sample selection bias
(Heckman, 1979; Zadrozny, 2004), when training samples
are drawn manually, might not follow random sampling
assumptions. The imbalance problem in training sample
selection for an object detector is due to not only biased
training samples (sample quality problem) but also a short-
age of samples (sample quantity problem). Such an imbal-
ance in training samples causes not only the incorrect
measure of the object class distribution but also the degra-
dation of the detection accuracies.
Active learning (AL) and Semi-supervised learning
(SSL) methods, which are originally invented for the clas-
sification accuracy improvement using both labeled and
unlabeled data, can be adopted to overcome the imbal-
ances of sample distribution, imperfect labeling, and selec-
tion biases in training an object detector. AL method adds
labeled data to the training set at each iteration from an
unlabeled or an imperfect dataset, by a selective sampling
strategy followed by queries, and is expected to improve
detection performance. While AL methods select the most
uncertain samples (the highest disagreement) based on the
assumption of a perfect expert, SSL methods samples with
the most confident (highest agreement among classifiers)
based on a cluster assumption that if samples are in the
same cluster, the probability that they might be in the same
class is high (Chapelle., 2006). In AL, the labeling is per-
formed by human experts. In SSL, the labeling is per-
formed on the basis of the current classification rules
based on cluster assumption. The assumption of infallible
human experts is not valid in object detection since if the
strict labeling rules are applied, the cost becomes too high
to be used in a real world application. If the assumption is
not satisfied, SSL will not produce a better performance,
even degrading the classification accuracy.
In this paper, the labeled samples are divided into per-
fect and imperfect samples. A perfect sample has almost
correct region of interest (RoI) information such as object
category, attribute, position, and size, but some informa-
tion of an imperfect sample may not be sufficiently correct
to be used in the training. For example, the attribute of an
imperfect ROI represented by a CNN deep feature may be
ill-posed, or noisy to be modeled in the training time. The
dataset of imperfect samples tends to be imbalanced in dis-
tribution or biased. Note that an imperfect RoI is different
from an unlabeled sample where the only given informa-
tion is the position and bounding box of an object in a
scene, i.e., region proposal, but no additional information
is provided such as the category or existence of an object.
The imperfect samples should be handled differently from
the perfect samples since some imperfect samples do not
contribute or even harmful effect in building a high-
performing object detector. The proposed method takes
the advantage of the concept of AL and SSL in terms of
exploration and exploitation in the training time. Assum-
ing that an imperfect training dataset is given (e.g. ILSVRC
Russakovsky et al., 2015), we first construct a simple object
detection model using a small number of perfect samples
instead of training using entire training samples. We parti-
tion the imperfect samples into several batches. Then, we
employ ASSL, which is a batch-mode learning of AL and
SSL combination for the effective incremental training
framework, which exploits a confident and reliable sam-
pling and explores a diversity and informative sampling.
The goal is to construct an object detector based on the
batch-mode active semi-supervised learning starting from
a small number of approximately perfect labeled samples
in conjunction by adding the set of imperfect or unlabeled
samples.
The novelty and contribution of ASSL are summarized
in the following:

The combined framework for AL and SSL for efficient
object detection for the quality assurance to solve data
imbalance problem.

We combine AL and SSL to minimize the effect of
imbalanced classifications due to the bias of training
samples and to model capability of the deep feature dis-
tribution for efficient object detection.

We extensively evaluate our method on challenging Cal-
tech pedestrian detection benchmark (Dollar, Wojek,
Schiele, & Perona, 2011), ILVRC object detection data-
set (Russakovsky et al., 2015) and online evaluation for
action detection. The proposed method demonstrates
outstanding performance compared with state-of-art
methods on the Caltech pedestrian detection dataset,
reducing the miss rate to 12.23%, which is significantly
smaller than current state-of-art 17.1% Checkerboard
(Zhang, Benenson, & Schiele, 2015). Experiment results
confirm that the proposed ASSL method can be applied
a wide variety of application domains including cogni-
tive systems, automotive applications, and surveillance
systems.
2. Related works
2.1. Active learning
Active learning is a one kind of semi-supervised learning
method that iteratively selects the most informative sam-
ples to be labeled by manually (Settles, 2010). In active
learning, uncertain and erroneous portion of the training
data are required to be queried, annotated manually with
minimum human cost, and added to the training dataset
to get the highest gain in classification accuracy (Roy &
McCallum, 2001). Thus, the query strategy is the main con-
sideration in an active learning technique. There are several
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
query functions, focus on the sample selection has been
proposed.
Uncertainty sampling strategy (Lewis & Gale, 1994),
tries to select the closest samples to the class decision
boundary. In uncertainty sampling, the samples that are
least certain by the current classifier is considered as infor-
mative sample. Li and Sethi (2006) proposed a confidence-
based active learning, which identifies and labels only
uncertain samples by computing the uncertainty level of
all samples according to the output value of current classi-
fier and selects samples whose outputs are within the uncer-
tainty range. There are several successful works which
adopt uncertainty sampling such as non-probabilistic
approaches; nearest-neighbor classifier (Lindenbaum,
Markovitch, & Rusakov, 2004), decision tree (Lewis &
Catlett, 1994) and support vector machines (Tong &
Koller, 2001) and probabilistic approaches (Culotta &
Mccallum, 2005; Lewis & Gale, 1994). As popular as
uncertainty sampling is, it has several problems; first, the
most uncertain samples do not represent the whole data
distribution well and might make classifier more focuses
on noise or outliers. Second, since it employee single classi-
fier, the performance of uncertainty sampling approaches is
very limited by the performance of the single classifier.
Unlike to uncertainty sampling, instead of relying on the
single classifier, Query by committee (QBC) strategy
(Dagan & Engelson, 1995; Seung, Opper, &
Sompolinsky, 1992) using a different hypothesis of a com-
mittee of classifiers which selects the samples which have
the highest disagreement between the classifiers. Generally,
Query by Committee approach generates accurate yet
diverse classifiers and makes better generalization than sin-
gle classifier by combining ensemble methods (Breiman,
1996; Lu, Wu, & Bongard, 2015; Xu, Li, & Chen, 2012).
While most active learning methods select only a single
sample at each iteration, in batch mode active learning
(Demir, Persello, & Bruzzone, 2011), a batch of samples
is selected at each step and instead of update model at each
iteration, updates the model once at each batch iteration.
We adopt batch mode active learning since batch querying
is more efficient and less expensive (M. Li & Sethi, 2006).
Considering the diversity, the selection strategy of batches
of samples at each iteration allows to speed up the learning
process with different consideration such as minimizing the
margin and maximizing the diversity (Brinker, 2003). The
diversity in the query function by clustering (Xu, Yu,
Tresp, Xu, & Wang, 2003), identifies uncertain samples
avoiding redundancy by clustering the samples.
Many researchers have investigated active learning
method for large-scale image classification. Active learning
has been employed for object region labeling (Siddiquie &
Gupta, 2010; Vijayanarasimhan & Grauman, 2009) and
image recognition tasks (Kapoor, Grauman, Urtasun, &
Darrell, 2007; Qi, Hua, Rui, Tang, & Zhang, 2008) with
less annotation effort and better annotation quality. Since
annotation quality varies from people to people, some
researchers have investigated automatic annotation preci-
111
sion assurance (Sorokin & Forsyth, 2008; Welinder &
Perona, 2010). Assuming that a single prominent object
of interest exists in an image, some approaches have tried
to learn object models directly from noisy keyword search
(Fergus, Fei-Fei, Perona, & Zisserman, 2005; Li & Fei-Fei,
2010; Vijayanarasimhan & Grauman, 2009).
2.2. Semi-supervised learning
SSL approaches, which leverage a classifier performance
using both labeled and unlabeled samples, are divided into
self-training, co-training, generative probabilistic models,
and graph-based SSL (Zhu, 2008). In the self-training
SSL, the classifier is constructed beginning with a few
labeled training samples, a portion of the unlabeled train-
ing dataset is labeled using the current classifier, the most
confident samples among the predicted labeled samples
are added to the training dataset, repeatedly, until conver-
gence. Uncertainty sampling method in AL is a comple-
mentary approach, where least confident samples are
selected for querying. In co-training (Mitchell & Blum,
1998), an ensemble method is employed. Firstly, separate
models are learned using independently labeled datasets.
The current models classify the unlabeled data and learn
the next models using a few selected samples with most
confident predicted labels. QBE (Query-by-committee) is
a complimentary active learning version, where the com-
mittee queries the unlabeled instances with the most agree-
ment. Active and semi-supervised learning try to solve the
same problem from different directions (Settles, 2010).
2.3. Deep feature for object detection
Motivated by recent advances of the deep convolutional
neural network (CNN) (Bengio et al., 2013; Hinton, 2006;
LeCun et al., 1989) on visual object recognition tasks
(Krizhevsky et al., 2012; Simonyan & Zisserman, 2015;
Szegedy et al., 2015), significant improvements have been
made in the object detection task (Girshick, 2015;
Girshick, Donahue, Darrell, & Malik, 2014; He et al.,
2015; Makantasis, Doulamis, Doulamis, & Psychas,
2016). Among them, most notably Girshick et al. (2014)
proposed the region based convolutional network object
detector, called R-CNN and achieved state-of-art perfor-
mance on object detection benchmarks with a large margin
over the previous best results, such as Overfeat (Sermanet
et al., 2013) and deformable part model (DPM)
(Felzenszwalb, Girshick, Mcallester, & Ramanan, 2009).
The RCNN framework that employees AlexNet
(Krizhevsky et al., 2012) and VGGNet (Simonyan &
Zisserman, 2014) to extract a fixed-length feature vector
from object proposals generated by selective search
(Uijlings, Van De Sande, Gevers, & Smeulders, 2013) and
then use linear SVM to classify each region. In order to
reduce the computation of forward pass for each region
proposal in RCNN (Girshick et al., 2014), Fast RCNN
(Girshick, 2015) and SPP (He et al., 2015) compute feature
112 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
maps once and then pool the features for object region only
from the last convolutional layer. Most recently, Faster
RCNN (Ren et al., 2015) introduced a region proposal net-
work that replaces region proposals provided by selective
search and achieved better performance and speed.
3. System overview
We very often encounter the problem of sample selec-
tion bias (Heckman, 1979; Zadrozny, 2004), when training
samples are drawn with manual classification rules,
whereby samples are error-prone and may not follow a ran-
dom sampling assumption. Furthermore, the collected
dataset tends to be imbalance in class distribution. Such
imperfect training samples result in incorrect modeling of
object classes and lead to the degradation of system perfor-
mance. Owing to the intrinsic complexity of fallible label-
ing and imbalanced data samples in object detection, we
need a novel approach to sample selection and learning
method. If imperfect or imbalanced samples exist in a
training dataset, heavy computation due to a huge number
of iterations is required, and results in a slow or inefficient
learning in a supervised learning framework. We address
how to combine and SSL to solve the imbalanced object
detector using biased or imperfect training samples. AL
and SSL methods can be combined to define an efficient
learning framework that exploits both labeled and semi-
labeled samples to ensure modeling capability of the deep
feature distribution for efficient object detection by select-
ing well-balanced samples being labeled or relabeled by
the human expert.
In this section, we discuss the overview of an efficient
batch-mode learning method for object detection in the
presence of imperfect training samples by combining AL
and SSL based on the collaborative sample selection, called
ASSL. In general, AL tends to explore the unknown data
distribution, and on the other hand, SSL tries to exploit
the unknown aspect. While AL can start with few labeled
samples without affecting much the generalization capabil-
ity, SSL requires a more confident labeled samples, which
affect much to the convergence of the classifier perfor-
mance. The combination of AL and SSL is expected to
produce better discrimination power. ASSL can improve
the accuracy of object detection based on a flexible sam-
pling strategy which alleviates selection bias and class dis-
tribution imbalance that might occur frequently in object
detection.
We formulate the object detector learning on imperfect
training data using a batch-mode active semi-supervised
learning. Batch-mode learning, whereby a batch of samples
is selected and learned iteratively, is a more practical
approach for object detection since it will not be realistic
to deal with one data sample at a time.
In this paper, the labeled samples are divided into per-
fect and imperfect samples. The confident dataset consists
of the samples labeled with correct object locations, poses,
scales and object classes. The imperfect dataset is allowed
to include samples which are not confidently labeled, i.e.,
(see Fig. 2) not only correctly labeled samples but also
includes samples some information of which is biased
labels under imbalanced underlying distribution. For
example, the information of object location is incorrect,
while that of the scale and pose is correct or the object cor-
rect located, but ill-posed or noisy. The imperfect samples
should be handled differently from the confident samples
since some sample do not contribute or even harmful effect
in learning a high-performing object detector. One can
notice that an unlabeled sample can be treated as the mem-
ber of an imperfect dataset with a missing label. In this con-
text, the concept of imperfect dataset can be thought as a
generalization of labeled and unlabeled dataset the samples
of which are described by confidence scores.
4. Proposed method
We present ASSL framework which combines the deep
feature and batch-mode AL algorithm in a similar manner
given in Persello and Bruzzone (2011), whereby the confi-
dence score function is used in the semi-labeled sample
selection to satisfy diversity criterion. A semi-label is
defined as a label assigned by a classification model, and
thus the label may be incorrect, i.e., either true positive
or false positive. We employ a batch mode active learning,
where the batches of samples are selected at each iteration,
maximizes the diversity to minimize the convergence time
(Brinker, 2003). Instead of random selection, we adopt
an efficient and flexible collaborative sampling strategy by
integrating the uncertainty and diversity criteria from the
concept of AL and the confidence criterion from that of
SSL. ASSL initially trains deep convolutional neural net-
work (CNN) using a small number of confidently labeled
samples (confident dataset), and it repeatedly retrains the
next deep feature model (CNN) by adding the batch of
samples selected using the current classifier (stored in the
semi-confident dataset) until convergence. Overview archi-
tecture of proposed ASSL framework for object detection
is illustrated in Fig. 1. We assume that human classification
(labeling) rules are based on the property of the validation
dataset, the distribution of which is similar to that of test
dataset. We select the set of semi-labeled samples based
on the confidence score function, whereby the semi-
labeled samples are fit with either a diversity criterion or
confidence criterion or both. This imperfect dataset
assumption is reasonable in a sense the diversity of the
training dataset comparing to the validation dataset causes
to inconsistency and imperfect of manual labeling process.
The network architectures (Simonyan & Zisserman,
2014; Zeiler & Fergus, 2014) used in this work have several
convolutional layers, rectified linear units (ReLU), max-
pooling layers, followed by a spatial pooling layer and sev-
eral fully connected layers. The final layer of network has
two sibling layers, one is softmax classification layer which
outputs a score between 0 to 1, indicating the class proba-
bility over the K object classes plus ‘‘background” and sec-
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
Fig. 1. Overview of ASSL framework. First, a simple object detection model is constructed using a small number of perfect samples. Then, we employ
ASSL, which is a batch-mode learning of AL and SSL combination for the effective incremental training framework, which exploits a confident and
reliable sampling and explores a diversity and informative sampling.
ond is a bounding box regression (Felzenszwalb et al.,
2009) layer that outputs four coordinates, indicating box
location for each of the K object classes.
4.1. ASSL collaborative sampling
Considering a batch mode AL framework combined
with SSL approach, the training phase of ASSL is as fol-
lows. First, the classifier (softmax classifier) F is learned
for multiple object classes by using the confident training
samples Dconf . We first select randomly pool candidate
samples Dbatch from the imperfect training dataset Dimp
(Dconf  Dimp ). Then, pseudo labels are (re)assigned to
the selected imperfect samples in terms of uncertainty crite-
rion of AL. We use a simple but effective sampling criterion
for the uncertainty measure of the classifier learning to fil-
ter samples with sufficient uncertainty. The uncertain-
labeled samples are filtered by a diversity criterion to pro-
duce active-labeled samples with minimum redundancy.
Then the confidence criterion of SSL is applied to produce
semi-labeled data. Finally, we train the deep feature model
on the semi-labeled data and repeat these steps until con-
verging or the imperfect dataset is exhausted.
Given an imperfect training set Dimp ¼ fxi gNi¼1 , a detec-
tion model (softmax classifier) F , we compute confidence
scores over K classes for each bounding box as follows.
Current classifier is applied to the each sample in the Dimp
and the probability (Hasan & Roy-Chowdhury, 2015) that
a sample xi belongs to class c is defined as:
expðW Tc xi Þ
CSðxi Þ ¼ pðy i ¼ cjxi ; W Þ ¼ PK
j¼1 expðW j xi Þ
T
where c 2 f1; . . . ; Kg is the set of class labels, W Tc is the cor-
responding weight vector of class c and superscript T
denotes transposition.
In uncertainty sampling stage, pseudo labels are (re)as-
signed to the set of unlabeled or imperfect samples in
Dimp by current classifier. For each object class, a set of
samples is randomly selected and scored (CSðxi Þ) using
113
the current classifier and added to the candidate set
Dbatch . The selected region of interests (RoIs) are eliminated
from the whole training dataset Dimp . Instead of only the
selection of one sample at an iteration, ASSL minimizes
the learning time by building a pool of 2g samples based
on the uncertainty criterion. Formally, considering training
dataset, the imperfect/unlabeled sample with the lowest
confidence can be selected using the uncertainty measure
xbi ¼ argminxi 2Dimp jCSðxi Þj, i.e., a sample that is closest to
the decision boundary, since the most uncertain samples
have the lowest probability to be correctly classified by
the current classification model (Demir et al., 2011). When
using a probabilistic model, uncertainty sampling simply
selects the samples whose the probability of being positive
is around 0.5 (Settles, 2010). We define a margin which is
upper and lower confidence score, similar manner
described in Li and Sethi (2006) and construct the set of
all the RoI samples Duncertain by selecting 2g the samples
inside the margin. That is, g samples with CSðxi Þ the score
closer to the upper boundary are selected from the upper
half of the margin, and the remaining g samples are
selected from the lower half of the margin with CSðxi Þ
the score closer to the lower boundary. We have total 2g
samples in uncertain-labeled sample set Duncertain . However,
the uncertainty criterion may result in the selection of
redundant noisy or samples. We investigate the
uncertain-labeled samples to select the representative and
non-redundant uncertain samples as follows.
The diversity step employs the k-means clustering to
analyze the distribution of uncertain-labeled samples to
ð1Þ apply the diversity measure. Since the samples in the same
cluster are similar, a representative sample should be
selected from each cluster in order to reduce redundant
samples. Our batch mode ASSL also provides the advan-
tage of being incorporated with a diversity measure (Xu
et al., 2003). A batch of samples with diversity criterion
Ddivers is determined by selecting # samples among the 2g
candidates ðg < #Þ with more diverse property, where more
informative samples are expected to contained in Duncertain .
The k-means clustering is applied in the deep feature space
114 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
in the uncertain-labeled samples Duncertain for defining h
clusters and finally taking #=h samples from each cluster.
Finally, we construct Dtr to include SSL philosophy by
initializing DD with a sample xtop ¼ argmaxxi 2Ddivers CSðxi Þ.
At each step, our sampling strategy chooses a sample from
Ddivers add to DD , which is the most confidence/similar sam-
ple in DD in terms of a similarity measure, i.e.,
xtop ¼ argminxi 2DdiversnD fmaxxj 2DD dðxi ; xj Þg. (Here we use
D
Euclidian distance between two deep features in calculating
dðxi ; xj Þ). When the cardinality of DD is c, the sample selec-
tion process is stopped, and the final sample set is DD . We
retrain the deep convolutional network (with softmax clas-
sifier) after the selection of the pool of samples, and the
process is repeated until there is no more data left or a con-
vergence criterion is satisfied. The overall algorithm for
ASSL is presented in Algorithm 1.
Algorithm 1. ASSL Collaborative Sample Selection.
Input: g, #, and c ð2g > # > cÞ; Confidently labeled
dataset Dconf and imperfectly labeled dataset
Dimp ¼ fxi gNi¼1 with Dconf  Dimp
Notations: Progressive batch dataset DD , and batch
candidate dataset Dbatch with DD  Dbatch .
Output: Optimally labeled dataset DD , target L
detection models F ¼ ff ð1Þ ; f ð2Þ ; . . . ; f ðLÞ g
Method:
Step 1. Let Dtr ¼ Dconf ; Train an initial softmax
classifier F 0 using Dtr .
Repeat
Step 2. Select a batch of candidate samples Dbatch from
Dimp ;
Compute CSðxi Þ; xi 2 Dbatch using current
classifier F t (Eq. (1))
Step 3. Determine a batch dataset.
1. Select the set of uncertain samples Duncertain
by defining upper and lower boundary.
2. Select the set of diversity samples Ddivers by
taking total # samples from the clusters in
uncertainty sample set Duncertain .
3. Initialized by the sample
xtop ¼ argmaxxi 2Ddivers CSðxi Þ, DD ¼ fxtop g.
4. xtop ¼ argminxi 2DdiversnD fmaxxj 2DD dðxi ; xj Þg;
D
DD ¼ DD [ fxtop g
5. Repeat 4 until jDD j ¼ c.
Step 4. Human annotator corrects incorrectly labeled
samples in DD .
Step 5. Retrain F t using DD ¼ Dtr [ DD ;
Dimp ¼ Dimp  DD .
Until a convergence criterion is satisfied or Dimp ¼ ;.
The time complexity of ASSL task would be simply O
(NLG) time to choose next query and require O(C2NLG),
where C is the number of class labels, N the size of unla-
beled pool, L the size of the current training set, G the num-
ber of gradient computations (Xu & Akella R., 2008).
5. Experiments
The main objectives of the experiments are to confirm
the effectiveness of the framework in different domains
and analyze the performances of our proposed framework
in learning detection models incrementally from imperfect
data. In other words, we want to see the performance is
increasing smoothly when new samples added to the train-
ing set. Depending on the data is presented to the model at
learning stage, each run of the framework on same dataset
shows variances in accuracies. Therefore, we run the same
experiments multiple times with the same parameter setting
and report the average of the results in this paper.
We carried out different kinds of experiments in order
do followings: (1) Compare the proposed ASSL method
with other methods used in sample selection, (2) evaluate
the incremental learning, and (3) confirm effectiveness of
our framework in different domains including object detec-
tion, pedestrian detection and action recognition.
5.1. Dataset overview
We conduct extensive experiments on ILSVRC detec-
tion dataset (Russakovsky et al., 2015), Caltech pedestrian
detection dataset (Dollar et al., 2011) and online evaluation
of activity recognition to verify the effectiveness of our
framework. To make initial model, we use MS COCO
detection dataset (Lin et al., 2014). Below we briefly
describe the datasets used in our experiments.
MS COCO. MS COCO dataset has 80 categories, fewer
categories than ILSVRC (Russakovsky et al., 2015). But it
has more instances per category which lets to learn more
complex and general models. It contains around 82,700
training, 40,500 validation, and 40,700 testing images.
Since MS COCO (Lin et al., 2014) consists of the samples
labeled with correct object localization, poses and object
categories, we employee it as a confident dataset and use
it to make an initial detection model.
ILSVRC. ILSVRC detection dataset has 200 categories.
There are three set of images and labels in ILSVRC2015
detection dataset (Russakovsky et al., 2015), training data
(456,567), validation data (20,121) and test data (40,152),
where in parentheses are the number of images in each
set. The val and test set are split from the same image dis-
tribution, in contrast, training set is drawn from ILSVRC
classification image distribution. All instances in val and
test set are fully annotated with bounding boxes. Unlike
validation and test set, training set is not fully annotated,
imbalanced and many examples are misclassified. ILSVRC
dataset (Russakovsky et al., 2015) used as imperfect data-
set, and semi-confident dataset is constructed from imper-
fect dataset by adopting proposed ASSL method. We
select 30,000 person images from ILSVRC training data
for ASSL learning, which is divided into 6 batches (5000
for each batch). Around 200 images randomly sampled
from ILSVRC dataset set, and used as test set. We make
several test sets and evaluate our framework on these test
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
sets and report mean of the results. We primarily evaluate
detection Average Precision (AP) for ILSVRC, because
this is the actual metric for object detection.
Caltech pedestrian detection dataset. We also evaluated
effectiveness of our method on the Caltech dataset
(Dollar et al., 2011), using subsets from set00 to set05 for
training and from set06 to set10 for testing. The Caltech
dataset is the dominant pedestrian detection datasets. It
consists of approximately 10 h of 30 Hz video taken from
a vehicle driving through street of Los Angeles, USA.
There are 350 k bounding boxes of around 2300 unique
pedestrians in 250 k frames. The Caltech evaluation devkit
provides several subsets of test set different aspect ratio,
scale, occlusion and overlaps. Overall performance is eval-
uated on the entire test set. In the reasonable evaluation
setting (Dollar et al., 2011), the performance is evaluated
on unoccluded pedestrians over 50 pixels tall. We mainly
consider overall and reasonable subsets for evaluating
our method. Readers are referred to (Dollar et al., 2011)
for detailed explanation of different types of evaluation set-
tings. We follow the evaluation rule of Caltech pedestrian
detection benchmark (Dollar et al., 2011), that uses the
log-average miss rate measured by averaging miss rate at
nine False Positive Per Image (FPPI) rates ranging from
102 to 100.
5.2. Experiment setup
Our experiments use publicly available ZF net (Zeiler &
Fergus, 2014) that has 5 convolutional layers and 3 fully
connected layers and VGG16 (Simonyan & Zisserman,
2015) that has 13 convolutional layers and 3 fully con-
nected layers. This networks are pre-trained ImageNet
for 1000 category classification task. For the detection sys-
tem ASSL employees the Faster-RCNN (Ren et al., 2015)
and ASSL frameworks is implemented on the popular deep
learning tool Caffe (Jia et al., 2014). All implementations
are on a single server with cuDNN (Chetlur & Woolley,
2014) and a single NVIDIA GeForce GTX 970.
For all datasets, we first fine-tune a pre-trained Ima-
geNet model on MS COCO dataset (Lin et al., 2014) to
make an initial model. Since categories on MS COCO
are is superset of ILSVRC (Russakovsky et al., 2015), we
can transfer previous knowledge COCO to ILSVRC and
Caltech (Dollar et al., 2011). We fine-tune only from the
conv3_1 convolutional layer and above. To train deep con-
volution neural network, we follow the guidelines described
in (Ren et al., 2015). The 1000 output channels classifica-
tion layer of the ImageNet model is replaced with 81 out-
put channels classification layer for the 80 classes of MS
COCO detection dataset plus one for background. We
trained the pre-trained ImageNet model for 240 k itera-
tions with a learning rate of 0.001 and then for 80 k itera-
tions with 0.0001 on MS COCO dataset. Completing this
step we get an initial detection model of our framework.
We spilt the training dataset into the confidently labeled
dataset and imperfectly dataset using a method modified
115
from the crowd sourcing (Su, Deng, & Fei-Fei 2012)
method in deciding bounding boxes instead of simple
annotation method based on majority voting, i.e., consen-
sus of multiple workers, (Deng et al. 2009; Sorokin &
Forsyth 2008) for our object detection experiments. The
challenge is how to obtain high annotation quality and
proper coverage for both perfect and imperfect datasets
with minimum cost. The steps are box drawing, quality ver-
ification, and coverage verification. In box drawing step,
the initial detection model is used to draw bounding boxes.
In the quality verification step, a worker decides whether a
bounding box is correctly drawn (perfect) or not (imper-
fect). In the third step, a second worker investigate whether
all objects in a scene have bounding boxes or not.
5.3. Experiments on ILSVRC dataset
In experiments on ILVRC dataset, we replaced 81 out-
put channels of VGG16 (Simonyan & Zisserman, 2015)
pre-trained net on MS COCO dataset by 2 output channels
for 2 classes of ILSVRC detection dataset (person and plus
one for background) and fine-tune it on ILSVRC person
data. In increment learning step, we continue previous
training on the semi-confident dataset which constructed
from the imperfect dataset using proposed ASSL method.
In increment learning phase, we used same learning rate
of 0.0001.
ASSL sampling parameters are selected by a grid search
(Girshick et al., 2014) over g = {0.5, 0.6, . . . , 0.9}, # =
{0.4, 0.5, . . . , 0.8} c = {0.2, 0.3, . . . , 0.6} on validation set
and set as uncertainty parameter g = 0.6, diversity param-
eter # = 0.5 and confidence parameter c = 0.3 which are
the values give the best performance. As an optimization
method for CNN training, we use mini-batch stochastic
gradient descent (SGD) with mini-batch size of 256 exam-
ples, weight decay of 0.0005 and the momentum of 0.9.
5.3.1. Comparison with the proposed ASSL with other
sampling methods
Fig. 3 shows the averaged detection accuracy versus the
number of semi-labeled samples obtained in the 1st and 6th
batch experiments. We compare ASSL with random sam-
pling, uncertainty sampling and confidence sampling. At
the first batches model is not stable, but on the following
batches becomes more stable. As shown in Fig 3, our
method performs better than compared methods; uncer-
tainty, confidence and random sampling. In Fig 3, orange
color indicates detection accuracy when we use all training
samples without any sampling method. Our proposed
ASSL sampling provides almost the same detection perfor-
mance with when all training samples used even it used few
small number of training samples than one used all sam-
ples. On the all batches, ASSL method is able to signifi-
cantly increase the detection accuracy.
As described in Section 3, since all of the samples in
training set are not important for efficient training, ASSL
selects only informative samples by combining AL and
116 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123

Fig. 2. Some examples of imperfect samples in the training set. Left: An example of missing labels of an object. Left middle: Mislabeled sample. Right
middle: An example of missing labels and mislabeling of multiple objects. In this example, computer keyboard, computer mouse, table and lamp’s labels
are missing and a laptop is mislabeled as TV monitor. Right: An example of a noisy sample.

Fig. 3. Average detection accuracy of the 1st and 6th batches over 5 runs provided by random, uncertainty, confidence and the proposed ASSL for person
category of ILSVRC dataset on each batch. (Higher curved indicates better performance).

SSL based on the collaborative sample selection. In Fig. 4,
we show some qualitative examples that indicate proposed
ASSL selects more informative samples than random
sampling.
Average detection performance on each batch is
reported in Table 1 and illustrated in Fig. 5. As shown in
Table 1, our method incrementally improves detection per-
formance at each batch and outperforms compared
methods.
Random sampling
ASSL sampling
5.3.2. Evaluation of incremental learning
In this experiment, we analysis effectiveness of incremen-
tal learning. At the beginning a detection may be misclassi-
fied, poor localization or it may be correctly classified with
a low probability score by the detection model. However,
in our framework, the models continue to improve at each
batch and later it can correctly classify the same misclassi-
fied activities with a higher probability score or provide
good localization. In Table 2, we report top false positive

Fig. 4. An illustrative comparison between random sampling and our proposed ASSL sampling method. We show four selected examples by random
sampling and ASSL sampling method for person category of ILSVRC dataset. Random sampling selects ill-posed or noisy samples, whereas ASSL
sampling selects samples with clear poses.
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123 117

Table 1
Comparison of average detection performance between different sampling methods; random, uncertainty, confidence and the proposed ASSL sampling for
person category of ILSVRC dataset. (Higher result indicates better performance).
Batch0 Batch1 Batch2 Batch3 Batch4 Batch5 Batch6
ASSL 64.1 69.4 70.8 71.2 71.6 72.2 72.7
Random 64.1 67.7 68.2 68.9 69.6 70.3 70.5
Uncertainty 64.1 68.2 68.7 70.2 70.1 71.6 71.7
Confidence 64.1 69.0 69.2 69.5 70.7 71.4 71.6

Fig. 5. Average detection accuracy over 6 batches with 5 runs provided by different sampling methods; random, uncertainty, confidence and the proposed
ASSL sampling for person category of ILSVRC dataset.

Table 2 Zisserman, 2015) model pre-trained on MS COCO dataset

Top false positive rates on each batch. Loc – poor localization, Sim – (Lin et al., 2014) by 2 output channels (pedestrian and
confusion with similar objects, other – confusion with other objects and background) and fine-tune it on Caltech train set. Same
BG – confusion with the background. (Lower result indicates better
as experiments on ILSVRC dataset, ASSL sampling
performance).
parameters are selected by a grid search and set as uncer-
Total Loc Sim Other BG
tainty parameter g = 0.8, diversity parameter # = 0.6 and
Batch1 1090 0.131 0.005 0.077 0.787 confidence parameter c = 0.5 which are the values give
Batch2 1072 0.127 0.002 0.076 0.796
the best detection performance. Other settings are the same
Batch3 1070 0.136 0.003 0.078 0.783
Batch4 1075 0.13 0.002 0.081 0.787 as experiment settings on ILSVRC dataset.
Batch5 1068 0.13 0.002 0.082 0.768
Batch6 1067 0.118 0.003 0.087 0.792 5.4.1. Comparison with the proposed ASSL with other
sampling methods
rates on each batch. In the beginning of the learning, there We compare the performance of our method with differ-
are total 1090 false positives. After incrementally updating ent sampling methods on the Caltech test set (Dollar et al.,
model on semi-confident dataset constructed by ASSL, it is 2011), including random, most uncertain and confidence
decreased into 1067 false positives. We can observe that the sampling. The comparison of log-average miss rate under
model becoming smarter later batches. In order to calculate different evaluation settings on Caltech pedestrian detec-
false positive impact and rate, we applied detection analysis tion dataset is summarized in Table 3. Our initial model
tool from Hoiem, Chodpathumwan, and Dai (2012). The which is the VGG16 (Simonyan & Zisserman, 2015) model
readers are referred to Hoiem et al. (2012) for more details fine-tuned on MS COCO (Lin et al., 2014) dataset achieves
about the analysis tool. miss rate of 31.4% performance in reasonable setting.
Figs. 6 and 7 report the comparison of false positive Incremental learning with random sampling achieves
types between the first batch and the last batch. As shown 25.3% miss rate. The detection results can be improved to
in Fig. 6, on batch 6 correct detection rate increases from miss rate of 12.2% by implementing proposed ASSL
batch 1, 58% to 63%. This confirms that effectiveness of method.
incremental learning. In overall evaluation setting, miss rates of all methods
are increased because overall setting uses all images in test
5.4. Experiment results on Caltech dataset set including very small pedestrian such as 30 pixels tall and
pedestrians occluded more than 75%. Performance mea-
In experiments on Caltech pedestrian dataset, we surements shown in Table 3 are described in Section 5.1.
replaced 81 output channels of VGG16 (Simonyan & Our initial model which is fine-tuned on MS COCO dataset
118 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123

(a) Batch 1 (b) Batch 6

Fig. 6. Evaluation of top-ranked false positives in incremental learning. Pie charts shows correct (Cor) labeling percent improved after several batches. Pie
charts: fraction of top-ranked false positives that are due to poor localization (Loc), confusion with similar objects (Sim), confusion with other objects
(Oth) or confusion with background or unlabeled objects (BG).

(a) Batch 1 (b) Batch 6

Fig. 7. Comparison of top ranked false positive types between batch 1 and batch 6. Cor – correctly detected, Loc – poor localization (a detection with IoU
overlap with the correct label between 0.1 and 0.5), Sim – confusion with similar objects, BG – confusion with background and Oth – confusion with other
objects. Line plots show recall.

Table 3
Comparison of log-average miss rate on Caltech dataset between ASSL method and different sampling methods. (Lower result indicates better
performance).
Sampling Overall Reasonable Scale Aspect ratio Occlusion Overlap
Far Large Medium Near All A typical Typical Heavy None Partial 25 50 75
Initial 75.6 31.4 100 4.9 71.7 9.1 28.8 34.9 27.4 88 28.8 49.5 24.5 31.4 73.9
Random 73.9 25.3 99.8 2.8 69.9 4.4 22.8 34.3 20.4 71.5 22.8 44.2 22.6 25.3 69.4
Confidence 71.4 18.7 99.1 1.1 67.5 3.9 16.7 24 15.1 66.7 16.7 34.9 14.9 18.7 68.4
Uncertainty 69.2 17 98.9 1.1 65.1 3.8 15.2 23.6 13.3 59.6 15.2 32.8 13.5 17 66.3
ASSL 62.5 12.2 98.8 1 54.2 2.7 9.7 19.2 7.7 54.7 9.7 29.2 8.5 12.2 66

achieves miss rate of 75.6, while our final result provided by
ASSL achieves significantly good performance of 62.5%
miss rate. Miss rates versus false positive per-image curves
shown in reasonable and overall evaluation setting are
shown in Fig. 8.
5.4.2. Comparison with state-of-art methods
We compare the performance of the proposed ASSL
method with existing state-of-the-art methods on the Cal-
tech test set, including DeepCascade+ (Angelova et al.,
2015), Katamari (Benenson, Omran, Hosang, & Schiele,
2015), TA-CNN (Tian, Luo, Wang, & Tang, 2015), Spar-
tialPooling+ (Paisitkriangkrai, Shen, & Hengel, 2014),
SCF + AlexNet (Hosang, Omran, Benenson, & Schiele,
2015), SCCPriors (Yang, Wang, & Wu, 2015), Checker-
boards (Zhang et al., 2015), CCF + CF (Yang, Yan, Lei,
& Li, 2015) and CCF (Yang et al., 2015). The comparison
of log-average miss rate between proposed ASSL and the
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123 119

(a) Reasonable (b) Overall

Fig. 8. The comparison of ASSL with different sampling methods and baseline result on Caltech pedestrian dataset (lower curves indicate better
performance). (a) Log-average miss rate in reasonable evaluation setting (unoccluded pedestrians over 50 pixels tall). (b) Overall performance on all
images in test set.

state-of-the-art methods is reported in Table 4. We evalu-
ate the comparison under different evaluation settings on
Caltech pedestrian test set. Among the comparison under
different settings, we illustrate the comparison with state-
of-the-art methods under reasonable and overall evalua-
tion settings in Fig. 9.
It can be observed that our method outperforms other
methods on most set subsets. ASSL achieves the lowest
log-average miss rate of 12.2%, which is significantly lower
than the current state-of-the-art method Checkerboards
(Zhang et al., 2015), by 4.9% in reasonable evaluation
setting.
5.5. Online evaluation of ASSL for action recognition
Results on the ILSVRC object detection and Caltech
pedestrian detection datasets show that the ASSL frame-
work can significantly outperform many other models on
object detection and pedestrian detection tasks and
confirm that the proposed method can be applied a wide
variety of application domains including cognitive
systems, automotive applications and surveillance systems.
In this section, we present that the ASSL method can be
applied to solving imperfect data problem in human action
recognition task. In online evaluation of ASSL for action
recognition, we define five types of human actions:
jumping, phoning, reading, taking photo, using a com-
puter. We collected around 7000 frames using web camera
that contain the actions we defined. In training phase,
ASSL selects the most informative samples from the
collected unlabeled data then incrementally updates the
model using the samples selected by ASSL. After complet-
ing training stage, we evaluate our model in real-time
action recognition. Same as previous experiments, ASSL
sampling parameters are selected by a grid search and
set as uncertainty parameter g = 0.8, diversity parameter
# = 0.7 and confidence parameter c = 0.4 which are the
values give the best performance. We illustrate some
example snapshots while evaluating real time online action
recognition in Fig. 10.

Table 4
Comparison with state-of-art methods. (Lower result indicates better performance.)
Methods Overall Reasonable Scale Aspect Ratio Occlusion Overlap
Far Large Medium Near All A typical Typical Heavy None Partial 25 50 75
DeepCascade+ 71.9 26.2 100 4 64.8 7.3 23.2 31.4 21.4 82.2 23.2 47.7 21.3 26.2 73.9
Katamari 71.3 22.5 100 6.8 63.1 9.8 20.1 30.2 18.1 84.4 20.1 41.7 18.5 22.5 62.6
TA-CNN 71.2 20.9 100 7 63.6 8 19 26.8 15.7 70.4 18.6 32.8 16.8 20.9 60
SpartialPooling+ 71.1 21.9 100 7.7 63.4 8 19.5 26.8 17.8 78.2 19.5 39.2 18.4 21.9 61.1
SCF + AlexNet 70.3 23.3 100 7 62.3 10.6 20 30.5 17.8 74.6 20 48.5 20.2 23.3 58.9
SCCPriors 70.3 21.9 100 8.8 61.3 8 19.3 26.7 17.6 80.9 19.3 41.3 17.4 21.9 61.6
Checkerboards+ 67.7 17.1 100 2.4 58 4.9 15.1 20.8 13.4 77.9 15.1 31.3 13.3 17.1 55.1
CCF + CF 68.6 17.3 100 4.2 59.6 5.5 14.6 22.6 12.8 72.7 14.6 37.7 13.1 17.3 83.4
CCF 66.7 18.7 100 2.9 56.3 4.7 15.9 25 14.2 72.4 14.6 40.6 14.8 18.7 84.7
ASSL 62.5 12.2 98.8 1 54.2 2.7 9.7 19.2 7.7 54.7 9.7 29.2 8.5 12.2 66
120 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123

(a) Reasonable (b) Overall

Fig. 9. The comparison of ASSL with state-of-art methods on Caltech pedestrian dataset (lower curves indicate better performance). (a) Log-average miss
rate in reasonable evaluation setting (unoccluded pedestrians over 50 pixels tall). (b) Overall performance on all images in test set.

Fig. 10. Example snapshots of real-time online evaluation of action recognition.

Table 5
Action recognition accuracy in online evalluation.
Avg. acc Jumping Phoning Reading Taking a photo Using a computer
Baseline 56.5 ± 0.7% 67.1 ± 0.8% 51.1 ± 0.6% 58.3 ± 0.2% 50.4 ± 0.9% 55.6 ± 0.3%
Random 74.8 ± 0.4% 78.5 ± 0.8% 75.9 ± 0.1% 76.8 ± 0.3% 72.9 ± 0.6% 69.9 ± 0.4%
ASSL 82.2 ± 0.8% 86.4 ± 00.0.4% 75.2 ± 0.9% 84.9 ± 0.7% 79.7 ± 0.5% 84.7 ± 0.6%

shows variances in accuracies. Therefore, we run the same

Fig. 11. Action-wise performance analysis compared proposed ASSL with
baseline result and random sampling in online evaluation.
Depending on the data presented to the model in learn-
ing phase, each run of the framework on same dataset
experiments multiple times with the same parameter setting
and report the average of the results in this paper. We pre-
sent over 200 frames in each run and performed 10 run in
online evaluation. In order to get accuracy, we divide num-
ber of correct action by the total number of frames pre-
sented to the detector. Online evaluation result is shown
in Table 5. We compare our ASSL method with against
baseline and random selection. ASSL outperforms random
selection and gives significant improvement compared to
the baseline.
In Fig 11, we show the performance of our framework
on each action categories. Each group of stacked bars
shows performances of action categories. Each group con-
tains three bars corresponding to baseline accuracy, ran-
dom sampling accuracy and proposed ASSL method
accuracy.
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
6. Discussion and conclusion
In this paper we presented the novel of the active learn-
ing (AL) and semi-supervised (SSL) learning, called ASSL,
to leverage the strong points of the both learning para-
digms for improving the performance of object detection.
Considering the pros and cons of the AL and SSL learning
methods, ASSL where SSL method provides the incremen-
tal improvement of semi-supervised detection performance
by combining the concept of diversity imported from AL
methods. We evaluated extensive experiments on ILSVRC
detection dataset and Caltech pedestrian detection dataset.
These experiments lead us to several observations that
will be useful in developing future a detection system based
on imperfect data. First, the results show that it is possible
to achieve detection performance that is similar to the per-
formance obtained with fully labeled data, even when a
small fraction of the training data is used in the training
set. Second, as a practical matter, the experiments show
that the active and semi-supervised learning can be applied
to an existing detector that was originally designed for
supervised training. Experiment results show that the
ASSL framework can significantly outperform many other
models on object detection and pedestrian detection tasks
and confirm that the proposed method can be applied a
wide variety of application domains including cognitive
systems, automotive applications, vision based intelligent
systems and surveillance systems.
Acknowledgement
This work was supported by Inha University research
grant. GPUs used in this research was generously donated
by NVIDIA Corporation.
References
Angelova, A., Krizhevsky, A., View, M., View, M., Vanhoucke, V., Ogale,
A., & Ferguson, D. (2015). Real-time pedestrian detection with deep
network cascades. BMVC, 2015, 1–12, Retrieved from<http://www.
vision.caltech.edu/anelia/publications/Angelova15RealTimePedes-
trian.pdf>.
Benenson, R., Omran, M., Hosang, J., & Schiele, B. (2015). Ten years of
pedestrian detection, what have we learned? Lecture notes in computer
science (including subseries lecture notes in artificial intelligence and
lecture notes in bioinformatics), 8926, 613–627. http://dx.doi.org/
10.1007/978-3-319-16181-5_47.
Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning:
A review and new perspectives. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 35(8), 1798–1828. http://dx.doi.org/10.1109/
TPAMI.2013.50.
Breiman, L. (1996). Bagging predictors. Machine Learning, 24(421),
123–140. http://dx.doi.org/10.1007/BF00058655.
Brinker, K. (2003). Incorporating diversity in active learning with support
vector machines. Strategy, 20, 59, Retrieved from<http://www.aaai.
org/Papers/ICML/2003/ICML03-011.pdf>.
Chapelle. (2006). Semi-supervised learning. Interdisciplinary sciences com-
putational life sciences (Vol. 1). doi:http://dx.doi.org/10.1007/s12539-
009-0016-2.
121
Chetlur, S., & Woolley, C. (2014). cuDNN: Efficient primitives for deep
learning. arXiv Preprint arXiv: . . ., 1-9. Retrieved <http://arxiv.org/
abs/1410.0759>.
Culotta, A., & Mccallum, A. (2005). Reducing labeling effort for
structured prediction tasks. Proceedings of the National Conference
on Artificial Intelligence, 20(2), 746, Retrieved from<http://scholar.-
google.com/scholar?hl=en&btnG=Search&q=intitle:Reducing+label-
ing+effort+for+structured+prediction+tasks#0>.
Dagan, I., & Engelson, S. P. (1995). Committee-based sampling for
training probabilistic classifiers. Proceedings of the Twelfth Interna-
tional Conference on Machine Learning, 150–157http://doi.org/10.1.1.
30.6148.
Demir, B., Persello, C., & Bruzzone, L. (2011). Batch-mode active-
learning methods for the interactive classification of remote sensing
images. IEEE Transactions on Geoscience and Remote Sensing, 49(3),
1014–1031. http://dx.doi.org/10.1109/TGRS.2010.2072929.
Deng, J, Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009).
ImageNet: A large-scale hierarchical image database. In CVPR09.
Dollar, P., Wojek, C., Schiele, B., & Perona, P. (2011). {P}edestrian
detection: {A}n evaluation of the state of the art. IEEE Transaction on
Pattern Analysis and Machine Intelligence, 1–2.
Felzenszwalb, P.F., Girshick, R.B., Mcallester, D., & Ramanan, D.
(2009). Object detection with discriminatively trained part based
models, doi:http://dx.doi.org/10.1109/TPAMI.2009.167.
Fergus, R., Fei-Fei, L., Perona, P., & Zisserman, A. (2005). Learning
object categories from Google’s image search. In Tenth IEEE
international conference on computer vision (ICCV’05) volume 1 (Vol.
2, pp. 1816–1823). doi:10.1109/ICCV.2005.142.
Girshick, R. (2015). Fast R-CNN, doi:http://dx.doi.org/10.1109/ICCV.
2015.169.
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature
hierarchies for accurate object detection and semantic segmentation. In
Proceedings of the IEEE computer society conference on computer
vision and pattern recognition (pp. 580–587), doi:http://dx.doi.org/10.
1109/CVPR.2014.81.
Hasan, M., & Roy-Chowdhury, A. K. (2015). A continuous learning
framework for activity recognition using deep hybrid feature models.
IEEE Transactions on Multimedia, 17(11), 1909–1922. http://dx.doi.
org/10.1109/TMM.2015.2477242.
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Spatial pyramid pooling in
deep convolutional networks for visual recognition. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 37(9), 1904–1916. http://
dx.doi.org/10.1109/TPAMI.2015.2389824.
Heckman, J. (1979). Sample selection bias as a specification error.
Econometrica, 47(1), 153–161. http://dx.doi.org/10.2307/1912352.
Hinton, G. E. (2006). Reducing the dimensionality of data with neural
networks. Science, 313(5786), 504–507. http://dx.doi.org/
10.1126/science.1127647.
Hoiem, D., Chodpathumwan, Y., & Dai, Q. (2012). Diagnosing error in
object detectors. In Lecture notes in computer science (including
subseries lecture notes in artificial intelligence and lecture notes in
bioinformatics) (Vol. 7574 LNCS(PART 3), pp. 340–353), doi:http://
dx.doi.org/10.1007/978-3-642-33712-3_25.
Hosang, J., Omran, M., Benenson, R., & Schiele, B. (2015). Taking a
deeper look at pedestrians. In Proceedings of the IEEE computer
society conference on computer vision and pattern recognition (Vol. 07-
12-June, pp. 4073–4082), doi:http://dx.doi.org/10.1109/CVPR.2015.
7299034.
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R.,
. . ., Darrell, T. (2014). Caffe: Convolutional architecture for fast
feature embedding. In Proceedings of the ACM international conference
on multimedia (pp. 675–678), doi:http://dx.doi.org/10.1145/2647868.
2654889.
Kapoor, A., Grauman, K., Urtasun, R., & Darrell, T. (2007). Active
learning with gaussian processes for object categorization. In 2007
IEEE 11th international conference on computer vision, doi:http://
dx.doi.org/10.1109/ICCV.2007.4408844.
122 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet
classification with deep convolutional neural networks. Advances in
Neural Information Processing Systems, 1–9. http://dx.doi.org/
10.1016/j.protcy.2014.09.007.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E.,
Hubbard, W., & Jackel, L. D. (1989). Backpropagation applied to
handwritten zip code recognition. Neural Computation. http://dx.doi.
org/10.1162/neco.1989.1.4.541.
Lewis, D. D., & Catlett, J. (1994). Heterogeneous uncertainty sampling for
supervised learning. In Proceedings of the 11th international conference
on machine learning (ICML’94) (pp. 148–156) <http://www.cs.bryn-
mawr.edu/cs372/LeC94.pdf>.
Lewis, D. D., & Gale, W. A. (1994). A sequential algorithm for training
text classifiers. In Proceedings of the 17th international conference on
research and development in information retrieval (SIGIR’94) (pp. 3–
12), doi:http://dx.doi.org/10.1145/219587.219592.
Li, M., & Sethi, I. K. (2006). Confidence-based active learning, 28(8),
1251–1261.
Li, L. J., & Fei-Fei, L. (2010). OPTIMOL: Automatic online picture
collection via incremental model learning. International Journal of
Computer Vision, 88(2), 147–168. http://dx.doi.org/10.1007/s11263-
009-0265-6.
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., . . .,
Zitnick, C. L. (2014). Microsoft COCO: common objects in context. In
Lecture notes in computer science (including subseries lecture notes in
artificial intelligence and lecture notes in bioinformatics) (Vol. 8693
LNCS, pp. 740–755), doi:http://dx.doi.org/10.1007/978-3-319-10602-
1_48.
Lindenbaum, M., Markovitch, S., & Rusakov, D. (2004). Selective
sampling for nearest neighbor classifiers. Machine Learning, 54
(2), 125–152. http://dx.doi.org/10.1023/B:MACH.0000011805.
60520.fe.
Lu, Z., Wu, X., & Bongard, J. C. (2015). Active learning through adaptive
heterogeneous ensembling. IEEE Transactions on Knowledge and Data
Engineering, 27(2), 368–381. http://dx.doi.org/10.1109/
TKDE.2014.2304474.
Makantasis, K., Doulamis, A., Doulamis, N., & Psychas, K. (2016). Deep
learning based human behavior recognition in industrial workflows. In
IEEE international conference on image processing (ICIP), 2016 (pp.
1609–1613). IEEE.
Mitchell, T., & Blum, A. (1998). Combining labeled and unlabeled data
with co-training. In Proceedings of the eleventh annual conference on
computational learning theory (pp. 92–100). doi:http://dx.doi.org/10.
1145/279943.279962.
Paisitkriangkrai, S., Shen, C., & Hengel, A. Van Den. (2014). Pedestrian
detection with spatially pooled features and structured ensemble
learning, 1–19, doi:http://dx.doi.org/10.1109/TPAMI.2015.2474388,
arXiv Preprint arXiv:1409.5209.
Persello, C., & Bruzzone, L. (2011). Active and semi-supervised learning
for the classification of remote sensing images. IEEE Transactions on
Geoscience and Remote Sensing, 52(11), 6937–6956, doi.org/81800c
10.1117/12.898483.
Qi, G.-J., Hua, X.-S., Rui, Y., Tang, J., & Zhang, H.-J. (2008). Two-
Dimensional Active Learning for image classification. In IEEE
conference on computer vision and pattern recognition (pp. 1–8),
doi:http://dx.doi.org/10.1109/CVPR.2008.4587383.
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards
real-time object detection with region proposal networks. Nips, 1–10.
http://dx.doi.org/10.1016/j.nima.2015.05.028.
Roy, N., & McCallum, A. (2001). Toward optimal active learning through
sampling estimation of error reduction. in Proceedings of the 18th
international conference on machine learning (pp. 441–448). Retrieved
from <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28.
9963&rep=rep1&type=pdf>.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., ... Fei-
Fei, L. (2015). ImageNet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3), 211–252. http://dx.
doi.org/10.1007/s11263-015-0816-y.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun,
Y. (2013). OverFeat: Integrated recognition, localization and detection
using convolutional networks. arXiv Preprint arXiv:1312.6229.
Retrieved from <http://arxiv.org/abs/1312.6229>.
Settles, B. (2010). Active learning literature survey. Machine Learning, 15
(2), 201–221http://doi.org/10.1.1.167.4245.
Seung, H. S., Opper, M., & Sompolinsky, H. (1992). Query by committee.
Proceedings of the fifth annual workshop on computational learning
theory - COLT’92, 287–294. http://dx.doi.org/10.1145/130385.130417.
Siddiquie, B., & Gupta, A. (2010). Beyond active noun tagging: Modeling
contextual interactions for multi-class active learning. In Proceedings
of the IEEE computer society conference on computer vision and pattern
recognition (pp. 2979–2986), doi:http://dx.doi.org/10.1109/CVPR.
2010.5540044.
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks
for large-scale image recognition. ImageNet Challenge, 1–10. http://dx.
doi.org/10.1016/j.infsof.2008.09.005.
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks
for large-scale image recognition. Iclr, 1–14. http://dx.doi.org/10.1016/
j.infsof.2008.09.005.
Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon
Mechanical Turk. In 2008 IEEE Computer society conference on
computer vision and pattern recognition workshops, CVPR workshops,
doi:http://dx.doi.org/10.1109/CVPRW.2008.4562953.
Su, H., Deng, J., Fei-Fei, Li. (2012). Crowdsourcing annotations for visual
object detection. Human Computation AAAI Technical Report WS-
12-08.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., . . .,
Rabinovich, A. (2015). Going deeper with convolutions. In Proceed-
ings of the IEEE computer society conference on computer vision and
pattern recognition, 07-12-June (pp. 1–9), doi:http://dx.doi.org/10.
1109/CVPR.2015.7298594.
Tian, Y., Luo, P., Wang, X., & Tang, X. (2015). Pedestrian detection
aided by deep learning semantic tasks. In Proceedings of the IEEE
computer society conference on computer vision and pattern recognition
(Vol. 07-12-June, pp. 5079–5087). doi:http://dx.doi.org/10.1109/
CVPR.2015.7299143.
Tong, S., & Koller, D. (2001). Support vector machine active learning with
applications to text classification. Journal of Machine Learning
Research, 45–66. http://dx.doi.org/10.1162/153244302760185243.
Uijlings, J. R. R., Van De Sande, K. E. A., Gevers, T., & Smeulders, A.
W. M. (2013). Selective search for object recognition. International
Journal of Computer Vision, 104(2), 154–171. http://dx.doi.org/
10.1007/s11263-013-0620-5.
Vijayanarasimhan, S., & Grauman, K. (2009). Multi-level active predic-
tion of useful image annotations for recognition. Advances in Neural
Information Processing Systems, 21, 1705–1712.
Welinder, P., & Perona, P. (2010). Online crowdsourcing: Rating
annotators and obtaining cost-effective labels. In 2010 IEEE computer
society conference on computer vision and pattern recognition -
workshops, CVPRW 2010 (pp. 25–32). doi:http://dx.doi.org/10.1109/
CVPRW.2010.5543189.
Xu, Z. & Akella R. (2008). Active relevance feedback for difficult queries
In Proceedings of the 17th ACM conference on Information and
knowledge management, Napa Valley, California, USA (pp. 459–468).
Xu, Z., Yu, K., Tresp, V., Xu, X., & Wang, J. (2003). Representative
sampling for text classification using support vector machines. In
Proceedings of ECIR-03, 25th European conference on information
retrieval (pp. 393-407). Retrieved from <http://link.springer.de/link/
service/series/0558/papers/2633/26330393.pdf>.
Xu, L., Li, B., & Chen, E. (2012). Ensemble pruning via constrained eigen-
optimization. In Proceedings - IEEE international conference on data
mining, ICDM (pp. 715–724). doi:http://dx.doi.org/10.1109/ICDM.
2012.97.
Yang, B., Yan, J., Lei, Z., & Li, S. Z. (2015). Convolutional channel
features. In ICCV, doi:http://dx.doi.org/10.1109/ICCV.2015.18.
Yang, Y., Wang, Z., & Wu, F. (2015). Exploring prior knowledge for
pedestrian detection. In BMVC2015 (pp. 1–12).
 P.K. Rhee et al. / Cognitive Systems Research 45 (2017) 109–123 123

Zadrozny, B. (2004). Learning and evaluating classifiers under sample
selection bias. In Twenty-first international conference on machine
learning - ICML ’04 (p. 114). doi:http://dx.doi.org/10.1145/1015330.
1015425.
Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding
convolutional networks. In Computer vision–ECCV 2014 (Vol. 8689,
pp. 818–833), doi:http://dx.doi.org/10.1007/978-3-319-10590-1_53,
arXiv:1311.2901v3 [cs.CV] 28 Nov 2013.
Zhang, S., Benenson, R., & Schiele, B. (2015). Filtered channel features
for pedestrian detection. In Proceedings of the IEEE computer society
conference on computer vision and pattern recognition (Vol. 07–12-June,
pp. 1751–1760). doi:http://dx.doi.org/10.1109/CVPR.2015.7298784.
Zhu, X. (2008). Semi-supervised learning literature survey contents.
Sciences New York, 10(1530), 10, http://doi.org/10.1.1.146.2352.