Applied Soft Computing 53 (2017) 181204

Contents lists available at ScienceDirect

Applied Soft Computing

journal homepage: www.elsevier.com/locate/asoc

A web page distillation strategy for efcient focused crawling based

on optimized Nave bayes (ONB) classier
Ahmed I. Saleh a, , Arwa E. Abulwafa a , Mohammed F. Al Rahmawy b
a
Dept. of Computer Eng. & Systems, Faculty of Engineering, Mansoura University, Mansoura, Egypt
b
Dept. of Computer Science, Faculty of Computers, Mansoura University, Mansoura, Egypt

a r t i c l e i n f o a b s t r a c t

Article history: The target of a focused crawler (FC) is to retrieve pages related to a specic domain of interest (DOI).
Received 15 September 2015 However, FCs may be hasted if bad links were injected into their crawling queue. Hence, they will be
Received in revised form 1 September 2016 gradually skewed away from their DOI. This paper introduces an effective modication on the behavior
Accepted 18 December 2016
of FCs by adding a domain distiller. Hence, before passing the retrieved page to the indexer or embedding
Available online 3 January 2017
its links into the crawling queue, the page must pass through a domain distiller. The proposed domain
distiller relies on an Optimized Nave Bayes (ONB) classier, which combines nave Bayes (NB) and Sup-
Keywords:
port Vector Machines (SVM). Initially, genetic algorithm (GA) is used to optimize the soft margins of
Wen page classication
Focused crawling
SVM. Then the optimized SVM is employed to eliminate the outliers from the available training exam-
Domain ontology ples. Next, the pruned examples are used to train the traditional NB classier. Moreover, ONB employs
Support vector machines word sense disambiguation (WSD) to identify the accurate sense of each domain keyword extracted from
Nave bayes the input page. This is accomplished by using a proposed domain ontology, which is called Disambigua-
Genetic algorithm tion Domain Ontology (D2 O). ONB has been tested against recent classication techniques. Experimental
results have proven the effectiveness of ONB as it introduces the maximum classication accuracy. Also,
results indicate that the proposed distiller improves the performance of focused crawling in terms of
crawling harvest rate.
2016 Elsevier B.V. All rights reserved.

1. Introduction
Due to the explosive growth in the eld of Internet and comput-
ers, millions of web pages are daily added to Internet. Accordingly,
searching the web becomes a true challenge [1]. Search engines
(SEs) are information retrieval systems, which are designed mainly
to help users nding what they need. A Crawler is one of the
main components of SE, which is used to retrieve web pages
then pass them to the indexer. In spite of their effectiveness,
the general-purpose search engines suffer from low precision and
recall, freshness problem, poor retrieval rate, time consuming due
to the long list of result, and storage problem caused by the huge
amount of expanded information. To overcome those problems,
more specialized (vertical) search engines (VSE), which are called
the domain-specic search engines [2], have been introduced. The
aim of VSEs is to reply the users queries in a specic domain of
interest (DOI), and accordingly, they have a special type of crawlers
Corresponding author.
E-mail address: aisaleh@yahoo.com (A.I. Saleh).
called focused crawlers [3]. A VSE offers a good solution for the
general-purpose search engines limitations, as it covers only the
portion of the web related to its DOI. On the other hand, VSEs can
easily provide more precise results and more customized functions.
However, building an accurate VSE is also a true challenge as the
web is full of noisy and volatile materials.
Focused crawling aims to download only pages related to a spe-
cic DOI. The web is then divided into a set of interconnected
domains, and then one (or more) crawler(s) is (are) allowed to
discover each domain. Hence, indexing the web can be done in par-
allel; and accordingly, more web coverage can be easily achieved.
However, the area of focused crawling still has many challenges
and unsolved problems. One of the effective problems that harm
the focused crawling efciency is the Haste Problem. Such prob-
lem takes place when bad links are injected into the crawling
queue of the focused crawler. After retrieving the pages behind
those bad links, more bad links are added to the crawling queue
causing the focused crawler to be skewed away from its DOI.
The main cause of the haste problem is that traditional focused
crawlers rely on estimation. Hence, they estimate whether the
page is related to DOI or not before actually retrieving the page.

http://dx.doi.org/10.1016/j.asoc.2016.12.028
1568-4946/ 2016 Elsevier B.V. All rights reserved.
182 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
Although recent focused crawlers implement accurate estimation
techniques, relevancy estimation is not always accurate. To over-
come such problem, web mining techniques can be applied to
calculate a true relevancy score that is based on the actual pages
contents rather than such inaccurate estimation. To accomplish
such aim, a domain distiller can be employed to calculate the rel-
evancy of the page after actually retrieving it. Then the decision
to pass the retrieved page to the search engines indexer or to add
the pages embedded links to the crawling queue is based on the
distillers decision.
Web page classication can be dened as the assignment of a
web page to one (or more) predened classes. It is often posed as
a supervised learning problem. Hence, a set of training examples
are used to train the classier by setting its classication rules,
which can be applied to classify future examples. Based on the
number of the employed classes, classication can be binary or
multi-class. Binary classication categorizes items into exactly one
of two classes, while multi-class classication employs more than
two classes. Web page domain distillers are binary classiers that
are designed to take a decision whether an input web page is related
to a specic DOI. Several classication techniques can be employed
in domain distillers such as; support vector machines (SVM) [4],
k-nearest Neighbor (KNN) [5], decision trees [6], neural networks
[7], and Bayesian classier [8]. However, binary classication is still
a challenge.
In current era, it has been found that results returned by search
engines are not what actually needed. This happens because, while
querying for certain data, there is a likelihood that the query
contains ambiguous (polysemous) words, which having multiple
meaning. Word Sense Disambiguation (WSD) [9], which tries to
assign a unique sense to a word, is an important area of NLP [10].
Generally, WSD can be employed to promote the classication per-
formance as it can effectively solve the classier confusion. Several
techniques can be used to implement WSD. One technique is based
on the collocation of other words in which nearby words are used to
provide consistent clues to the sense of a target word [9]. Another
technique is the word sense based on discourse in which the sense
is consistent within any given document.
The originality of this paper is concentrated in introducing a
new architecture for focused crawling by integrating evidence from
Machine Learning and Web Mining. The paper introduces an effec-
tive modication on the behavior of focused crawlers by employing
a domain distiller to decide whether the retrieved page is related
to crawlers DOI, and then take a decision to index the page and add
its links to the crawling queue accordingly. The proposed domain
distiller combines SVM and NB classiers in a new instance called
Optimized Nave Bayes (ONB) classier. Initially, genetic algorithm
(GA) is used to optimize the soft margins of SVM. Then, the opti-
mized SVM is employed to eliminate the outliers from the available
training examples. Next, the pruned examples are used to train
the traditional NB classier. Furthermore, in order to guarantee
an effective classication task, WSD has been implemented to cre-
ate innovative features to perfectly represent the input page for
the classication. With the help of WSD, a set of specially selected
ambiguous domain keywords, which is called Confusion Set (CS),
is identied. Then, the sense of each ambiguous keyword is identi-
ed based on a pre-stored collection of discriminative keywords
(for each ambiguous keyword), called Partners. CS is the sub-
set of ambiguous domain keywords that most likely confuse the
classier. ONB employs proposed domain ontology for both map-
ping domain keywords to the corresponding concepts as well as
implementing WSD. Hence, it is called; Disambiguation Domain
Ontology (D2 O). ONB has been tested against recent classication
techniques. Experimental results have proven the effectiveness of
ONB as it introduces the maximum classication accuracy. Also,
results indicate that the proposed distiller improves the perfor-
mance of focused crawling in terms of crawling harvest rate. This
paper is organized as follows; section 2 introduces the background
and basic concepts, section 3 illustrates the effective focused crawl-
ing that combines the traditional focused crawling with the domain
distiller, section 4 presents the previous efforts in the area of web
page classication, section 5 introduces the employed disambigua-
tion domain ontology (e.g., D2 O), section 6 illustrates in details
the proposed ONB classier, section 7 presents the performance
analysis and experimental results, while section 8 summarizes our
conclusions.
2. Background and basic concepts
In this section, an explanation about traditional focused
crawlers as well as the crawling haste problem will be introduced.
Then, a brief introduction for word sense disambiguation (WSD)
will be illustrated.
2.1. Web search engines
Search engines are the most popular search tools for nding
the required information on the web. A typical search engine con-
sists of ve basic components, which are; (i) crawler, which may
be focused or unfocused, (ii) indexer, (iii) database, (iv) query man-
ager, and (v) user interface. Hence, the user sends his query to
the search engine in the form of search keywords. Then search
engine retrieves the relevant pages to the users query from the
database. Finally, a ranked list of pages is presented to the user.
Search engines rely on crawlers to traverse the web [11]. Crawlers,
which may be focused or unfocused, collect pages, pass them to
the indexer, and then follow links from one page to another [12].
Indexer on the other hand, analyzes the page and stores its features
in the database.
2.2. Focused crawlers and the haste problem
A focused crawler [13] is a special type of crawlers that retrieves
pages related to a specic topic or a domain of interest based on
both content and link structures [14]. As illustrated in Fig. 1, the
focused crawler operates in ve steps; initially, its priority crawl-
ing queue is initialized with a number of seed pages [15], which are
highly relevant pages that are manually chosen. Then, the focused
crawler fetches the link located at the head of its priority queue
and retrieves the corresponding page. In the third step, it analyzes
the page (using parsers to extract keywords and links). Forth, the
focused crawler assigns a score for each link in the processed page
based on several criteria such as the link position in the page, the
links anchor window, and/or the pages rank. Those extracted links,
after scoring them, are injected into the crawling queue. Finally,
the crawler sorts the links in its queue so that the links with higher
scores appear at the queue head; hence, they will be processed rst.
A focused crawler will continue operating as its queue has URLs for
processing. This procedure ensures that the crawler moves towards
relevant pages with the assumption that relevant pages tend to be
neighbors to each other. However, focused crawlers are very sen-
sitive to the quality of the seeds initially injected into its queue.
Hence, falsely chosen seeds will dramatically affect the crawler per-
formance. Moreover, inaccurate link scoring strategy badly impacts
the crawler behavior in the future crawling cycles. Hence, more and
more bad links, which are extracted from the retrieved low quality
pages, will be added to the crawling queue. As a result, the crawler
may be involuntary skewed away from its main target, which is
retrieving high quality pages relevant to a specic domain. We call
such skew; the crawling Haste Problem.
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 183

Fig. 1. The Structure of a Typical Focused Crawler (TFC).

Table 1 ontologies. On the contrary, in corpus-based methods the informa-

Types of Word Sense Disambiguation.
tion is gathered from contexts of previously annotated instances
Type of ambiguity Denition Example (e.g., examples) of the word.
Homographs Words with same minute (extremely
spelling, same small, measure of time)
pronunciation but
3. Effective focused crawler
either same or
different meaning Usually, the crawling queue of the focused crawler is initially
Homonyms Words with same rose (ower, past tense fed with high quality pages (e.g., seeds), which are highly related to
spelling, same form of verb rise)
DOI. Also, the focused crawler maintains an efcient link weighting
pronunciation but
different meaning strategy. However, some of the retrieved pages may be obsolete
Heteronyms Words with same dove (bird, past tense as traditional focused crawlers suffer from Haste Problem. This
spelling but different of verb dive) happened because the focused crawler is a blind entity. Its oper-
pronunciation and ation relies mainly on predictions. A focused crawler retrieves a
different meaning
page if it predicts that it is a good page. But what is the situation if
the crawlers prediction is inaccurate. According to the traditional
focused crawling, the links extracted from any retrieved pages
2.3. Word sense disambiguation
are weighted, and then injected into the crawling queue. If such
retrieved page is irrelevant one (e.g., the page is not related to the
Human language is fairly ambiguous; hence, numerous words
domain of interest), bad links will be injected into the crawling
can be portrayed in several different ways based on the context in
queue causing the crawler to skew from its main target of retriev-
which they occur. Most of the words in natural languages are pol-
ing high quality pages that are related to a specic domain. To go
ysemous as they have multiple possible meanings or senses. Word
around such problem, as illustrated in Fig. 2, a domain distiller can
sense disambiguation (WSD) is dened as the task of identifying the
be used to guide the crawler operation and compensate its blind-
appropriate sense of an ambiguous word in a context. In English lan-
ness.
guage, there exist a variety of ambiguities types; Table 1 illustrates
Hence, before passing the retrieved page to the indexer or
the most famous ones.
adding pages links to the crawling queue, the page is passed to a
WSD typically involves two main tasks; (1) determining the
domain distiller to decide whether it is relevant to DOI. Only good
different possible senses (or meanings) of each word, and (2) tag-
pages (e.g., pages that are highly related to DOI) are passed to the
ging each word of a text with its appropriate sense. The former
indexer. The decision here is taken according to the pages contents;
task, that is, the precise denition of a sense, is still a chal-
hence, it is an accurate decision.
lenge within the Natural Language Processing (NLP) community.
Recently, the most used Sense Repository is WordNet [16]. On
the other hand, the second task (e.g., tagging of each word with 4. Related work
appropriate sense) involves the development of a system capa-
ble of tagging polysemic words in running text with sense labels. The main contribution of this paper is to enhance the perfor-
WSD community classies these systems into two main general mance of focused crawling by using domain distillers. A distiller is
categories, namely; knowledge-based and corpus-based. Although a binary classier that is initially learned with the knowledge of a
both categories build a representation of the examples to be tagged specic domain in the form of classication rules. Then, it can use
using some previously collected information, they differ in the such pre-stored rules to discover those web pages that are rele-
source of this collected information. Knowledge-based methods vant to crawlers DOI. In this section, a quick review summarizing
obtain the information from external knowledge sources such the recent work in the area of web page classication will be intro-
as Machine Readable Dictionaries (MRDs) and/or lexico-semantic duced, which can be applied in binary or multi-class classications.
184 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
Fig. 2. the Structure of Effective Focused Crawler (EFC).
SVM has been applied to web page classication in [17], in which
the original SVM classier is combined with BEV (Bagging Ensem-
ble Variation) algorithm to create a new classier called VOTEM. A
web document has been assigned to a sub-category based on voting
from all category-to-category classiers. In this work, a hierarchical
classication algorithm starts from the top of the hierarchical tree
downward recursively until it triggers a stop condition or reaches
the leaf nodes. Because of the imbalanced data that decreases the
performance of original SVM classier, VOTEM is used to provide
an improved binary classier to solve the problem brought by BEV.
In [18], a web page classication method using an SVM based on a
weighted voting schema has been proposed. The feature vectors are
extracted from both the LSA (latent semantic analysis) and WPFS
(web page feature selection) methods. LSA can extract common
semantic relations between terms and documents. Then, LSA clas-
sies semantically related web pages, and WPFS extracts four text
features from the web page content. The category of a web page
can be correctly determined by the WPFS. [19] Presented an algo-
rithm based on Cost-Sensitive Support Vector Machine (CS-SVM) to
improve the classication accuracy. During the training process of
CS-SVM, different cost factors are attached on the training errors to
generate an optimized hyperplane. Experiments have shown that
CS-SVM outperforms SVM on the standard ODP dataset.
In [20], the problem of feature selection has been highlighted;
the aim is to nd a subset of features for optimal classication.
A critical part of feature selection is to rank features according to
their importance for classication. [26] Has developed a new fea-
ture scaling method, called classdependentfeatureweighting
(CDFW) using naive Bayes (NB) classier. A new feature scaling
method, which is called; CDFWNBRFE, combines CDFW and
recursive feature elimination (RFE). In [21], an auxiliary feature
method is proposed. It determines features by an existing fea-
ture selection method, and selects an auxiliary feature, which can
reclassify the text space aimed at the chosen features. Then, the cor-
responding conditional probability is adjusted in order to improve
classication accuracy.
In [22], a hybrid k-nearest neighbor (KNN) and SVM classiers
for multiclass classication of gene expression data has been intro-
duced. This hybrid classier, which is called HKNNSVM, uses KNN
to prune training samples and uses SVM to classify samples. Com-
pared with SVM and KNN, the Misclassication rate of HKNNSVM
for datasets were lower, which indicated that the classication per-
formance of HKNNSVM was stable. [23] Proposes a discriminant
analysis method for categorization of text documents. It catego-
rizes the text by nding coordinate transformations that reect
similarity from data by using generalized singular value decompo-
sition (GSVD). However, the cost of classication is extremely high
in document analysis. In [24], an effective re-examination of text
categorization approaches of statistical test using ve categoriza-
tion method as KNN, SVM, NN, NB, and LLSF (Linear Least square
Fit) has been introduced. Among them SVM, KNN, LLSF outperform
the results than the NB, NN when the number of positive training
set per categories are small.
In [25], a hybrid algorithm based on Variable Precision Rough
Set (VPRS) is proposed, which combines the strength of KNN and
Rocchio techniques to overcome their weaknesses. Firstly, feature
space of training data is partitioned using VPRS, and lower and
upper approximations of each category are dened. Then KNN and
two Rocchio classiers are built on these new subspaces respec-
tively. The two Rocchio classiers are used to classify most of the
new documents effectively and efciently. KNN is used to nd
nearest neighbors of the new document in the subset of train-
ing dataset, which can save time obviously compared with nding
nearest neighbors in the whole training dataset. Experimental
results indicate that the proposed hybrid algorithm achieves sig-
nicant performance improvement. [26] Proposed a model for text
categorization that concentrates on the underlying meaning of
words in their context (i.e., concentrates on learning the meaning of
words, identifying and distinguishing between different contexts of
word usage). This model can be summarized in the following steps:
rst, it maps each word in a text document to explicit concepts.
Then, it learns classication rules using the newly acquired infor-
mation; nally, it interleaves the two steps using a latent variable
model. The proposed model combines Natural Language Process-
ing techniques such as word sense disambiguation, part of speech
tagging, with statistical learning techniques such as Nave Bayes in
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
order to improve the classication accuracy and to achieve robust-
ness with respect to language variations.
In [27], a new text classier is proposed by integrating the near-
est neighbor (NN) and SVM algorithms. The proposed SVM-NN
approach aims to reduce the impact of parameters in classication
accuracy. In the training stage, SVM is used to reduce the training
samples for each of the available classes to their support vectors
(SVs).The SVs from different classes are used as the training data of
nearest neighbor classication algorithm in which the nearest cen-
troid distance function is used to calculate the average distance
instead of Euclidean function, which reduce time consumption.
[28] Proposes hybrid classiers involving various two-classier and
four-classier combinations for two-level text categorization. It
shows that the classication accuracy of the hybrid combination is
better than the classication accuracies of all the corresponding sin-
gle classiers. The constituent classiers of the hybrid combination
operate on different subspaces obtained by semantic separation of
data. Experiments show that dividing a document space into dif-
ferent semantic subspaces increases the efciency of such hybrid
classier combinations.
[29] Introduced a new crawler architecture, which is called
Treasure-Crawler (TC). According to TC, a new methodology that
employed specic HTML elements of the input page was employed
to predict the target topical domain of each page that has an
unvisited link inside that input page. Then, only those on-topic
pages are sorted based on their relevancy the crawlers domain of
interest for further actual downloads. In TC, a hierarchical struc-
ture called T-Graph was employed, which assigns the appropriate
priority score to each unvisited link. Then, these URLs will be
downloaded later based on their pre-assigned priorities. In [30],
an effective focused crawler had been developed, which is called
OntoCrawler. It provide a semantic level solution that provides fast,
precise, and stable query results based on ontology-supported web-
site models. Hence, OntoCrawler can benet both user requests
and domain semantics. It has practically applied on Yahoo and
Google search engines to actively search for webpages of related
information. Experimental results had shown that OntoCrawler
could denitely promote both precision and recall rates of webpage
query.
5. Disambiguation domain ontology (D2 O)
In this section, a proposed structure for domain ontology will
be presented, which will be used for mapping domain keywords
to the corresponding domain concepts as well as achieving key-
word disambiguation. Hence, it is called Disambiguation Domain
Ontology (D2 O). With the aid of Wordnet, D2 O organizes the con-
sidered domain keywords into groups so that each group consists
of a set of synonymous keywords that are used to express a
unique domain concept. For dimensionality reduction, one key-
word, from each group, is selected to express the underlying
concept, which is called concepts Representative Term (RT). In
addition to Synonymous relation, D2 O maintains several rela-
tions among the domain keywords that indicate the semantic
strength among each domain keyword and other keywords
D2 O.
To the best of our knowledge, all web page classication tech-
niques are mainly relying on bag-of-words representation for the
input page. However, employing individual keywords as features
may lead to features ambiguity, which is called; the polysemy effect
[9]. This happened as some keywords may be shared among several
domains and accordingly have several meanings. For illustration,
suppose that page X includes the keywords; Java, Programming,
Computer, while another page Y includes the keywords; Java,
Cup, Coffee. If the keywords are used directly as the classi-
185
cation features, this will certainly mislead the classier as it may
give the same label to both pages (e.g., classify them to the same
class). However, if Java is considered as an ambiguous keyword,
WSD can be used to disambiguate it into its correct sense, and then
classify the page accordingly.
As the nearby keywords can provide strong consistent clues
to the sense of the ambiguous keyword, D2 O maintains a list of
discriminative keywords for each domain ambiguous keyword,
which is called Partners List (PL). Then, rely on those partners
to sense the correct meaning of the polysemous (ambiguous)
keyword. However, selecting those partners is a true challenge,
which will be claried through the next sections. Hence, when
classifying a new page P, which includes an ambiguous key-
word Kamb , initially PL(Kamb ) is identied with the aid of D2 O,
then, the correct meaning of Kamb is sensed accordingly based on
the absence/existence of keywords PL(Kamb ) in the tested page
(e.g., P). Afterword, a decision can be taken whether to include
Kamb as a feature during the classication process or not. The
next subsections will illustrate in details; D2 O construction, the
methodology used to identify the ambiguous keywords, and the
procedure followed to elect the partner list for each ambiguous
keyword.
5.1. D2 O construction
Algorithm 1 shows the procedure for D2 O construction that con-
sists of four steps, which are; (i) Conceptualization, (ii) Concept
Weight Calculation, (iii) Inter-Concepts Relationships Assignment,
and (iv) Graph Construction.
In the rst step of D2 O construction (e.g., Conceptualization),
the domains keywords are collected by a domain expert from
highly domain related web pages, which can be represented by
the set K = {k1 , k2 , .,kg }. Then, synonymous are grouped into one
cluster, so that each cluster represents distinct domain concept.
Then, the most popular keyword in each cluster is selected to be
the concepts Representative Term RT, while the remaining cluster
members are the RTs synonyms. Finally, after conceptualization,
DOI is expressed by a group of concepts, which are represented by
the set C = {c1 , c2 , c3 , . . .., cn }.
In the second step (e.g., Concept weight calculation), the weight
of each domain concept is calculated, denoted as; w(ci ) i {1,2,
. . .,n}. This weight is calculated using the web pages collected
from the considered domain corpus using the Odd Ratio Numerator
(OddN) [31] method as calculated (1).
w (ci ) = OddN (ci ) = tpr (c) [1 fpr (ci )] (1)
where, tpr (ci ) = tp (ci ) /pos and fpr (ci ) = fp (ci ) /neg
Where tpri is the sample true positive rate of concept ci , which is
calculated as; tpr (ci ) = tp (ci )/pos, and fpri is the sample false pos-
itive rate of concept ci , which is calculated as; fpr(ci ) = fp(ci )/neg,
tp(ci ) expresses the true positive of the domain given concept ci ,
which is the number of positive pages for the domain(e.g., pages
that were already related to DOI) containing the concept ci , fp(ci )
expresses the false positive of the domain given concept ci , which is
the number of negative pages for the domain(e.g., pages that were
not related to the domain) containing the concept ci , pos is the
number of positive pages of the domain, and neg is the number of
negative pages of the domain.
186 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
In the third step (e.g., Inter-Concepts Relationships Assignment),
the relation between every pair of concepts, denoted as; r(cx ,cy )
cx , cy C is calculated using the WebOverlap coefcient [32] as
illustrated in (2).
     
r ci , cj = WebOverlap ci , cj = N ci cj /min (N (ci ) , N (ci )) (2)
Where; N(c) is the number of retrieved pages from Google that con-
tain the concept c and N (cx cy ) is the number of retrieved pages
from Google that contain both concepts cx and cy . In the fourth
step (e.g., Graph Construction), the domain concepts are arranged
in a graph like structure called the disambiguation domain ontology
(D2 O), considering the inter-concept relationships between every
pair of concepts and the concepts weights.
5.2. D2 O generation: illustrative example
In this section, an illustrative example showing how to construct
a simple D2 O will be introduced. The task is to generate a new ontol-
ogy that represents a specic domain by following the proposed
procedure illustrated in algorithm 1.
A Conceptualization and Calculating the Concepts Weights
As depicted in Fig. 3, domain concepts are identied. Then,
the Odd Ratio Numerator (OddN) method is used to calculate the
weight of each concept w(ci ) using (1). The used positive pages
(e.g., pos) = 10, also the negative pages (e.g., neg) = 10. Those nega-
tive pages are the ones that contain several domain concepts but
are not related to the domain.
B Inter-Concepts Relationships Assignment
In this step, the relationship r(cx ,cy ) between each con-
cept pair will be calculated by using (2). For illustration,
consider the concept pair (A,B). As the query of A + B
appears in 642,000,000 pages in Google, and for the query
of A and for B are 25,270,000,000 and 10320,000,000
pages respectively, so r(A,B) = 642,000,000/min(25,270,000,000,
10320,000,000) = 642,000,000/10320,000,000 = 0.06. And continue
the calculations for the rest of the domain concepts as shown in
Fig. 3.
C Graph Construction
Graph construction is the nal step at which all domain concepts
are arranged together in the form of a weighted conceptual graph
that representing the domain of interest as illustrated in the last
step of Fig. 3.
5.3. Identifying ambiguous keywords (Confusion set)
In order to solve the polysemy dilemma, after setting up D2 O,
it is essential to identify the set of keywords that need to be dis-
ambiguated. This set of keywords, which is called Confusion Set
(CS), may cause serious problems as it confuses the classication
algorithm. As manual identication of CS is a tedious and subjective
issue, through this section, an automatic methodology for identi-
fying CS will be illustrated in details.
WordNet, as a valuable lexical resource, has attempted to model
the knowledge of a native speaker of English. However, identify-
ing the ambiguous set by directly employing WordNet has resulted
in too many senses for almost all considered domain keywords.
Generally, WordNet maintains a semantic relation among key-
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 187

1
Calculate
Filter pages Inter-concepts
contents 4 3 relationships
Page ID Page Contents
p1 A,B,E
p2 A,B,C Graph Construction Concepts # of Retrieved
Concepts
p3 A,B,D Pair
p4 A,D,E Pair Web Pages
Relation
p5 A,E,F
p6 B,D,E A,B 642,000,000 0.06
p7 B,C A,C 313,000,000 0.02
p8 C,D,E r (AC) A,D 249,000,000 0.03
p9 B,E A,E 116,000,000 0.01
r (A B)
p10 B,E,F A A,F 86,200,000 0.01
p11 C,D B,A 234,000,000 0.02
p12 D,F B,C 498,000,000 0.05
r(A E) r(BC) B,D 59,000,000 0.01
p13 A,F B
r(AF) B,E 72,200,000 0.01
p14 A,C,E r (AC)
p15 E,F B,F 37,500,000 0.01
p16 A,C,D r(B F) C,A 218,000,000 0.01
p17 A,E C,B 55,000,000 0.01
r(B D)
C C,D 496,000,000 0.03
p18 D,E,F F
p19 B,C,D C,E 101,000,000 0.01
P20 A,F r(D F) r(B E) C,F 61,500,000 0.01
D,A 250,000,000 0.02
r(C D)
r(E F) D,B 59,200,000 0.01
D D,C 77,800,000 0.01
D,E 468,000,000 0.03
D,F 49,600,000 0.01
r l(CE)
2 E E,A 535,000,000 0.03
r (D E)
E,B 50,100,000 0.01
E,C 84,900,000 0.01
E,D 92,100,000 0.01
Calculate r(C F) E,F 399,000,000 0.05
Concept Weight F,A 157,000,000 0.02
F,B 33,300,000 0.01
Concept Weight F,C 51,200,000 0.01
A 0.3 F,D 37,600,000 0.01
B 0.63 F,E 48,500,000 0.01
C 0.18
D 0.2
E 0.42
F 0.1

Fig. 3. D2 O Generation in the Illustrative Example.

word senses by grouping keywords into the same semantic domain
(Education, Sports, Medicine, etc.). However, too many senses may
be considered for the same keyword, which may be harmful and
time consuming for the classication task. For illustration, the key-
word bank, has ten senses in WordNet 3.1, in which three of
them, namely; bank#1, bank#3 and bank#6, are grouped
into the same domain label, which is Economy. On the other
hand, bank#2 and bank#7 are grouped into the labels Geog-
raphy and Geology respectively. Moreover, WordNet considers
also the rare senses of the word, for illustration, the word java
has three senses in WordNet 3.1, hence, java is shared between
three domains, which are; Food with the sense coffee, Commu-
nication with the sense object oriented programming language,
and Location with the sense an island in Indonesia. However,
the second sense is the most widely used one, hence, as WSD is
expensive, it is unnecessary to disambiguate every domain key-
word appeared in the tested page. Too much WSD may increase the
classiers risk of overtting as WSD is not always accurate. From
another point of view, WordNet has a subtle distinction between
keyword senses, which could be harmful for the classication task.
For illustration, WordNet distinguishes between bass (the lowest
part in polyphonic music) and bass (the lowest part of the musi-
cal range). Hence, relying completely on WordNet for identifying
CS will not be a good decision. In this section, a simple but effec-
tive methodology will be introduced for identifying the subset of
keywords that most likely confuse the classier, which is called
Fuzzy Based Ambiguity Identication (FBAI) Strategy. As illustrated
in Fig. 4, FBAI consists of two sequential phases, which are; (i)
Pre-Processing (PP) and (ii) Fuzzy Inference Engine (FIE).
5.3.1. Pre-Processing (PP)
During PP, the task is to calculate three different parameters for
the input keyword k D2 O, which are; (i) Domain Relatedness of
k, denoted as; DR(k), (ii) Keyword Popularity, denoted as; KP(k),
and (iii) domain Information Content of k, denoted as; IC(k). For
calculating those parameters for k, initially, Google is queried to
retrieve the  pages that contains k, which are expressed by the
set Pages(k). Then, for each retrieved page P Pages(k), the set of
snippets that contains k are identied and combined together to be
a representative for the page P. A snippet is dened as the set of
keywords around k in the page P, hence, the center of the snippet is
the considered keyword k. The snippet has two wings with length
, hence, each wing is represented be words in one side of k. The
collected snippets of k of page P are combined together and used to
classify P using traditional NB classier either to the domain (class
D+ ) or not (class D ).
This process is continued until all pages Pages(k) have been
classied. Then, the number of pages D+ as well as the number
of pages D are calculated, denoted as; P+ (k) and P (k) respec-
tively. Moreover, the frequency of k is calculated in all snippets
collected from all pages D+ , denoted as; TF+ (k), and also in all snip-
pets collected from all pages D , denoted as; TF (k). Also consider
the number of occurrences of k in the considered domain corpus,
denoted as; N(k), as well as the total number of occurrences of other
188 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
Fig. 4. Identifying Ambiguous Keywords.
domain keywords, denoted as; NT , Finally, DR(k), KP(k), and IC(k)
are calculated using (1), (2), and (3) respectively.
TF + (k) TF
DR(k) = (k) (1)
P+
KP(k) = (k)P (k) (2)
 
IC(k) = log P(k) = log N(k)NT (3)
Denition 1. Keyword Domain Relevancy, denoted as; DR(k) mea-
sures the degree of relevancy of a keyword k to the Domain of interest
and is dened as the percentage of occurrences of k in the positive pages
(e.g., pages related to the domain) to occurrences of k in the negative
ones (e.g., pages unrelated to the domain).
Denition 2. Keyword Domain Popularity, denoted as; KP(k) mea-
sures the popularity of a keyword k in the considered domain against
other domains, and is dened as the percentage of positive pages that
contains k to the negative ones.
Denition 3. Keyword Domain Information Contents, denoted as;
IC(k) measures ks ability to convey domain information. It mea-
sures the amount of domain knowledge gained by the keyword
occurrence.
As illustrated in denition 3, a highly recurring keyword con-
veys little information due to its ubiquitous use, and then it has a
small IC. On the other hand, a rare keyword conveys much more
domain information as it tends to be more specialized and con-
veys more independent meaning, and accordingly it owns a high IC
value. Finally, after calculating DR(k), KP(k), and IC(k) k D2 O, the
Confusion Set (CS) can be identied through a fuzzy inference pro-
cess. Generally, the higher the domain relevancy, popularity, and
information content of a keyword, the lower its ambiguity level.
5.3.2. Fuzzy inference engine (FIE)
Fuzzy inference is suitable for approximate reasoning as it can be
used efciently in decision making under incomplete or uncertain
data. Hence, fuzzy inference can be successfully employed to assign
an ammbiguation value for a specic keyword. Generally, fuzzy
inference can be applied through three sequential steps, which
are; (i) fuzzication of inputs, (ii) Fuzzy Rule Induction, and (iii)
defuzzication.
a Fuzzication of Inputs
Three different fuzzy sets, which are; DR, KP, and IC will be con-
sidered. During fuzzication, the input crisp values are mapped
into grades of membership for linguistic terms, Low and High
of the used fuzzy sets. The employed membership functions for the
considered fuzzy sets (e.g., DR, KP, and IC) are illustrated in Fig. 4.
b Fuzzy Rule Induction
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 189

Table 2 5.4. Identifying partner list (PL) for each ambiguous keyword
The used fuzzy Rules.

ID DR
1 L
2 L
3 L
4 L
5 H
6 H
7 H
8 H
KP IC
L L
L H
H L
H H
L L
L H
H L
H H
Rule output After domain CS has been identied, the task is to choose PL(k)
L k CS. A partner of a keyword k is a nearby domain keyword that
L provides a strong consistent clue to the sense of k with respect to
L the domain of interest (DOI). Hence, a good way is to choose the
H closest keywords to the ambiguous keyword k in D2 O to represent
L
the partners of k (e.g., PL(k)). To accomplish such aim, as depicted in
H
H Fig. 7, a circle of center k is drawn over D2 O with radius , which is
H called; Partner Circle (PC). Hence, represents the number of hops
that separates k from the farthest partners in PL(k). However, not all
the keywords inside PC are considered as partners of k. Only those
keywords that are related to k with an acceptable strength are con-
sidered as partners. Hence, we rely on the relation strength among
k and those keywords inside PC(k) that were previously calculated
during D2 O construction. A threshold strength value is assumed as
, then any a keyword M is considered as a partner of k if r(k,M) .
Also, as illustrated in Fig. 7, if a keyword is rejected from the partner
list of a keyword, its directly connected keywords are also rejected.

6. The proposed domain distiller

As shown in Fig. 8, the input for the proposed distiller is a

Fig. 5. The used identication function.
For the inference process, a set of fuzzy rules are employed in
the form if (A is X) AND (B is Y) AND (C is Z) . . .. . . THEN (D is M),
where A, B, and C represent the input variables (e.g., DR, KP, and IC),
while X, Y, and Z represent the corresponding linguistic terms (e.g.,
low or high), D represents the rule output, and nally M represents a
linguistic terms (low or high). Hence, the output of the fuzzication
is the input for the fuzzy rule induction. There are 8 rules, which are
listed in Table 2 (assuming L refers to Low, H refers to High).
For illustration, the rst rule in Table 2 indicates that; IF DR(k) is
Low AND KP(k) is Low AND IC(k) is Low THEN Output is Low.
Generally, there are four fuzzy rules inference methods, namely;
max-min, max-product, sum-dot, and drastic product. The max-
min is the used method in this paper. It is based on choosing the min
operator for the conjunction in the premise of the rule and for the
implication task, while the max operator is used for aggregation.
Hence, for w input variables and q states of the output linguistic
terms, the max-min inference rule can be illustrated in (4).
aggregation
   before, if the considered Domain Of Interest (DOI) has k keywords,
out (x) = MAX MIN x {1, 2, 3, ...., q}
  inp(1) , inp(w)
implication
c Defuzzication
Defuzzication was accomplished using the output member-
ship function illustrated in Fig. 4. Hence, consider a keyword k
whose input parameters are; DR(k); KP(k); and IC(k). The output
value of the defuzzication process is a crisp value that expresses
the Ambiguation Value (AV) of the keyword k; e.g.; AV(k). Finally;
the decision is taken whether k is an ambiguous or not based on
a simple rule; which is expressed by the simple step identication
function illustrated in Fig. 5. On the other hand; Fig. 6; illustrates a
simple example considering the keyword k; with the corresponding
parameters DR(k); KP(k); and IC(k) are; 3; 3.5; 7 and respectively.
web page to be classied. The distiller then analyzes the page
and extracts existing domain keywords using the Disambiguation
Domain Ontology (D2 O), which maintains all available domain key-
words. However, before mapping extracted domain keywords to
the corresponding concepts, a disambiguation process is done on
the extracted domain keywords to discard those ambiguous key-
words which are not related to the domain.
As the target of the proposed distiller is to decide whether the
input page is related to the domain of interest or not, the core of our
distiller is a binary classier. However, to perfectly accomplish the
distillation process, as illustrated in Fig. 8, the proposed distiller is
divided into two sequential modules, which are; analysis module
(AM) and classication module (CM). During AM, the input page is
expressed in a vector space model. On the other hand, CM takes the
decision whether to accept or reject the page using an optimized
Nave Bayes classier. These modules will be explained through the
next subsections.
6.1. Analysis module (AM)
The target of AM is to represent the input page in the vector
space model. To accomplish such aim, domain keywords found
in the page will be extracted and mapped to the corresponding
domain concepts using the aid of the proposed D2 O. As illustrated
(4)
these keywords are clustered using WordNet. For each cluster syn-
onymous keywords are grouped and represented by one domain
concept (e.g., the underlying concept), which is the frequently used
term of the clusters terms, hence, it is called the concepts Rep-
resentative Term (RT) and the remaining clusters terms are RTs
synonyms.
Considering an input page P, after extracting the pages domain
keywords, expressed by the set Keywords(P), ambiguous keywords
are identied with the aid of the proposed D2 O. Disambiguation is
done by discovering the correct sense of the word by looking for the
partners of the ambiguous keyword inside the processed page (e.g.,
P). Considering an ambiguous keyword kamb , The existence of the
kamb s partners gives a good conclusion that kamb gives the correct
domain sense. Hence, to disambiguate kamb , initially, Partners(kamb )
are identied, then the set RES = Keywords(P) Partners(kamb ) is
constructed. Finally, the strength of the relations among kamb and
all keywords RES are indicated from D2 O, then the average of
190 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204

Fig. 6. Illustrative example showing how to calculate the importance value of a cached item using a fuzzy inference system.

Fig. 7. Identifying the partner list of an ambiguous keyword assuming =2.

strengths for the identied relations is calculated. Then, the deci-
sion whether the sense of kamb is in DOI label, and then kamb is
considered as a domain keyword is taken based on Rule 1.
Rule 1: Disambiguation Rule
An ambiguous keyword is considered as a domain keyword if the
average of the strengths of the relations of the partners of kamb
that exists in the processed page is greater that a threshold value,
denoted as; . This can be expressed by the following expression.
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
Fig. 8. The Proposed Distiller Framework.
If [Average(Relations Strength(kamb k))k RES]
ThenSense(kamb ) DOIElseSense(K amb ) /inDOI
After removing those keywords that are not related to DOI,
the remaining domain keywords are mapped into the correspond-
ing domain concepts using D2 O. Afterword, the input page can be
represented in the form of a vector of domain concepts. The dimen-
sion of that vector equals the number of the considered domain
concepts. Hence, if the concept is found in the input page, the cor-
responding place of that concept in the vector is set to 1, otherwise,
it is set to 0. At the end of this stage, an input page can be expressed
as a vector of ones and zeros, which is called the Vector Space Model.
As our domain has a set of n concepts C = {c1 , c2 , c3 . . ., cn }, so the
vector of the input page will be of n-dimensions.
6.2. Classication module (CM)
This module is a binary classier, in which pages will be clas-
sied into only two classes. The processed web page may relate
to our DOI or it may not. Related pages are classied to Class
1, while the others are classied to Class 2. In CM, a new of
binary classier is proposed by integrating an optimized Support
Vector Machine (SVM), by the use of Genetic Algorithm (GA), and
Nave Bayes (NB) algorithms. Although the optimization was done
on SVM, such process also optimizing the behavior of NB. Hence,
the new binary classier, which is the core of the proposed dis-
tiller, is called Optimized NB (ONB) classier. Initially, ONB rejects
outliers by selecting the most informative examples using an opti-
mized SVM (with the aid of GA). Hence, after rejecting outliers, the
remaining pages (examples) will be used to train the NB classier.
Finally, the decision if an input page is related to Class1 or Class
2 will be taken using NB classier for testing. ONB operates in three
phases; which are; (i) Outlier Rejection (ii) Training, and (iii) Test-
ing. More details about those three steps will be introduced in the
next sub-sections, also they are depicted in algorithm 2.
6.2.1. Outlier rejection
This step aims to discard the false examples that may result in
constructing wrong classication rules during the next step, which
is the classier training. Surely, constructing wrong rules will badly
impact the performance of the distiller, which in turn directs the
191
focused crawler in the wrong direction when embedding such dis-
tiller in an EFC. Rejecting outliers can be accomplished in two steps,
which are; (A) SVM Optimization, and (B) Selection of informative
pages.
(A) SVM Optimization
Genetic algorithm (GA) is one of most effective, powerful, and
unbiased heuristic search approach in the area of Articial Intelli-
gence (AI). It guarantees an optimized solution for a given problem
based on several approaches such as; mutation, inheritance, and
selection. GA has several advantages, which makes it as one of
the most common search algorithms in AI, such as; (i) GA has the
ability to solve any optimization problem based on chromosome
approach, (ii) It can handle multiple solution search spaces as well
as perfectly solving the given problem in such an environment (iii)
GA is less complex and more straightforward compared to classical
algorithms, (iv) it is easier to be transferred and applied in differ-
ent platforms, thereby increasing systems exibility, (v) GA has the
ability to support multi-objective optimization, and accordingly it
is a good choice for noisy environments. Moreover, GA has the
ability to nd a global optimum.
On the other hand, GA differs from traditional search and opti-
mization methods in several signicant points, which make it
a perfect choice to be implemented in our work such as; (i) it
searches in a parallel manner through the population. Accordingly,
GA avoids being trapped in local optimal solution such as traditional
techniques, which search from a single point. (ii) GA implements
probabilistic selection rules, not deterministic ones. (iii) GAs work
on the Chromosome, which is an encoded clone of potential solu-
tions considered parameters, rather the implemented parameters
themselves. (iv) GAs employ a tness score, which is derived from
objective functions, with no derivative or auxiliary information. (v)
GA requires no explicit expression for the solving model; rather it
needs the tness function and related variables. (vi) GA can effec-
tively handle arbitrary types of constraints and objectives. (vii) GAs
complexity is almost linearly correlated with the scale of the con-
sidered problem, hence, no chance to have dimension disaster.
In this step (e.g., SVM Optimization), genetic algorithm (GA) is
used to improve the performance of the traditional SVM. GA will
be used to optimize a specic parameter of SVM, which is the soft
margin parameter C [33]. Parameter C is used during the training
of SVM and indicates how much outliers are taken into account in
192 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
Fig. 9. The region around SVM Hyperplane.
calculating support vectors. To implement our proposed approach,
this paper used the linear kernel function for the SVM classier as
the linear kernel function is the most suitable one for text.
The optimization procedure can be divided into ve processes,
which are; (i) Population, (ii) Evaluation, (iii) Encoding, (iv) Selec-
tion, and (v) Crossover. In population process, different values for
the parameter C are assumed as chromosomes, which are the pos-
sible solutions of the problem. Then, in evaluation process, for each
chromosome, that representing parameter C, there is a tness func-
tion used to evaluate its tness (Ft) by using the training dataset to
train the SVM classier, then use a testing dataset to calculate the
accuracy as depicted in (5).
CorrectAssignments Ai + Ci
Fti = Acci = = (5)
TotalPages Ai + Bi + Ci + Di
Where, Fti is the tness value of the ith chromosome, Acci is the
accuracy of SVM classier when using the ith chromosome, Ai is
the number of pages that are assigned correctly when using the ith
chromosome, Bi is the number of pages that are assigned incorrectly
when using the ith chromosome, Ci is the number of documents that
are rejected correctly when using the ith chromosome, and Di is
the number of documents that are rejected incorrectly when using
the ith chromosome. Then, in encoding process, the solutions can
be encoded as a binary numbers. On the other hand, during the
selection process, the better solutions are selected using Roulette
Wheel Selection (RWS) technique [34] as it is the most simple selec-
tion technique in implementation. Hence, the probability of each
chromosome will be selected is calculated as in (6)
x
Pi = Fti / Fti
i=1
Where Pi is the selection probability of chromosome i, x is the
number of all chromosomes in the population, and Fti is the t-
ness value of chromosome i. then the sum of all probabilities of all
chromosomes is calculated as depicted in (7)
x
SumPi = Pi
i=1
Where, SumP is the sum of all probabilities of all chromosomes, x
is the number of all chromosomes in the population, and Pi is the
selection probability of the ith chromosome. Then a random number
(Rn) from interval (0, SumP) is generated as expressed in (8).
Rn = Rand(0, SumP) (8)
Where, Rn is a random number between 0 and SumP, and SumP is
the sum of all probabilities of all chromosomes in the population.
Then, return again to the considered population to accumulatively
sum the probability values from 0 to SumP. While summing, if SumP
reaches a value that is greater than or equal the random number Rn
(e.g., SumP > = Rn), the process should stop and select this chromo-
some. The crossover process is used to interchange genes between
chromosomes to create offsprings using Single Point (SP) technique
[35]. In this technique, rstly, again a random number Rn[i] from
0 to SumP should be generated. Then, if Rn[i] is smaller than the
crossover probability (Pc), then the chromosome i will be chosen
as a parent. The crossover point within a chromosome is chosen
randomly, and then the two parent chromosomes are interchanged
at that point to produce two new offspring.
(B) Selection of informative pages
SVM has several salient properties that other techniques do not
have such as; (i) it guarantees accurate classication, even if the
input data is non-linearly separable, (ii) the classication accuracy
does not depend on the expertise of choosing of the kernel func-
tion (in the case of non-linear input data), (iii) as SVM has only two
free parameters, which are; kernel function and the upper bound,
it can be easily controlled, (iv) SVM insures the existence of global,
(6) unique, and optimal solution since its training is equivalent to solv-
ing a linearly constrained Quadratic Programming, (v) SVM has the
ability to adapt its learning characteristic using the kernel function,
also it owns a good ability to adequately classify data even in a high-
dimensional feature space with little training data, (xi) SVM works
well on real-world applications, moreover, it can be successfully
applied to any kind of data since a kernel is available.
(7) In this step, the optimized SVM is used to select the most
informative examples from the available ones. When the distribu-
tion of the available training examples between the two classes is
inspected, as illustrated in Fig. 9, it will be clear those examples that
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 193

Table 3
The rst generation.

Population, Evaluation and Encoding Selection Crossover

ID Chromosome Binary Fitness Probability Cumulative SumRandom NumberNew ID New Random NumberNew
Encoding Value (Ft) (P) probability (Rn) Population (Rn) Population
(SumP)

1 0.5 00.1 0.66 0.18 0.18 0.84 5 2.5 0.28 0.5

2 1 01.0 0.71 0.19 0.37 0.02 1 0.5 0.55 0.5
3 1.5 01.1 0.69 0.18 0.55 0.75 5 2.5 0.96 2.5
4 2 10.0 0.7 0.19 0.74 0.42 3 1.5 0.97 1.5
5 2.5 10.1 0.5 0.13 0.87 0.35 2 1 0.16 3.5
6 3 11.0 0.5 0.13 1.0 0.88 6 3 0.95 3
Total 3.76 1.0

are far from the hyperplane are highly related to their classes. This cl2 is the opposite class. During NB training, the task is to calcu-
is because the hyperplane separate between the two classes, hence late the conditional probabilities P(ci |clj )ci D C, clj D CL, and the
those examples that are closed to the hyperplane may be related classes prior probabilities P(clj ) clj D CL as illustrated in (9) and
to both classes or their assignments may have errors; hence some (10) respectively.
of them may be assigned incorrectly. Accordingly, they are not dis-
P(clj ) = Pgj /Pg (9)
criminative examples and may lead to obsolete classication rules
during the training phase. A good behavior is to eliminate those P(ci |clj ) = Ni,j /Nj (10)
ambiguity examples. However, to keep as much training exam-
Where; Pgj is the number of pages related to class clj , and Pg is
ples as possible, such elimination should be controlled. A simple
the total number of pages related to all domain classes, Nij is the
approach, which is the followed one, is to eliminate the support
number of occurrences of concept ci in pages of class clj , and Nj is
vectors as they are the most ambiguity examples as illustrated in
the total number of concepts in pages of class clj .
Fig. 9.

6.2.3. Testing phase

6.2.2. Training phase
During the testing phase, the decision whether an input page is
NB has proven to be the most effective classier in the area of
related to the DOI or not will be taken. If its related, the page will be
text classication. It has several advantages such as; (i) NB has
targeted to cl1 . Otherwise, it will be targeted to cl2 , hence, it will
a short computational time for training with the assumption of
be rejected. During the testing phase, initially, the probability that
independent features, (ii) It is easy to be implemented and often
pi related to each class is calculated (also called the likelihood of being
has superior performance, (iii) applying NB requires low resource
in the class) based on the domain concepts extracted from pi . If pi
requirements in terms of time and memory during the testing
contains z domain concepts, the likelihood of being in class clj clj
phase, hence, it is suitable for use by focused crawlers that need
CL can be calculated as expressed in (11). Then, pi will be targeted
to take the classication decision on time, and (iv) NB is robust in
to the class (cltarget ) with the maximum calculated likelihood, using
noisy environments, moreover, it requires little amount of training
(12).
data.
During the training phase, traditional NB classier is trained
z
Likelihood(pi |clj ) = P(clj ) P(cx,i clj ) (11)
using the most informative examples that were selected during the x=1
previous phase. Hence, the simple binary NB classier is employed, z
which has two target classes represented by the set CL = {cl1 ,cl2 }, Target(pi ) = cltarget = arg max(P(clj ) P(cx,i clj )) (12)
x=1
where cl1 is the class including those pages related to DOI, while
194 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204

In this step, different values for the parameter C are assumed as

6.3. SVM optimization: illustrative example
chromosomes as shown in Table 3, column 2.
The optimization procedure can be divided into ve steps: (i)
Population, (ii) Evaluation, (iii) Encoding, (iv) Selection, and (v) (ii) Evaluation
Crossover. Here are some assumptions:
In this step, for each chromosome, the tness value (Ft) is
calculated using tness function. The tness value is calculated
No. of chromosomes (size of population) = 6
by training dataset with the SVM classier and using the testing
No. of generation = 50
dataset to calculate the accuracy as tness function as in (5) as
Crossover probability (Pc) = 0.8
illustrated in Table 3 column 4.

(i) Population (iii) Encoding

 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 195

Table 4
The second generation.

Population, Evaluation and Encoding Selection Crossover

ID ChromosomeBinary Fitness Probability Cumulative Random New New Random New

Encoding Value (P) Sum Number (Rn) ID population Number (Rn) Population
(Ft) probability
(SumP)

1 0.5 00.1 0.66 0.18 0.18 0.79 5 3.5 0.36 3.5

2 0.5 00.1 0.66 0.18 0.36 0.96 6 3 0.44 2
3 2.5 10.1 0.5 0.13 0.49 0.66 4 1.5 0.96 1.5
4 1.5 01.1 0.69 0.19 0.68 0.04 1 0.5 0.65 2.5
5 3.5 11.1 0.71 0.19 0.87 0.85 5 3.5 0.85 3.5
6 3 11.0 0.5 0.13 1 0.33 2 0.5 0.97 0.5
Total 3.72 1.0

Table 5
The used tunable parameters through the experiments.

Parameter Assigned value Description

n 485 Number of domain concepts (Popular terms only).

t 1305 Number of domain keywords (popular and un-popular terms).
CL 2 Number of classes: Cl1 is related to DOI, Cl2 otherwise.
Training web pages 10000 From Web Data Commons dataset.
Testing web pages 500
Kernel Function Linear K(xi ,xj ) = xi .xj xi xj
C From 0.55 Soft margin parameter
x 100 Number of chromosomes of the population
Pc 0.8 Crossover Probability
No. of generations 50 Number of generations or iterations.
2 Number of hops that separates keyword k from the farthest partners in its Partner List PL(k).
0.2 Threshold relation Strength between the keyword K and its partners.
100 Number of retrieved pages that contain keyword K from Google.
50 Length of wing of snippet in page P that contains keyword K.

Fig. 10. Data Structure used to.

In this step, the chromosomes can be encoded in a binary rep- lation and compute the cumulative sum of the probabilities, and if
resentation as illustrated in Table 3 column 3. SumP is greater than the random number (e.g., SumP > = Rn), then
stop and select this chromosome, do this 6 times, so then the new
(iv) Selection population will be as in Table 3 column 9.

In this step, the Roulette Wheel Selection (RWS) technique is
used. First, the probability (P) of each chromosome will be selected
is calculated using (6) as in Table 3 column 5, then the sum of all
probabilities (SumP) of all chromosomes is calculated using (7),
then a random numbers (Rn) from interval (0, SumP) is generated
using (8) as in Table 3 column 8. Then go again through the popu-
(v) Crossover
As illustrated in algorithm 2, rst generate a random number
from 0 to 1 as shown in Table 3 column 10, which is Rn[i] where i = 1,
2, . . ., 6. And then if this random number Rn[i] is smaller than the
crossover probability Pc then select the chromosome i as a parent
196 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204

Fig. 11. Precision in all used techniques.

Fig. 12. Accuracy of all used techniques.

chromosome. As the crossover probability Pc = 0.8, then the parent

chromosomes will be, chromosomes 1, 2, and 5, so we can take
chromosome 1with 2, chromosome 2 with 5, and chromosome 5 Chromosome5 = chromosome5 >< chromosome1
with 1. Then the crossover point is chosen randomly at bit number = 01.1 > < 10.1 = 11.1 = 3.5
2 for the three crossovers as the following:

Chromosome1 = chromosome1 > < chromosome2 Finally, the chromosome population after crossover process is
= 10.1 >< 00.1 = 00.1 = 0.5 illustrated in Table 3 column 11, which will be the population for
the next generation of the genetic algorithm. In the next genera-
tion, we follow the same steps as in the rst generation and the
results are illustrated in Table 4. The population after crossover
Chromosome2 = chromosome2 >< chromosome5 process is illustrated in Table 4 column 11 which will be the next
= 00.1 >< 01.1 = 00.1 = 0.5 population, the process continues until the assumed number of
generation ends. Finally, the tness value of all chromosomes will
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 197

Fig. 13. Error of all used techniques.

Table 6
All possible outcomes of a binary classier.

Classier outcome Description

A # of documents that are assigned correctly

B # of documents that are assigned incorrectly
C # of documents that are rejected correctly
D # of documents that are rejected incorrectly

Table 7
Confusion Matrix.

Actual Class Predicted Class

Fig. 14. Crawling behavior for ideal and traditional focused crawlers.
Positive Negative

Positive A D
be calculated, and then the highest one will be selected as the best
Negative B C
solution.

7. Performance analysis and implementation performance metrics that will be used are illustrated in (13)(15)
[38].
In this section, the proposed distiller, e.g., ONB, which is the
core of CM, will be evaluated against traditional classication tech- PagesAssignedCorrectly A
Precision = P = = (13)
niques, which are SVM, NB, and KNN as well as some recent TotalAssignedPages A+B
classication techniques, namely; Domain Oriented Nave Bayes CorrectAssignments A+C
(DONB) [36] and Domain Oriented KNN (DOKNN) [36]. For each Accuracy = Acc = = (14)
TotalPages A+B+C +D
one, the accuracy, precision, and error will be reported.
The Web Data Commons dataset series contains all structured IncorrectAssignments B+D
Error = E = = (15)
data extracted from the various Common Crawl corpora [37]. Cur- TotalPages A+B+C +D
rently, the extraction process considers the data formats Microdata,
RDFa, and microformats. The documents of Web Data Commons 7.2. Implementing D2 O
were not pre-designated as training or testing patterns. Hence, we
choose some of them as training and testing subsets. 10000 web To speed up the searching, mapping, and conceptualization pro-
sites are randomly selected for training and 500 are used for test- cesses, D2 O is implemented using a database procedure in the form
ing. The parameters that will be used though the next experiments of three related tables as illustrated in Fig. 10. The rst table stores
with the corresponding values are illustrated in Table 5. the used representative terms for the considered domain concepts,
while the second table stores the synonyms terms corresponding
7.1. Performance metrics to each of the representative terms. On the other hand, the last
table stores the partners of each ambiguous keyword as well as the
Table 6 depicts the possible outcomes of a binary classier. They relations among the keyword and the other domain keywords if
will be used to measure the performance of the proposed classier. exist. This table is used mainly for keyword disambiguation. The
And, Table 7 shows the confusion matrix. On the other hand, the Partners eld in the Relation-Partner table is considered as a ag
198 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204

Fig. 15. Harvest rate of TFC and EFC against retrieved pages using STS as a link weighting strategy.

Fig. 16. Harvest rate of TFC and EFC against retrieved pages using Bay as a link weighting strategy.

that is set to 1 if the corresponding related keyword is a partner;
otherwise its value is set to 0.
7.3. Evaluating the proposed distiller
The performance of the proposed distiller (e.g., ONB) is affected
by; (i) the number of training pages (TPs), (ii) the kernel function,
and (iii) the value of soft margin parameter C used in SVM train-
ing. According to [39], the most appropriate kernel function for
binary text classication is the linear kernel, especially for large
number of concepts. Based on the Linear Kernel function, through
this section, the pre-mentioned performance metrics (e.g., P, R, Acc,
and E), will be measured for ONB as well as its competitors using
different values of training pages.
Considering Figs. 1113, basically, by increasing the number of
TPs, the performance of all classication techniques is promoted.
The reason is that as the number of TPs increases, the classiers
are better trained as they collect more domain knowledge. Hence,
it is obvious that performance promotion can be accomplished by
training the classier with more TPs. The best precision, accuracy,
and error are obtained at the maximum number of training pages
(e.g., when TPs = 10000).
As generally depicted in Figs. 1113, ONB outperforms all other
classication techniques. When TPs = 10000, ONBs Precision, Accu-
racy, and Error are; 0.86, 0.89, and 0.11 respectively. On the other
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 199

Fig. 17. Harvest rate of TFC and EFC against retrieved pages using STD as a link weighting strategy.

Fig. 18. Harvest rate of TFC and EFC against retrieved pages using TC as a link weighting strategy.

hand, KNN has the worst performance, because it is highly affected
by the noise due to outliers that may exist in the training pages,
which have been successfully neglected in ONB by eliminating the
support vectors. When TPs = 10000, precision, accuracy, and error
of KNN are; 0.7, 0.75, and 0.25 respectively. DOKNN has better
performance than KNN. For DOKNN, when TPs = 10000, precision,
accuracy, and error are; 0.72, 0.79, and 0.21 respectively. More-
over, NB and DONB have good performance as they are probabilistic
classiers. They both outperform SVM and KNN as they depend on
NB theorem. When TPs = 10000, NBs precision, accuracy, and error
are; 0.78, 0.83, and 0.17 respectively, while, DONBs precision, accu-
racy, and error are; 0.81, 0.86, and 0.14. On the other hand, SVMs
precision, accuracy, and error are; 0.74, 0.81, and 0.19 respectively.
Again, consider Figs. 1113. Fig. 11 shows the precision against
TPs. Generally, the precision for all classiers increases gradually
by increasing the number of TPs. It is noticed that, ONB has the
highest precision compared with the others. On the other hand, as
illustrated in Fig. 12, ONB introduces higher accuracy than other
classiers. ONBs accuracy is gradually improved by increasing the
number of TPs till reaches 0.89 when TPs = 10000. Finally, Fig. 13
depicts the error; it is concluded that ONB introduced signicant
error reduction compared with its competitors. The reason is that
ONB combines evidence from NB and SVM classiers.
200 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204

Fig. 19. Relevancy score against retrieved pages of TFC and EFC using STS as a link weighting strategy.

Fig. 20. Relevancy score against retrieved pages of TFC and EFC using Bay as a link weighting strategy.

7.4. Effects of the proposed distiller on the crawling performance
In general, focused crawlers can be evaluated by measuring their
ability to retrieve good pages. A good page is the one that is
highly relevant to the domain of interest. As depicted in Fig. 14, all
web pages are represented by the Web set, which is denoted as;
W, while the set of good pages (e.g., relevant to the domain of
interest), is called Relevant set and is denoted as; R. On the other
hand, the set of Crawled is denoted as; C, which represents the
pages that the crawler has visited, hence, (R,C)W. The target of the
focused crawler is to maximize the set RC, which indicates a large
number of good retrieved pages while keeping the set X = C-(RC) as
small as possible. An ideal focused crawler, which is unrealistic, has
CR and accordingly X = . Fig. 14(A) represents the ideal focused
crawler, while Fig. 14(B) represents the traditional focused crawler.
The target of the focused crawler is expressed in (16).
Target(Focused Crawler) = [max(C R)]AND[min(C (C R)] (16)
For illustration, assuming a focused crawler CRL whose domain
of interest is Education. The set W, which represents all the avail-
able pages in all domains, are assumed to be 1000 pages in which
100 pages are related to Education domain, which represent the
set R. In order to minimize the cost in terms of time and storage
penalties, CRL tries to visit only those relevant pages while dis-
carding irrelevant ones. The set C, which indicates the pages that
the crawler already visits, is assumed to be 200 pages, in which
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204 201

Fig. 21. Relevancy score against retrieved pages of TFC and EFC using STD as a link weighting strategy.

Fig. 22. Relevancy score against retrieved pages of TFC and EFC using TC as a link weighting strategy.

only 50 pages are related to Education. Hence |CR| = 50, and |C-
(CR)| = 150 accordingly, CRLs precision is 50/200 = 0.25. However,
if CRL crawls the same number of pages (e.g., 200 pages), in which
70 pages are relevant to Education, hence |RC| = 70, then CRLs
precision is 70/200 = 0.35. Then, one of the targets of CRL is to max-
imize |CR|. On the other hand, if CRL has the ability to retrieve 50
pages related to Education (e.g., |CR| = 50) by visiting only 125
pages (e.g., |C| = 125, then |C-(CR)| = 75), then CRLs precision is
50/125 = 0.4. Hence, an effective focused crawler should maximize
the number of crawled pages that are related the domain (e.g., the
set CR), while minimizing the retrieved irrelevant ones (e.g., the
set C-CR).
Education domain was selected to be the domain of interest. In
this section, the focused crawling task is done in two different sce-
narios. The rst is the Tradition Focused Crawler (TFC) as explained
in section 2.2, while the second is the Effective Focused Crawler
(EFC) which is illustrated in section 3. The crawling queue, in both
types of crawlers, is lled with 10 seed pages. Although those seeds
are highly related to the domain of interest, which is the Education
domain, they also contain bad links (e.g., links point at irrele-
vant pages), such as advertisement and utility pages. Several link
weighting strategies are used for implementing both crawlers (e.g.,
TFC and EFC), which are illustrated in Table 8. The tested crawlers
202 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204

Table 8
The link weighting strategies for implementing both crawlering ways (e.g., TFC and EFC).

Weighting strategy Description

Similarity To Seeds (STS) According to this strategy, the score of the currently processed page is the sum of its similarities to
the considered seeds. Hence, the input page as well as the seeds are expressed in a vector space
model of 485 dimensions (number of concepts in Education domain) {a1 ,a2 ,. . .,a485 }. In this
representation, if the concept Ci exists in the seed, the corresponding value ai will be assigned to
1 in the vector, otherwise it will be 0. Let Doc is the input page, the page score will be the sum
of all similarity scores to all seeds. Hence, the score for a page Doc is calculated by (17) (assuming z
training examples {e1 , e2 , . . .ez }). Finally, the pages score is assigned to all links extracted from it.

z

Score(Doc) = sim(Doc, ei ) (17)

i=1

 
The used similarity measure is the cosine similarity [40], which can be calculated using (18) as:
z 2
z 2
Simcos (Doc, e) = (ak (Doc).ak (e))/ ((ak (Doc)) . ((ak (e)) (18)
k (Doce) k=1 k=1
Bayesian (Bay) According to such weighting technique, the crawler uses a Bayesian similarity score to calculate
the relevancy of the processed page according to Bayes theorem using seeds as training examples.
In training phase, all seeds texts are combined, then domain keywords are extracted and mapped
to the corresponding concepts. The probability of each domain concept is calculated. When its
needed to assign a Bayesian score to an input page, pages domain keywords are extracted then
mapped to the corresponding domain concepts. The pages relevancy to the domain is calculated
(e.g., domain membership). Such domain relevance score is assigned (as a weighting score) to all
links extracted from the page.
Similarity To Domain (STD) In STD, Term Frequency (TF) is used to estimate the similarity between the currently processed
page and the domain of interest. Hence, the more the domain keywords found within the
processed page, the more the page-to-domain relevancy. Then, the calculated page-to-domain
relevancy is assigned to all links extracted from the page.
Treasure Crawler (TC) In TC [41], specic HTML elements are extracted from the input page. These elements are then
given to a domain relevancy calculator, which is supplied with the Dewy Decimal Classication
DDC system. Then, the set of DDC entries specify the crawlers topic. If the relevancy calculator
decides that an unvisited link is on-topic, then its HTML elements are compared to T-Graph nodes
and its priority score is assigned. However, if an unvisited link is off-topic, it receives the lowest
priority score and ignored by the crawler. On-topic URLs with their priority scores are then
injected into the fetcher queue for future downloads.

Table 9
Harvest Rate (HR) using different link weighting strategies in both crawlering ways (e.g., TFC and EFC).

Strategy Visited pages TFC EFC

Relevant Irrelevant Harvest Rate Relevancy Score Relevant Irrelevant Harvest Rate Relevancy Score

Similarity To Seeds 4000 2204 1796 0.551 0.469263 2245 1755 0.56125 0.530737
(STS) 5000 2240 2760 0.448 0.362875 3320 1680 0.664 0.637125
6000 2264 3736 0.377333333 0.362202 3376 2624 0.562666667 0.637798
7000 2273 4727 0.324714286 0.291012 4478 2522 0.639714286 0.708988
8000 3360 4640 0.42 0.348836 5510 2490 0.68875 0.651164
9000 3394 5606 0.377111111 0.334054 5567 3433 0.618555556 0.665946
10000 4430 5570 0.443 0.36998 6613 3387 0.6613 0.63002
Bayesian (Bay) 4000 2279 1721 0.56975 0.363967 3302 698 0.8255 0.636033
5000 3336 1664 0.6672 0.443952 3378 1622 0.6756 0.556048
6000 3398 2602 0.566333333 0.389742 4456 1544 0.742666667 0.610258
7000 4429 2571 0.632714286 0.422513 5534 1466 0.790571429 0.577487
8000 5503 2497 0.687875 0.403021 6623 1377 0.827875 0.596979
9000 5569 3431 0.618777778 0.411013 7723 1277 0.858111111 0.588987
10000 6613 3387 0.6613 0.439517 8802 1198 0.8802 0.560483
Similarity To Domain 4000 3309 691 0.82725 0.482959 3342 658 0.8355 0.517041
(STD) 5000 3369 1631 0.6738 0.410157 4421 579 0.8842 0.589843
6000 4453 1547 0.742166667 0.426159 5512 488 0.918666667 0.573841
7000 5521 1479 0.788714286 0.405317 6623 377 0.946142857 0.594683
8000 5587 2413 0.698375 0.402777 7723 277 0.965375 0.597223
9000 6614 2386 0.734888889 0.411544 8807 193 0.978555556 0.588456
10000 6653 3347 0.6653 0.37163 9901 99 0.9901 0.62837
Treasure Crawler (TC) 4000 3699 301 0.92475 0.483875 3948 52 0.987 0.516125
5000 4060 940 0.812 0.450434 4932 68 0.9864 0.549566
6000 4997 1003 0.832833 0.455419 5902 98 0.983667 0.544581
7000 5845 1155 0.835 0.465589 6822 178 0.974571 0.534411
8000 6233 1767 0.779125 0.442433 7855 145 0.981875 0.557567
9000 7188 1812 0.798667 0.449597 8853 147 0.983667 0.550403
10000 8864 1136 0.8864 0.474608 9930 70 0.993 0.525392
 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
are allowed to retrieve 10000 pages, and then the crawling harvest
rate and average relevancy are calculated.
7.4.1. Measuring the harvest rate
Harvest Rate (HR) is dened as the parentage of the retrieved
and relevant pages over the overall retrieved pages during the
crawl. Therefore, if 10 relevant pages are found in the rst 100
crawled pages, then a harvest rate of 10% at 100 pages is then con-
cluded. HR can be calculated using (19), while results are shown in
Table 9. Assuming nr is the number of retrieved and relevant pages
while n is the total number of retrieved pages.
HR = nr /n
Figs. 1518 show the harvest rate of the proposed EFC as well
as TFC against the number of retrieved pages using several link
weighting strategies, which are; STS, Bay, STS, and TC respectively.
As illustrated in such gures, EFC outperforms TFC regardless of
the used link weighting strategy. This proves the effectiveness of
adding domain distillers to traditional focused crawlers.
7.4.2. Relevancy score
In this experiment, evaluating the quality of pages retrieved by
various crawlers (e.g., TFC and EFC) has been done by employing the
different considered weighting strategies illustrated in Table 8 (e.g.,
STS, Bay, STD, and TC). Generally, if we have N different crawlers,
which are allowed to start from the same seeds (manually cho-
sen 10 seeds related to the Education domain) and run till each
crawler retrieves P pages. Hence, N * P pages (from all crawlers)
are collected. After removing similar pages retrieved by different
crawlers, M pages are available, which are ranked using a human
assessor into 5 different categories with a corresponding score
according to the Table 10. Then, the crawler score can be calculated
using (20).

P 
N 
P
Score(Crawlerj ) = score(pageji )/ ( score(pageji ))
i=1 j=1 i=1
Where; pageji is the ith page retrieved by the jth crawler. When such
evaluation strategy is followed using P = 4000, 5000, . . .., 10000, N = 2
(the number of crawling strategies, which are EFC and TFC). Result is
illustrated in Figs. 1922. Those gures show that EFC outperforms
TFC as it achieves the higher score, which indicates the high quality
of the retrieved pages. Surely, this indicates the effectiveness of
adding domain distillers to traditional focused crawlers.
8. Conclusion
Traditional focused crawlers rely on estimations, hence, they
estimate whether the page is related to DOI or not before actually
retrieving it. If the estimation decides that the page is good, the page
will be retrieved then passed to the indexer, then the pages links
are injected into the crawling queue. However, if the estimation is
not accurate, the crawler will be skewed away from its target of
retrieving pages related to a specic domain. The originality of this
paper is concentrated in introducing an effective modication on
the behavior of focused crawlers by employing a domain distiller
to decide whether the retrieved page is related to crawlers DOI,
and then take a decision to index the page and add its links to the
crawling queue accordingly. The proposed domain distiller com-
bines SVM and NB classiers in a new instance called Optimized
Nave Bayes (ONB) classier. Initially, genetic algorithm (GA) is used
to optimize the soft margins of SVM. Then the optimized SVM is
employed to discard the outliers from the available training exam-
ples. Next, the pruned examples are used to train the traditional NB
classier. Moreover, ONB employs word sense disambiguation to
203
Table 10
Different categories and the corresponding scores.
Category ID Description Score
1 Very relevant 20
2 Relevant 15
3 Medium 10
4 Related 5
5 Irrelevant 0
identify the accurate sense of each domain keyword extracted from
the input page. ONB has been tested against recent classication
(19) techniques. Experimental results have proven the effectiveness of
ONB as it introduces the maximum classication accuracy. Also,
results indicate that the proposed distiller improves the perfor-
mance of focused crawling in terms of crawling harvest rate.
References
[1] M. Razek, Credible mechanism for more reliable search engine results, Int. J.
Inf. Technol. Comput. Sci. 3 (2015) 1217.
[2] R. Shettar, R. Bhuptani, A vertical search engine based on domain classier,
Int. J. Comput. Sci. Secur. 2 (4) (2007) 1827.
[3] A. Elyasir, K. Anbananthen, Focused web crawler, International Conference on
Information and Knowledge Management Vol. 45 (2012).
[4] A. Sun, E. Lim, W. Ng, Web classication using support vector machine, in:
Proceedings of the 4th International Workshop on Web Information and Data
Management, New York, ACM Press, 2002, pp. 9699.
[5] O. Kwon, J. Lee, Web page classication based on k-nearest neighbor
approach, in: Proceedings of the 5th International Workshop on Information
Retrieval with Asian Languages, Hong Kong,China, ACM Press, 2000, pp. 915.
[6] J. Orallo, Extending decision trees for web categorization, Proceeding of 2nd
Annual Conference of the ICT for EU India Cross Cultural Dissemination
(2005).
[7] Z. Liu, Y. Zhang, A competitive neural network approach to web-page
categorization, Int. J. Uncertainty Fuzziness Knowl. Based Syst. 9 (6) (2001)
731741.
[8] A. Saleh, A. El Desouky, S. Ali, Promoting the performance of vertical
recommendation systems by applying new classication techniques, Knowl.
Based Syst. 75 (2015) 192223.
[9] R. Navigli, Word sense disambiguation: a survey, ACM Comput. Surv. 41 (2)
(20) (2009).
[10] H. Isahara, K. Kanzaki, Advances in natural language processing, in:
Proceedings 8th International Conference on NLP, Springer, Japan, October
2224, 2012.
[11] T. Udapure, R. Kale, R. Dharmik, Study of web crawler and its different types,
IOSR J. Comput. Eng. (IOSR-JCE) 16 (1) (2014) 15.
[12] S. Brin, L. Page, The anatomy of a large-scale hypertextual Web search engine,
Comput. Netw. ISDN Syst. 30 (7) (1998) 107117.
[13] M. Selvakumar, A. Vijaya, Design and development of a domain specic
focused crawler using support vector learning strategy, Int. J. Innov. Res.
Comput. Commun. Eng. 2 (5) (2014).
[14] M. Jamali, H. Sayyadi, B. Hariri, H. Abolhassani, A method for focused crawling
using combination of link structure and content similarity, In Web
Intelligence, IEEE Comput. Soc. (2006) 753756.
[15] S. Zheng, P. Dimitriev, C.L. Giles, Graph based crawler seed selection,
Proceedings of the 18th International Conference on World Wide Web
(WWW) (2009) 10891090.
[16] G.A. Miller, WordNet: a lexical database for english, Commun. ACM 38 (11)
(1995) 3941.
[17] Y. Wang, Z. Gong, Hierarchical Classication of Web Pages Using Support
Vector Machine, Lecture Notes in Computer Science, Vol. 5362, Springer,
2008, pp. 1221.
[18] R. Chen, C. Hsieh, Web page classication based on a support vector machine
using a weighted vote schema, Expert Syst. Appl. 31 (2) (2006) 427435.
[19] W. Liu, G. Xue, Y. Yu, H. Zeng, Importance-based web page classication using
cost-sensitive SVM, Proceedings of International Conference on Web-Age
Information Management (2005) 127137.
[20] E. Youn, M. Jeong, Class Dependent Feature Scaling Method Using Naive Bayes
Classier for Text Data Mining, Pattern Recognition Letters, Science Direct,
Vol. 30, ELSEVIER, 2009, pp. 477485.
[21] W. Zhanga, F. Gaoa, An improvement to naive bayes for text classication,
Proc. Eng. 15 (2011) 21602164.
[22] Z. Mei, Q. Shen, B. Ye, Hybridized KNN and SVM for gene expression data
classication, Life Sci. J. 6 (1) (2009) 6166.
[23] T. Li, S. Zhu, M. Ogihara, Text categorization via generalized discriminant
analysis, Inf. Process. Manage. 44 (5) (2008) 16841697.
[24] F. Li, Y. Yang, A Loss Function Analysis for Classication Methods in Text
Categorization, ICML, 2003, pp. 472479.
[25] D. Miao, Q. Duan, H. Zhang, N. Jiao, Rough set based hybrid algorithm for text
classication, Expert Syst. Appl. 36 (5) (2009) 91689174.
204 A.I. Saleh et al. / Applied Soft Computing 53 (2017) 181204
[26] G. Ifrim, A. Bayesian, Learning Approach to Concept-Based Document
Classication, Computer Science Dept., Saarland University, Saarbrucken,
Germany, 2005 (February M.Sc Thesis).
[27] R. Vinoth, A. Jayachandran, M. Balaji, R. Srinivasan, A hybrid text classication
approach using KNN and SVM, Int. J. Adv. Found. Res. Comput. (IJAFRC) 1 (3)
(2014) 23484853.
[28] N. Tripathia, M. Oakesa, S. Wermterb, Hybrid classiers based on semantic
data subspaces for two-level text categorization, Int. J. Hybrid Intell. Syst. 10
(2013) 3341.
[29] A. Sey, A. Patel, J.C. Jnior, Empirical evaluation of the link and content-based
focused Treasure-Crawler, Comput. Stand. Interf. 44 (2016) 5462.
[30] S. Yang, OntoCrawler: a focused crawler with ontology-supported website
models for information agents, Expert Syst. Appl. 37 (2010) 53815389.
[31] G. Forman, An extensive empirical study of feature selection metrics for text
classication, J. Mach. Learn. Res. 3 (2003) 12891305.
[32] D. Bollegala, Y. Matsuo, M. Ishizuka, Measuring semantic similarity between
words using web search engines, Proceedings of International Conference on
World Wide Web (2007) 757766.
[33] V. Cherkassky, M. Yunqian, Practical selection of SVM parameters and noise
estimation for SVM regression, Neural Netw. 17 (1) (2004) 113126.
[34] T. Pencheva, K. Atanassov, A. Shannon, Modelling of a roulette wheel selection
operator in genetic algorithms
[35] F. Alabsi, R. Naoum, Comparison of selection methods and crossover
operations using steady state genetic based intrusion detection system, J.
Emerg. Trends Comput. Inf. Sci. 3 (7) (2012).
[36] H. Ali, A. El Desouky, A. Saleh, Studying and analysis of a vertical web page
classier based on continuous learning Nave bayes (CLNB) algorithm, IGI
Global (2009) 210245.
[37] R. Meusel, P. Petrovski, C. Bizer, The web data commons microdata, RDFa and
microformat dataset series, in: Proceedings of the 13th International
Semantic Web Conference (ISWC 2014), Italy, Springer Berlin Heidelberg,
2014, pp. 277292.
[38] Y. Lin, J. Jiang, S. Lee, A similarity measure for text classication and
clustering, IEEE Trans. Knowl. Data Eng. 26 (7) (2014) 15751590.
[39] C. Hsu, C. Chang, C. Lin, A Practical Guide to Support Vector Classication,
Technical Report, Department of Computer Science and Information
Engineering, National Taiwan University, Taipei, 2003.
[40] J. Zhao, M. Lan, J. Tian, ECNU: using traditional similarity measurements and
word embedding for semantic textual similarity estimation, in: Proceedings
of the 9th International Workshop on Semantic Evaluation (SemEval 2015),
Denver, Colorado, June, 2015, pp. 117122.
[41] A. Sey, A. Patel, J. Jnior, Empirical evaluation of the link and content based
focused Treasure Crawler, Comput. Stand. Interf. 44 (2016) 5462.