Precision Evaluation of an Information Retrieval System using Link Scores
Propagation.
Idir Chibane
Supélec, Plateau de Moulon, 3 rue Joliot
Curie, 91 192 Gif/Yvette, France
Idir.Chibane@supelec.fr
Abstract
Web search engines have become indispensable in
our daily life to help us find the information we need.
Several search tools, for instance Google, use links to
select the matching documents against a query. In this
paper, we propose a new ranking function that
combines content and link rank based on propagation
of scores over links. This function propagates scores
from sources pages to destination pages in relation
with the keywords of a query. We assessed our ranking
function with experiments on the TREC-9 collection.
We conclude that propagation link scores according to
query terms provide significant improvement for
information retrieval.
1. Introduction
The explosive growth of the web has led to surge of
research activity in the aria of information retrieval
(IR) on the Wold Wide Web. Ranking has always been
an important component of any information retrieval
system (IRS). In the case of web search its importance
becomes critical. Due to the size of the Web (Google
counts more 4.28 billion Web pages in January
2005[12]), it is imperative to have ranking function
that capture the user needs. To this end the Web offers
a rich context of information which is expressed
thought links. In this paper we investigate,
theoretically and experimentally, the application of
link analysis to ranking on the Web.
In recent years, several information retrieval
methods using the information about the link structure
have been developed and proved to provide significant
enhancement to the performance of Web search in
practice. Actually, most of systems based on link
structure information combine content with popularity
measure of the page to rank query results. Google’s
PageRank[1] and Keinberg’s HITS[2] are two
fundamental algorithms by employing the link
Bich-Liên Doan
Supélec, Plateau de Moulon, 3 rue Joliot
Curie, 91 192 Gif/Yvette, France
Bich-Lien.Doan@supelec.fr
structure among the Web page. A number of
extensions of these two algorithms are also proposed,
such as [3][4][5][6] [7][8]. All these link analysis
algorithms are based on two assumptions: (1) If there
is a link from page A to page B, then we may assume
that page A endorses and recommends the content of
page B. (2) Pages that are co-cited by a certain page
are likely to share the same topic as well as to help
retrieval.
The rest of this paper is organized as follows. In
Section 2, we review the recent works on link analysis.
We first review the related literature on link analysis
ranking algorithms and also present some extension of
these algorithms. Then we present our information
retrieval model with the new ranking function. In
section 4, we show the experimental results on
multiple queries using the proposed algorithm,
including a comparative study of different algorithms.
In Section 5, we summarize our main contributions and
discuss possible new applications for our proposed
method.
2. Related Works
Various studies suggested taking account links
between documents in order to increase the quality of
information retrieval. PageRank[1] of Google and
HITS[2] of Keinberg are the basic algorithms using
link structure information. Generally, these systems
function in two steps. In first stage, a traditional search
engine returns a list of pages in response to user query.
In the second stage, these systems take account links to
rank the documents results. In this section we describe
some of previous link analysis ranking algorithms.
2.1 Page Rank
PageRank was invented by Lawrence Page and
Sergey Brin [1] and used in Google search engine. The
principle of the algorithm consists to evaluate the
importance of each page according to each link research carried out. Because search engines does not
pointing them according to the following assumption: take into account semantics, context or user profile.
“a page is important when it is much cited or cited by From where, the idea to compute personalized PR.
an important page”. PageRank precomputes a rank Last years, research led to three radically different
vector that provides a-priori “importance” estimates solutions [3]: modular pagerank, blockrank and topic
for all the pages on the Web. This vector is computed sensitive pagerank. These approaches approximate PR
once, offline, and is independent of the search query. with some approximation, although they differ
At query time, these importance scores are used in substantially in their computational requirements and
conjunction with query-specific IR scores to rank in the granularity of personalization achieved.
query results. PageRank simulates a user navigating
randomly in the Web who jumps to a random page 2.1.1. Modular PageRank. Modular pagerank
with probability (1-d) or follows a random hyperlink approach was proposed by Jeh and Widom [4]
(on the current page) with probability d. This process according to the following principle: "it is possible to
can be modelled with a Markov chain, from where the compute a good approximation of PR of all the Web
stationary probability of being in each page can be pages by cutting out the entire Web starting from an
computed. initial set of pages, called Hub set". The Hub set can
Let C(A) be the number of outgoing links of page A be composed by pages whose PR is highest (pages that
and suppose that A is pointed to by pages T1, T2, …, are very important), pages mentioned in Yahoo! ODP
Tn. The PageRank, PR (A) of the page A is given by (Open Directory Project) or important pages for
the following formula: particular company (like Microsoft). From each
N
PR (Ti ) pœ”Hub set”, a zone of pages is built. A zone is made
PR ( A) = (1 − d ) + d ∑
i =1 C (Ti )
up of all the pages which we can attain it by starting
from page p. Jeh and Widom represented PPV’s
With d is a damping factor which can be set between 0
(personalized PageRank vector) as a linear
and 1. They usually set d to 0.85. Notice that the
combination of |Hub set| hub vectors, one for each
ranking (weight) of other pages is normalized by the
pœ“Hub set”. Any PPV based on hub pages can be
number of links in the page.
constructed quickly from the set of precomputed hub
Intuitively, this formula means that the PR of a page
vectors, but computing and storing all hub vectors is
A depends at the same time on the quality and the
impractical. To compute a large number of hub vectors
number of pages which pointes A. That means a page
efficiently, they further decompose them into partial
which has a link from a very relevant page (a page
vectors and the hubs skeleton, components from which
with good reputation for example the home page of
hub vectors can be constructed quickly at query time.
Yahoo!) is considered relevant. Pages with in-links
The best results are obtained when the Hub set is
from many not those relevant pages are also
composed of pages whose PR is high.
considered relevant.
Pagerank can be computed using an iterative
2.1.2. The BlockRank. This algorithm starts from an
algorithm. Initially, all the pages are equiprobable,
experimental statement: "the links between Web pages
their PR value is equal to 1/N, where N is the total
are not distributed uniformly". Indeed, there are
number of documents in text collection. A cycle of
groups of strongly inter-connected pages, like those
iteration is necessary to propagate the probabilities on
constituting domain, Web site or repertory. This
the pages. The algorithm stops theoretically when a
group of pages is then connected to the other pages by
new iteration does not produce any more modifications
a low number of links. Experiments showed that about
of PR values of all the pages. In practice convergence
79% of links are in the same host. And about 84% of
is obtained at tens iterations. The System using PR has
links are in the same domain [8]. Considering the Web
a basic characteristic which distinguishes it from the
is a nested structure, the Web graph could be
other systems. The most complex and longest
partitioned into blocks according to the different level
computations are made offline. Therefore, the system
of Web structure, such as page level, directory level,
has only very simple computations to execute in order
host level and domain level. Furthermore, the
to build the query results.
hyperlink at the block level could be divided into two
The PR computations are long and require cleaning
types: Intra-hyperlink and Inter-hyperlink, where inter-
the entire Web. Moreover, the results obtaining by
hyperlink is the hyperlink that links two Web pages
Google shows that the algorithm witch compute PR
over different blocks while intra-hyperlink is the
value of a page is not completely relevant. The query
hyperlink that links two Web pages in the same block.
results do not have sometimes any relationship with
The idea of blockrank algorithm consists in first step to
compute a "local" PR on only one block. With number
reduced of pages (only pages that constitute the block),
the PR algorithm converges quickly. In the second
step, to obtain an excellent approximation of full PR,
the block importance is computed from a matrix of
blocks, but not the entire Web. This value is called
BlockRank(BR)[5]. The BR algorithm is defined as
follows:
- The Web is cut out into blocks according to the
type of block (repertory, site or domain).
- The local PR of each page in each block is
computed
- The importance relative to each block (noted BR) is
also computed
- The local PR of each page and the BR are
combined to estimate the full PR of the page.
This algorithm decreases PR computing time (up to 20
times less). It also allows determining a good effective
approximation of PR for all new page.
2.1.3. Topic Sensitive Pagerank. This new algorithm
proposed by Haveliwala Taher [7] consist on reuse of
traditional PageRank. The idea of this algorithm is:
"the traditional PR rank pages according to their
importance whose contents are identical. But it does
not distinguish a page which speaks about the animal
jaguar or the car jaguar". The solution suggested in
Topic Sensitive Pagerank is to compute a set of
importance scores for each page according to various
topics. The computations are defined as follow:
The first step in this approach is to generate a set of
biased PageRank vectors using a set of basis topics.
This step is performed once, offline, during the pre-
processing of the Web crawl. There are many possible
sources for the basis set of topics. However, using a
small basis set is important for keeping the pre-
processing and query-time costs low. One option is to
cluster the Web page repository into a small number of
clusters in the hopes of achieving a representative
basis. Haveliwala Taher chose instead to use the freely
available, hand constructed Open Directory as a source
of topics. To keep the basis set small, he made use of
the 16 top level categories. The result gives 17
different vectors of pagerank for each page: full
pagerank, and 16 biased pageranks correspondent at
each level categories. These values are stored in the
index. The second step is to determine the topics most
closely associated with the query, and use the
appropriate topic-sensitive pagerank vectors for
ranking the documents satisfying the query. This can
be obtained by syntactic and semantic analysis of
query terms, easy to calculate.
2.2 Kleinberg’s HITS procedure
Kleinberg distinguishes between two different
notions of relevance: an authority is a page that is
relevant in itself; a hub is a page that is relevant since
it contains links to many related authorities. There is,
what Kleinberg calls, a natural equilibrium between
authorities and hubs: "Hubs and authorities exhibit
what could be called a mutually reinforcing
relationship: a good hub is a page that points to many
good authorities; a good authority is a page that is
pointed to by many good hubs." [2].
In [2], Kleinberg introduced a procedure for
identifying web pages that are good hubs or good
authorities, in response to a given query. To identify
good hubs and authorities, Kleinberg’s procedure
exploits the graph structure of the web. Each web page
is a node and a link from page A to page B is
represented by a directed edge from node A to node B.
When introducing a query, the procedure first
constructs a focused subgraph G, and then computes
hubs and authorities scores for each node of G (say N
nodes in total). In order to quantify the quality of a
page as a hub and an authority, Kleinberg associated
every page with a hub and an authority weight.
According to the mutual reinforcing relationship
between hubs and authorities, Kleinberg defined the
hub weight to be the sum of the authority weights of
the nodes that are pointed to by the hub, and the
authority weight to be the sum of the hub weights that
point to this authority. Let A denote the n-dimensional
vector of the authority weights, where Ai is the
authority weight of node i (or page i) and let H denote
the n-dimensional vector of the hub weights, where Hj
is the hub weight of node j. We have that
Ai = ∑ H j and H j = ∑ Ai
j∈In (i ) i∈Out ( j )
Where In(i) represent the inlinks set of page i, Out(j)
the outlinks set of page j. Hi and Ai represent the hub
and authority weight of page i.
An extension of HITS algorithm is also proposed by
a group of Microsoft search [6]. Ji-Rong Wen & al
propose a novel block-based HITS algorithm to solve
the noisy links and topics drifting problem of the
classical HITS algorithms. The basic idea is to segment
each Web page into multiple semantic blocs using a
vision based page segmentation algorithm. The main
step of the HITS algorithm can be performed at the
bloc level instead of the page level. For each page, the
bloc with highest block rank (BRMax) is selected, and
then it combines with hub weight of the page as
follow:
Rank(d ) = α ⋅ rankPR (d ) + β ⋅ rankPR _ BRMax(d )
3. Our work
The idea underlying our work is that the popularity
of a page depends on scores of pages that point to it
according to terms of query. We proposed a function
that combines link rank and document rank in relation
with query terms. We will describe this function later.
First, we start with the text processing.
function on the ranking of query results. RankLR(Pi,Q)
is a link rank based on propagation scores over links
according to Query terms and RankDR(Pi,Q) the
document rank of the page Pi witch is based on
content-only of the page. We will more explain such
function in the following sections.
3.3. Scoring Function Details

3.1. Text Processing
Our system processed documents by first removing
HTML tags and punctuation, and then excluding high-
frequency terms by using a stop words list. After
punctuation and stop word removal, the system
replaced each word by its representative class (root) by
using a porter stemming algorithm [11]. To represent
documents and query, we have used the vector model
[9]. This choice is justified by its success in the Web
community and the satisfactory results that it
The primary innovation in our work arises from the
use of a ranking function which depends on content
and popularity of page according to query terms. This
dependence allows better adequacy of found results by
traditional model of IR with user needs. Our ranking
function is based on two measures: one is traditional
and widely used in current systems witch is the cosine
measure. It computes the cosine of the angle between
query and document vector. This measure is defined as
follows:
P ⋅Q
∑ w ⋅w t i , Pi ti ,Q

Rank ( Pi , Q ) = i
=
t i ∈ Pi ∩ Q
(2 )
generates. In the following section, we define the DR
Pi
2
⋅ Q
2
∑ w t2i , Pi . ∑ w t2i , Q
retrieval methodology that we respect and we detail the t i ∈ Pi ti

scoring function that we used. We also present the Where wij, wqj , j=1 to t (total of terms in the entire
experiments and the results of our evaluation on TREC collection), are term weights assigned to different
collection. terms for the page Pi and query Q respectively. The
best known term-weighting schemes use weights witch
are given by
3.2. Retrieval methodology N
w ij = f ij × log
DF j
Our experiments are conducted according to the
Where fij is the normalized frequency of term kj in
following steps:
page Pi and DFj is the Document Frequency (DF),
that measures the number of documents in which a
Step 1. Initial Retrieval: An initial list of ranked web
term kj has appeared in the entire document collection.
pages that contain at least one query term is obtained.
fij is given by :
The Document rank obtained by using the TF-IDF tf ij
f ij = 0 . 5 + 0 . 5 ×
scores in this step is called DR (Document Rank). It is max tf ij
based on content-only of the page. Where tfij is the Term Frequency (TF), which measures
the number of times a term kj appears in page Pi.
Step 2. Page Partition: A partition method is applied The second one is the structural measure that takes
to partition the retrieved pages into sets. Each set is account link structure information. In order to
composed by pages that contain exactly k terms of understand our function, we start with the following
query (k=1...n) where n is the number of terms of assumption “We considered that a page is well-known
query. for a term t of a query if it contains incoming links
from pages which have the term t”. This measure can
Step 3. Final Retrieval: The final list of ranked web be computed as follows:
pages is constructed by pages that contain all query Let Q is a query containing n terms. Let us suppose
terms. For each page of the final list, we calculate the a found page Pj by a traditional system of IR. Let T(Q)
link rank by applying the function proposed in the next and T(Pj) the set of query terms and Pj terms
section. After obtaining the link rank, a combined respectively. We denote by In(Pj) the set of pages that
rank is used, in which the rank of each web page Pi is point to the page Pj (incoming link) and |In(Pj)| the
determined by: number of incoming link of the page Pj. Our link rank
rank (Pi , Q ) = α ∗ rank DR (Pi , Q ) + (1 − α ) ∗ rank LR (Pi , Q ) (1) is defined as follows.
With α is a parameter witch can be set between 0 and ∑[ ( )]∩ [ T ( Pi )∩ T (Q ) = k ] Rank ( Pi , Q )
(P ,Q )=
n
k DR
∑C (3 )
Pi ∈ IN P j
Rank k
∗ ∗β ∗
1. It allows us to see the impact of our link rank LR j
k =1
n
n IN ( Pj )
 k
With Cn is the number of sets that contain exactly k algorithm. The red line shows the results of using
INDEGREE algorithm witch is a simple heuristic that
terms of query. It is given by this formula: can be viewed as the predecessor of all Link Analysis
n!
C nk = Ranking algorithms. It ranks the pages according to
k !∗ (n − k )!
their popularity where the popularity of a page is
β is a parameter which is between 0 and 1 and verifies measured by the number of pages that link to this page.
the following condition: The last one shows the results of combining link rank
n
n! k n
(n − 1)! ∗ β = 1. and document rank with α being optimal for the
∑
k =1 k!∗(n − k )! n
∗ ∗β =1 ≡ ∑
k =1 (k − 1)!∗(n − k )!
specific method (α=0,15).
We have:
n
(n − 1)! = 2 n−1 ⇒ β = 1
∑
k =1 (k − 1)!∗(n − k )! 2 n −1
0,6 Combining link and document rank
Content-Only (Baseline)

After transformation of the function (3), the final 0,5 INDEGRE

function is presented as follows: 0,4

(n − 1)! ∑ [ ( )] [ ( ) ( ) ] Rank (P )

Precision
Rank (P ) =
n DR i
∑ (4 )
Pi ∈ IN P j ∩ T Pi ∩ T Q = k
LR j
2 n −1
* IN ( Pj )
k =1 (k − 1)!∗ (n − k )! 0,3

0,2

4. Experiments on TREC collection 0,1

0
In this section, we start by describing the test 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

collection that we have used in our experiments. Then 11 standard recall levels

we detail the tests carried out. We carried out 50 Figure 1. Comparisons of three ranking function on
queries and we compared three ranking functions. precision over the 11 standard recall levels

4.1 The WT10g collection The dependency between precision at 0%, 10%,
20% of recall levels and α is illustrated in Figure 2, in
In our experiments, we have chosen WT10g which all the curves converge to the baseline when
collection like test collection. This collection contains α=1.
1.692.096 documents (Web page) with 1.532.012 0,55 0% recall 10% recall 20% recal
pages with incoming links and 1295841 pages with 0,5
outgoing links. Due to our limited storage resources
0,45
we had to restrict our experiments to a relatively small
Précision

subset of the TREC collection. For our experiments we 0,4

downloaded pages of 490 Web sites appear in the 0,35

collection. We have selected the sites that containing at 0,3
least two relevant pages for one of queries carried out
0,25
on our system. The number of pages that were
downloaded is 546.423 with 477.064 pages with 0,2

incoming links and 410.378 pages with outgoing links.

05

15

25

35

45

55

65

75

85

95
0,

0,

0,

0,

0,

0,

0,

0,

0,

0,

combining parameter α

Figure2. Precision at 0%, 10% and 20% recall level with

4.2. Experimental Setup and Results different combining parameter α on WT10g

In this section we present an experimental
evaluation of the algorithm that we propose, as well as
some of the existing algorithms. We study the rankings
they produce. In our experiments, the precision over
the 11 standard recall levels which are 0%, 10%, …,
100% is the main evaluation metric, and we also
evaluate the precision at 5 and 10 documents retrieval
(P@5 & P@10).
Figure 1 shows the experimental results on
information retrieval using different ranking methods.
The first one witch is based on content-only of the
page and presented by the blue line is the baseline
As can be seen from Figure 1, the INDEGREE
algorithm performs the worst result. With this
algorithm, a page has the same score (or popularity
value) for all queries performed on the system because
it takes account the popularity of a page independently
of query terms. For that, the results were worse.
Combining link rank and document rank is strongly
better than the baseline, though it is the best among all
the methods. The performance of our method increases
significantly when α decrease. That means, more we
give importance to link rank, more the results are
greater. The better value of α is 0.15 to have more
relevant document at the top of rank list.
Table 1. P@5 and P@10 comparison
Link
Baseline 0.15*DR+
Precision InDegree Rank
DR 0.85*LR
LR
P@5 0,077 0,216 0,265 0,306
P@10 0,072 0,163 0,192 0,208
We also compare the different algorithms with the
average precision at 5 and 10 documents retrieval
(P@5, P@10). The performance of our algorithm is
still better than the baseline algorithm. That means
that there are more relevant documents in the top rank
list. From table 1, we can see that either link rank only
or combining link rank with document rank performed
best results than baseline, either on average precision
P@5 or P@10. For example, the P@5 and P@10 of
our algorithm is 0,306 and 0,208 respectively. This
result achieved 42% and 28% improvements over the
baseline algorithm on average precision at P@5 and
P@10 (0,216 and 0,163 respectively). The results in
this table show that InDegree algorithm was still the
worst one.
5. Conclusion
Several algorithms based on link analysis approach
were developed. But, until now many experiments
showed that there is no significant profit compared to
the methods based on content-only of the page. In this
paper, we introduce an approach for combining
content and link rank based on propagation scores over
links according to query terms. During the
computation, the algorithm that we proposed
propagates a portion of rank scores of the source web
pages to the destination web pages in accordance with
query terms. We performed experimental evaluations
of our algorithm using IR test collection of TREC 9
and found that this algorithm outperforms significantly
a content-only retrieval. The study concluded from
our experiments shows that propagation link scores
according to query terms provide significant
improvement. It is still better than the baseline method
based on content-only.
More study and experiments will be conduced, e.g.,
we will use the weighted inter-host and intra-host link
score propagation. We also plan to test this framework
at the semantic blocks level to see the structural effects
of blocks on ranking query results. Finally, new
measure representing additional semantic information
may be explored.
6. References
[1] S. Brin and L. Page, “The anatomy of a large-scale
hypertextual Web search engine”, Computer Networks and
ISDN Systems, 1998, 107-117.
[2] Kleinberg L., “Authoritative sources in a hyperlinked
environment”, In Proceeding of 9th ACM-SIAM Symposium
on Discrete Algorithms, 1998, 604-632.
[3] T. Haveliwala and S. Kamvar and G. Jeh, “An Analytical
Comparison of Approaches to Personalizing PageRank”,
technical report, Stanford University, 2003.
[4] G. Jeh and J. Widom, “Scaling personalized web search”,
In Proceedings of the Twelfth International World Wide Web
Conference, 2003.
[5] S. Kamvar and T. Haveliwala and C. Manning and G.
Golub, “Exploiting the Block Structure of the Web for
Computing PageRank”, Technical Report, 2003.
[6] Deng Cai; Shipeng Yu; Ji-Rong Wen; Wei-Ying Ma
(2004), “Block-based Web Search”, SIGIR '04: Proceedings
of the 27th annual international conference on Research and
development in information retrieval, ACM Press, Sheffield
United Kingdom, 456--463, 2004.
[7] Taher H. Haveliwala, “Topic-Sensitive PageRank : A
Context-Sensitive Ranking Algorithm for Web Search”,
Knowledge and Data Engineering, IEEE Transactions on,
2003.
[8] X.M. Jiang, G.R. Xue, W.G. Song, H.J. Zeng, Z. Chen
and W.Y. Ma, “Exploiting PageRank at Different Block
Level”, International Conference on Web Information
Systems Engineering, 2004.
[9] G. Salton, C.S. Yang and C.T. Yu, “A theory of term
importance in automatic text analysis”, Journal of the
American Society for Information Science and Technology,
1975, 33-44.
[10] M.F. Porter, “An algorithm for suffix stemming”, 1980.
[11] http://seattlepi.nwsource.com/business/160997_google-
18.html