Ranking

Web Search Engine

Web Search Engine is a tool enabling document search, with respect to specified keywords, in the Web and returns a list of documents where the keywords were found.

Components of Web Search Engine :1. User Interface 2. Parser 3. Web Crawler 4. Database 5. Ranking Engine

USER INTERFACE It is the part of Web Search Engine interacting with the users and allowing them to query and view query results.
PARSER It is the component providing term (keyword) extraction for both sides. The parser determines the keywords of the user query and all the terms of the Web documents which have been scanning by the crawler.

WEB CRAWLER A web crawler is a relatively simple automated program, or script, that methodically scans or "crawls" through Internet pages to create an index of the data it is looking for. Alternative names for a web crawler include web spider, web robot, bot, crawler, and automatic indexer. INDEXING THE WEB Once a crawl has collected pages, full text is compressed and stored in a repository Each URL mapped to a unique ID

Ranking

The primary challenge of a search engine is to return results that match a users needs. A word will potentially map to millions of documents.

Types Of Ranking

Popularity Ranking Content Based Ranking Evolutionary Ranking Integrated Ranking

POPULARITY RANKING ALGORITHM

The popularity ranking uses the link structure of URLs to infer which pages are important in the web graph. The popularity ranking algorithm uses the page ranking is as follows: Qp(A) =(1-d)+d(Qp(T1)/C(T1)+..+Qp(Tn)/C(Tn )) Where Qp(A) is the Page Rank of page A, Qp(Ti) is the Page Rank of pages Ti which link to page A. C(Ti) is the number of outbound links on page Ti d is a damping factor which can be set between 0 and 1.

The Page Rank does not rank web sites as a whole, but it determined for each page individually. The Page Rank of page A is recursively defined by the Page Ranks of those pages which link to page A. Within the Page Rank algorithm, the Page Rank of a page T is always weighted by the number of outbound links C (T) on page T. This means that the more outbound links a page T has, the less will page a benefit from a link .

Evolution ranking

Popularity ranking and content ranking can produce static ranking of information. However, the importance of each web change decreases as time goes on. The evolution ranking reflects this decrease over time.

Content-based ranking
In evaluating the content of new information carried by web changes, there are two considerations: How much information carried and how timely that the information is. The amount of information can be evaluated simply from the length of the NIFs.

In order to evaluate how timely web changes are from the content of web changes, we need to know what kind of words can be good indicators of new information. Then count the word frequency of the web changes found as well as the word frequency of all web pages. Remove the stop words, general verbs and too common words that are unrelated to new information .

Divide the words into groups: Time information words, Time related words, related words, popular topic words and the rest go to misc words. It shows that web changes are more likely to contain time related information.
News or event information in web changes often have short lifetime. Such information also has more significance when we present web changes. Therefore, Count the appearances of time related words in NIFs as a metric of quality ranking.

Integrated ranking

The ranking scores of popularity ranking, content ranking and evolution ranking, Denote Qp as the normalized score of popularity ranking, Qc as the normalized score of content-based ranking and the of evolution ranking. Integrated Ranking score of New Information Fragments at query time is: Q (T) = (Qp + Qc) e(tt0) (3.4) Where t0 is the creation time of the web change.

Let choose the sum of Qp and Qc rather than the product as static score. Qp measures the importance of the page that contains the change. Such combination can well distinguish the significance of multiple changes of the same page at the same time. For changes on different pages, which is evaluated more than one another is determined by the configuration of the numerical scale between Qp and Qc.

Most Popular Ranking Algorithms

1.

2.
3.

PageRank Algorithm (Google) HITS Algorithm (IBM) Sim Ranking Algorithm

PageRank Algorithm (Google)

The PageRank algorithm, proposed by founders of Google Sergey Brin and Lawrance Page, is one of the most common page ranking algorithms that is also currently used by the leading search engine Google. In general the algorithm uses the linking (citation) info occurring among the pages as the core metric in ranking procedure. Existence of a link from page p1 to p2 may indicate that the author of p1 is interested in p2.

PageRank Algorithm
The PageRank metric PR(p), denes the importance of page p to be the sum of the importance of the pages that point to p. More formally, consider pages p1,,pn, which link to a page pi and let cj be the total number of links going out of page pj. Then, PageRank of page pi is given by: PR(pi)=d+(1-d)[PR(p1)/c1++PR(pn)/cn] where d is the damping factor.

PageRank Algorithm

This damping factor d makes sense because users will only continue clicking on links for a nite amount of time before they get distracted and start exploring something completely unrelated. With the remaining probability (1-d), the user will click on one of the cj links on page pj at random. Damping factor is usually set to 0.85. So it is easy to infer that every page distributes 85% of its original PageRank evenly among all pages to which it points.

Problems of PageRank Algorithm

1.

2.

3.

4.

It is a static algorithm that, because of its cumulative scheme, popular pages tend to stay popular generally. Popularity of a site does not guarantee the desired information to the searcher so relevance factor also needs to be included. In Internet, available data is huge and the algorithm is not fast enough. It should support personalized search that personal specifications should be met by the search result.

HITS Algorithm (IBM)

It is executed at query time, not at indexing time, with the associated hit on performance that accompanies query-time processing. Thus, the hub(going) and authority(coming) scores assigned to a page are queryspecific. It is not commonly used by search engines. It computes two scores per document, hub and authority, as opposed to a single score of PageRank. It is processed on a small subset of relevant documents, not all documents as was the case with PageRank.

SimRank Algorithm
Sim Rank is a measure that says two objects are similar if they are related to similar objects. It contains two phases which are: 1. Apply the similarity measure to the k-means algorithm to partition the crawled web pages into distinct WSNs. 2. Use an improved PageRank algorithm in which two distinct weight values are assigned to the title and body of a page, respectively.

Web Crawler of SimRank

It has the ability of eliminating useless documents such as advertising web pages that resulting in both disk and time saving.