An Improved Focused Crawler: Using Web Page Classification and Link Priority Evaluation

Joint Authors

Lu, Houqing
Zhan, Donghui
Zhou, Lei
He, Dengchao

Source

Mathematical Problems in Engineering

Issue

Vol. 2016, Issue 2016 (31 Dec. 2016), pp.1-10, 10 p.

Publisher

Hindawi Publishing Corporation

Publication Date

2016-05-19

Country of Publication

Egypt

No. of Pages

10

Main Subjects

Civil Engineering

Abstract EN

A focused crawler is topic-specific and aims selectively to collect web pages that are relevant to a given topic from the Internet.

However, the performance of the current focused crawling can easily suffer the impact of the environments of web pages and multiple topic web pages.

In the crawling process, a highly relevant region may be ignored owing to the low overall relevance of that page, and anchor text or link-context may misguide crawlers.

In order to solve these problems, this paper proposes a new focused crawler.

First, we build a web page classifier based on improved term weighting approach (ITFIDF), in order to gain highly relevant web pages.

In addition, this paper introduces an evaluation approach of the link, link priority evaluation (LPE), which combines web page content block partition algorithm and the strategy of joint feature evaluation (JFE), to better judge the relevance between URLs on the web page and the given topic.

The experimental results demonstrate that the classifier using ITFIDF outperforms TFIDF, and our focused crawler is superior to other focused crawlers based on breadth-first, best-first, anchor text only, link-context only, and content block partition in terms of harvest rate and target recall.

In conclusion, our methods are significant and effective for focused crawler.

American Psychological Association (APA)

Lu, Houqing& Zhan, Donghui& Zhou, Lei& He, Dengchao. 2016. An Improved Focused Crawler: Using Web Page Classification and Link Priority Evaluation. Mathematical Problems in Engineering،Vol. 2016, no. 2016, pp.1-10.
https://search.emarefa.net/detail/BIM-1112450

Modern Language Association (MLA)

Lu, Houqing…[et al.]. An Improved Focused Crawler: Using Web Page Classification and Link Priority Evaluation. Mathematical Problems in Engineering No. 2016 (2016), pp.1-10.
https://search.emarefa.net/detail/BIM-1112450

American Medical Association (AMA)

Lu, Houqing& Zhan, Donghui& Zhou, Lei& He, Dengchao. An Improved Focused Crawler: Using Web Page Classification and Link Priority Evaluation. Mathematical Problems in Engineering. 2016. Vol. 2016, no. 2016, pp.1-10.
https://search.emarefa.net/detail/BIM-1112450

Data Type

Journal Articles

Language

English

Notes

Includes bibliographical references

Record ID

BIM-1112450