Topical web crawling for domain-specific resource discovery enhanced by selectively using link-context

Abstract EN

To enable topical Web crawling, link-context is the critical contextual information of anchor text for retrieving domain-specific resources.

While some link-contexts may misguide topical Web crawling and extract wrong Web pages, because several relevant anchor texts become irrelevant or several irrelevant anchor texts become relevant after calculating the relevance between the link-contexts and the feature terms of the specific topic.

In view of above, this paper presents a heuristic-based approach by selectively using link-context and implements DOM tree to locate the anchor text.

Unlike previous crawling algorithms, which only zero in on link-context and ignore whether it is really needed or not? Our method cares both link-context and evaluating its necessity to correctly use link-context to guide topical crawling.

Accordingly, our topical crawler can retrieve more relevant Web pages.

Experimental results indicate that this approach outperforms breadthfirst, best-first, anchor text only, link-context both in harvest rate and target recall.