Self ranking and evaluation approach for focused crawler based on multi-agent system

Abstract EN

The need of better way of retrieving information and dealing with the increasing complexity and volume of information for users is an important research theme.

Retrieving information from the www via search engine may be deliberate as the most significant one.

Most of the recent efforts that had been done in this area suggest a better solution for general-purpose search engine limitations.

That leads to a new generation of search engines called vertical-search engines.

However, all current focused crawlers; (crucial component within vertical search engine) crawling strategies introduced high penalties as they consume network bandwidth and resources of hosting servers.

This paper introduces a new approach for focused crawling that integrates evidence from both focused crawling and intelligent multi-agent technology.

It will help in distributing the computation among data sites, hence overcoming the drawbacks of traditional approaches.

We evaluate its performance compared to the client / server-based computation model from perspectives of amount of data transfer and execution time through both simulation and analytical study.

The results achieved from the experiments confirm efficiency of the proposed approach.

Both of execution time and traffic are reduced in the case of mobile agent compared with client / server model (i.e., eliminating the bottleneck, and overhead problem at crawler site).

Furthermore, such combination will also simplify the design and implementation of focused crawlers.