Mining the web pages by evolutionary algorithm

Abstract EN

The increasing amount of information on the web raises new and challenging problems for information retrieval.

We propose a framework for web mining, the applications of data mining and knowledge discovery techniques to data collected in World Wide Web (WWW), and a genetic search for search engines by showing that important relation existed between web statistical studies and search engines standard techniques in optimization.

Since the web is a graph that can be searched for relevant information with an evaluation function and with operators based on standard search engines or local exploration.

Then it is straightforward to define an evaluation function that is a mathematical formulation of the user request and to define a steady state genetic algorithm that evolves a population of pages with binary tournament selection.

Querying standard search engines performs the creation of individuals.

The crossover consists in combining links present in two parent web pages, if those two pages have a link in common, or links pointing to the same web site, then it might be desirable to combine this information and to focus the search on this common link or web site (local search).

Our tool digs out pages, which are significantly better than those found by standard search engines for complex queries.