Lexical noise analysis and removal in intelligent search engines

Abstract EN

In the field of intelligent information retrieval (IR), latent semantic indexing (LSI) is a popular technique used to retrieve information related more in meaning than in lexical matching.

This technique overcomes the problems associated with synonymy and polysemy (common causes of inaccuracy in matching algorithms).

A core component in the process is the use of the singular value decomposition (SVD) which acts as a mathematical model for the lexical noise in the term document matrix (TDM).

This paper investigates various aspects of LSI from the viewpoint of noise modeling and removal in image processing.

A discussion about and an investigation into, mathematical modeling for lexical noise in the TDM is presented.

The work addresses a definition for noise in text processing and seeks to determine the best structure of the TDM.

In other words, the structure of the TDM that would facilitate efficient searching within the LSI.