Experimenting N-grams in text categorization

Abstract EN

This paper deals with automatic supervised classification of documents.

The approach suggested is based on a vector representation of the documents centered not on the words but on the n-grams of characters for varying n.

The effects of this method are examined in several experiments using the multivariate chi-square to reduce the dimensionality, the cosine and Callback and Libeler distances, and two benchmark corpuses the routers-21578 newswire articles and the 20 newsgroups data for evaluation.

The evaluation was done, by using the macro averaged F1 function.

The results show the effectiveness of this approach compared to the Bag-Of-Word and stem representations.