A string prefix dependent dictionary structure based on hashing and indexing

Abstract EN

Dictionaries are fundamental components, typically, of every Natural Language Processing system.

We proposed a hashing-indexing method to speed up looking up process inside dictionaries.

It is a reconstruction of English dictionary of about 300,000 lexical entries using a combination of hash function and an indexing table.

This hash function achieves random access according to tokens prefixes, index table keeps track of where the packet referred by the hash key is starting and how long it is.

The hash function is perfect but not uniform; indexing is based on two levels, both are stated depending on similarity among tokens.