A deep learning approach for the Romanized Tunisian dialect identification

Abstract EN

Language identification is an important task in natural language processing that consists of determining the language of a given text.

It has increasingly picked the interest of researchers for the past few years, especially for code-switching informal textual content.

This paper, focuses on the identification of the Romanized user-generated Tunisian dialect on the social web.

Segmented and annotated a corpus extracted from social media and propose a deep learning approach for the identification task.

A Bidirectional Long Short-Term Memory neural network with Conditional Random Fields decoding (BLSTM-CRF) had been used.

For word embeddings, a combination of word-character BLSTM vector representation and Fast Text embeddings that takes into consideration character n-gram features.

The overall accuracy obtained is 98.65%.