An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

Abstract EN

Sign language translation (SLT) is an important application to bridge the communication gap between deaf and hearing people.

In recent years, the research on the SLT based on neural translation frameworks has attracted wide attention.

Despite the progress, current SLT research is still in the initial stage.

In fact, current systems perform poorly in processing long sign sentences, which often involve long-distance dependencies and require large resource consumption.

To tackle this problem, we propose two explainable adaptations to the traditional neural SLT models using optimized tokenization-related modules.

First, we introduce a frame stream density compression (FSDC) algorithm for detecting and reducing the redundant similar frames, which effectively shortens the long sign sentences without losing information.

Then, we replace the traditional encoder in a neural machine translation (NMT) module with an improved architecture, which incorporates a temporal convolution (T-Conv) unit and a dynamic hierarchical bidirectional GRU (DH-BiGRU) unit sequentially.

The improved component takes the temporal tokenization information into consideration to extract deeper information with reasonable resource consumption.

Our experiments on the RWTH-PHOENIX-Weather 2014T dataset show that the proposed model outperforms the state-of-the-art baseline up to about 1.5+ BLEU-4 score gains.