Hybrid FiST-CNN approach for feature extraction for vision-based Indian sign language recognition

Abstract EN

Indian Sign Language (ISL) is the commonly used language by the deaf-mute community in the Indian continent.

Effective feature extraction is essential for the automatic recognition of gestures.

This paper aims at developing an efficient feature extraction technique using Features from Fast Accelerated Segment Test (FAST), Scale-Invariant Feature Transformation (SIFT), and Convolution Neural Networks (CNN).

FAST with SIFT are used to detect and compute features, respectively.

CNN is used for classification with the hybridization of FAST-SIFT features.

The system is implemented and tested using the python-based library Keras.

The results of the proposed techniques have been tested on 34 gestures of ISL (24 alphabets set and 10 digit sets) and then compared with the CNN and SIFT_CNN, and it is also tested on two publicly available datasets on Jochen Trisech Dataset (JTD) and NUS-II dataset.

The proposed study outperformed some existing ISLR works with an accuracy of 97.89%, 95.68%, 94.90% and 95.87% for ISL-alphabets, MNIST, JTD and NUS-II, respectively.