Enhancement quality and accuracy of speech recognition system using multimodal audio-visual speech signal

Abstract EN

Most developments in speech-based automatic recognition have relied on acoustic speech as the sole input signal, disregarding its visual counterpart.

However, recognition based on acoustic speech alone can be afflicted with deficiencies that prevent its use in many real-world applications, particularly under adverse conditions.

The combination of auditory and visual modalities promises higher recognition accuracy and robustness than can be obtained with a single modality.

Multimodal recognition is therefore acknowledged as a vital component of the next generation of spoken language systems.

This paper aims to build a connected-words audio visual speech recognition system (AV-ASR) for English language that uses both acoustic and visual speech information to improve the recognition performance.

Initially, Mel frequency cepstral coefficients (MFCCs) have been used to extract the audio features from the speech-files.

For the visual counterpart, the Discrete Cosine Transform (DCT) Coefficients have been used to extract the visual feature from the speaker's mouth region and Principle Component Analysis (PCA) have been used for dimensionality reduction purpose.

These features are then concatenated with traditional audio ones, and the resulting features are used for training hidden Markov models (HMMs) parameters using word level acoustic models.

The system has been developed using hidden Markov model toolkit (HTK) that uses hidden Markov models (HMMs) for recognition.

The potential of the suggested approach is demonstrated by a preliminary experiment on the GRID sentence database one of the largest databases available for audio-visual recognition system, which contains continuous English voice commands for a small vocabulary task.

The experimental results show that the proposed Audio Video Speech Recognizer (AV-ASR) system exhibits higher recognition rate in comparison to an audio-only recognizer as well as it indicates robust performance.

An increase of success rate by 4% for the grammar based word recognition system overall speakers is achieved for speaker independent test.