Mobile Phone-Based Audio Announcement Detection and Recognition for People with Hearing Impairment

Abstract EN

Automatic audio announcement systems are widely used in public places such as transportation vehicles and facilities, hospitals, and banks.

However, these systems cannot be used by people with hearing impairment.

That brings great inconvenience to their lives.

In this paper, an approach of audio announcement detection and recognition for the hearing-impaired people based on the smart phone is proposed and a mobile phone application (app) is developed, taking the bank as a major applying scenario.

Using the app, the users can sign up alerts for their numbers and then the system begins to detect audio announcements using the microphone on the smart phone.

For each audio announcement detected, the speech within it is recognized and the text is displayed on the screen of the phone.

When the number the user input is announced, alert will be given by vibration.

For audio announcement detection, a method based on audio segment classification and postprocessing is proposed, which uses a SVM classifier trained on audio announcements and environment noise collected in banks.

For announcement speech recognition, an ASR engine is developed using a GMM-HMM-based acoustic model and a finite state transducer (FST) based grammar.

The acoustic model is trained on audio announcement speech collected in banks, and the grammar is human-defined according to the patterns used by the automatic audio announcement systems.

Experimental results show that character error rates (CERs) around 5% can be achieved for the announcement speech, which shows feasibility of the proposed method and system.