Automated Arabic sign language recognition using neural networks

Dissertant

Mashaqibah, Iman Faris

Thesis advisor

Kanan, Ghassan Jaddu

Comitee Members

al-Shaykh, Isam
al-Shalabi, Riyad
al-Dabbas, Umar Ṣuhaib

University

Arab Academy for Financial and Banking Sciences

Faculty

The Faculty of Information Systems and Technology

Department

Computer information systems

University Country

Jordan

Degree

Ph.D.

Degree Date

2010

English Abstract

Automated recognition of sign language greatly facilitates communication between hearing–impaired people and hearing people; and it is an obvious substitute for such mean of keyboards, screens and speech in the communication process for the application of sign language It is also important for the development of humanmachine interface.

This thesis proposes a framework for automatically recognizing Arabic sign language for Jordanian accent except for number and letters which are the standard sign adopted by all Arabic speaking countries.

Using an image processing extract the motion regions of head and hand regions, The movements of hand regions contain semantic meanings for certain gestures while the head region in a frame is used as a reference point to describe hand locations.

This thesis uses the motion trajectories of the hands to recognize ArSL signs.

The user wears gloves with different colors when performing the signs.

The two colored regions are detected and marked as separate components.

Segmentation image of three colored regions and outlier according to the mean and covariance of each color region, toward this end transforming the red, green and blue (RGB) image to HSI where hue (H) an angular value, saturation(S) a value for the amount of color present and intensity (I).We can use hue to uniquely identify color segments to reduce the critical color dimension from two to one and gain additional color label knowledge.

When applying the multivariate Gaussian mixture the characteristics of the HSI color space can be used.

For a given video sequence, in this thesis, we make a list of the position of the centroid for each of the right hand, left hand, and face in each frame.

Tracking the hand motion trajectories of the right hand and left hand over time is fairly simple.

we make a list of features, the position of the centriod for each of the right hand and left hand using head as references, the horizontal and vertical velocity of both hands across the two frames using change in position over time, and angle of velocity of both hands.

To achieve a high recognition rate, we use time delay neural networks (TDNN) as a classification algorithm, since TDNNs have been demonstrated to be very successful in learning spatio-temporal patterns.

For comparisons purposes, we compare five categories of sign language which become all possible sign word, and we conclude that the training TDNN time is due to VII the number of used features which are assigned to each sign word ,the larger of training sets in each category, and the TDNN which need longer time to training.

Recognition rate in training set is 100 %.

Recognition rate in testing set is 75.48 %.

Results were obtained using the approach presented by this thesis All experiments and evaluation are performed by using our own Arabic sign language ArSL video database.

ArSL still needs further research to gain better performance.

Main Subjects

Information Technology and Computer Science

Topics

• Data processing
• Computational linguistics
• Arabic language
• Neural networks(Computer science)

No. of Pages

140

Table of Contents

• APA
• MLA
• AMA

Language

English

Data Type

Arab Theses

Record ID

BIM-307258