Automated Arabic sign language recognition using neural networks

مقدم أطروحة جامعية

Mashaqibah, Iman Faris

مشرف أطروحة جامعية

Kanan, Ghassan Jaddu

أعضاء اللجنة

al-Shaykh, Isam
al-Shalabi, Riyad
al-Dabbas, Umar Ṣuhaib

الجامعة

الأكاديمية العربية للعلوم المالية و المصرفية

الكلية

كلية نظم و تكنولوجيا المعلومات

القسم الأكاديمي

قسم نظم المعلومات الحاسوبية

دولة الجامعة

الأردن

الدرجة العلمية

دكتوراه

تاريخ الدرجة العلمية

2010

الملخص الإنجليزي

Automated recognition of sign language greatly facilitates communication between hearing–impaired people and hearing people; and it is an obvious substitute for such mean of keyboards, screens and speech in the communication process for the application of sign language It is also important for the development of humanmachine interface.

This thesis proposes a framework for automatically recognizing Arabic sign language for Jordanian accent except for number and letters which are the standard sign adopted by all Arabic speaking countries.

Using an image processing extract the motion regions of head and hand regions, The movements of hand regions contain semantic meanings for certain gestures while the head region in a frame is used as a reference point to describe hand locations.

This thesis uses the motion trajectories of the hands to recognize ArSL signs.

The user wears gloves with different colors when performing the signs.

The two colored regions are detected and marked as separate components.

Segmentation image of three colored regions and outlier according to the mean and covariance of each color region, toward this end transforming the red, green and blue (RGB) image to HSI where hue (H) an angular value, saturation(S) a value for the amount of color present and intensity (I).We can use hue to uniquely identify color segments to reduce the critical color dimension from two to one and gain additional color label knowledge.

When applying the multivariate Gaussian mixture the characteristics of the HSI color space can be used.

For a given video sequence, in this thesis, we make a list of the position of the centroid for each of the right hand, left hand, and face in each frame.

Tracking the hand motion trajectories of the right hand and left hand over time is fairly simple.

we make a list of features, the position of the centriod for each of the right hand and left hand using head as references, the horizontal and vertical velocity of both hands across the two frames using change in position over time, and angle of velocity of both hands.

To achieve a high recognition rate, we use time delay neural networks (TDNN) as a classification algorithm, since TDNNs have been demonstrated to be very successful in learning spatio-temporal patterns.

For comparisons purposes, we compare five categories of sign language which become all possible sign word, and we conclude that the training TDNN time is due to VII the number of used features which are assigned to each sign word ,the larger of training sets in each category, and the TDNN which need longer time to training.

Recognition rate in training set is 100 %.

Recognition rate in testing set is 75.48 %.

Results were obtained using the approach presented by this thesis All experiments and evaluation are performed by using our own Arabic sign language ArSL video database.

ArSL still needs further research to gain better performance.

التخصصات الرئيسية

تكنولوجيا المعلومات وعلم الحاسوب

الموضوعات

• معالجة البيانات
• اللسانيات الحاسوبية
• اللغة العربية
• الشبكات العصبية(الحاسبات الإلكترونية)

عدد الصفحات

140

قائمة المحتويات

• APA
• MLA
• AMA

لغة النص

الإنجليزية

نوع البيانات

رسائل جامعية

رقم السجل

BIM-307258