Real-time Macedonian Sign Language Recognition System by using Transfer Learning

Abstract

The objective for developing a real-time sign language recognition system is twofold: improving inter-personal communication and supporting inclusive human-computer interaction with hearing-impaired population using a particular sign language. This study describes the design and implementation of a system for real-time Macedonian Sign Language recognition in images and videos. A robust and lightweight model was proposed based on transfer learning of suitable pretrained architectures, namely, Single Shot Detector (SSD) MobileNetV2 and SSD MobileNetV2 FPNLite. The proposed models were fine-tuned and extensively evaluated in a number of diverse scenarios to account for the inherent difficulties in recognizing particular letters. In the absence of publicly available dataset, we have created a dataset consisting of two-handed images of 28 out of 31 letters of the Macedonian alphabet; the three letters expressed by dynamic gestures were excluded from the study. The results point out to a state-of-the-art prediction accuracy on the classification task of Macedonian sign language alphabet.