To read this content please select one of the options below:

An approach based on deep learning for Indian sign language translation

Kinjal Bhargavkumar Mistree (Chhotubhai Gopalbhai Patel Institute of Technology, Uka Tarsadia University, Bardoli, India)
Devendra Thakor (Chhotubhai Gopalbhai Patel Institute of Technology, Uka Tarsadia University, Bardoli, India)
Brijesh Bhatt (Dharmsinh Desai Institute of Technology, Dharmsinh Desai University, Nadiad, India)

International Journal of Intelligent Computing and Cybernetics

ISSN: 1756-378X

Article publication date: 16 December 2022

Issue publication date: 12 July 2023

110

Abstract

Purpose

According to the Indian Sign Language Research and Training Centre (ISLRTC), India has approximately 300 certified human interpreters to help people with hearing loss. This paper aims to address the issue of Indian Sign Language (ISL) sentence recognition and translation into semantically equivalent English text in a signer-independent mode.

Design/methodology/approach

This study presents an approach that translates ISL sentences into English text using the MobileNetV2 model and Neural Machine Translation (NMT). The authors have created an ISL corpus from the Brown corpus using ISL grammar rules to perform machine translation. The authors’ approach converts ISL videos of the newly created dataset into ISL gloss sequences using the MobileNetV2 model and the recognized ISL gloss sequence is then fed to a machine translation module that generates an English sentence for each ISL sentence.

Findings

As per the experimental results, pretrained MobileNetV2 model was proven the best-suited model for the recognition of ISL sentences and NMT provided better results than Statistical Machine Translation (SMT) to convert ISL text into English text. The automatic and human evaluation of the proposed approach yielded accuracies of 83.3 and 86.1%, respectively.

Research limitations/implications

It can be seen that the neural machine translation systems produced translations with repetitions of other translated words, strange translations when the total number of words per sentence is increased and one or more unexpected terms that had no relation to the source text on occasion. The most common type of error is the mistranslation of places, numbers and dates. Although this has little effect on the overall structure of the translated sentence, it indicates that the embedding learned for these few words could be improved.

Originality/value

Sign language recognition and translation is a crucial step toward improving communication between the deaf and the rest of society. Because of the shortage of human interpreters, an alternative approach is desired to help people achieve smooth communication with the Deaf. To motivate research in this field, the authors generated an ISL corpus of 13,720 sentences and a video dataset of 47,880 ISL videos. As there is no public dataset available for ISl videos incorporating signs released by ISLRTC, the authors created a new video dataset and ISL corpus.

Keywords

Acknowledgements

Ethical statement: All participants (parent and legal guardian in the case of children under 16) gave written informed consent to participate in the study. Consent was given for publication by all participants (parent and legal guardian in the case of children under 16).

Plain language summary: The work described in this paper is an important step forward in this domain as we created (1) a dataset of 47,880 ISL videos of 13,720 ISL sentences and (2) an ISL gloss corpus from the Brown corpus to promote research in ISL gloss to translation in text and speech of natural languages.

Citation

Mistree, K.B., Thakor, D. and Bhatt, B. (2023), "An approach based on deep learning for Indian sign language translation", International Journal of Intelligent Computing and Cybernetics, Vol. 16 No. 3, pp. 397-419. https://doi.org/10.1108/IJICC-08-2022-0227

Publisher

:

Emerald Publishing Limited

Copyright © 2022, Emerald Publishing Limited

Related articles