Published May 30, 2025 | Version CC-BY-NC-ND 4.0
Journal article Open

Sign Language Detection and Recognition using Image Processing for Improved Communication

Creators

  • 1. Department of Computer Engineering, St. John College of Engineering and Management, Palghar (Maharashtra), India.

Contributors

Contact person:

  • 1. Department of Computer Engineering, St. John College of Engineering and Management, Palghar (Maharashtra), India.

Description

Abstract: This study presents an advanced deep learning framework for the real-time recognition and translation of Indian Sign Language (ISL). Our approach integrates Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to capture the spatial and temporal features of ISL gestures effectively. The CNN component extracts rich visual features from the input sign language videos, while the LSTM component models the dynamic temporal patterns inherent in the gesture sequences. We evaluated our system using a comprehensive ISL dataset of 700 fully annotated videos representing 100 spoken language sentences. To assess the effectiveness of our approach, we compared two model architectures: CNN-LSTM and SVM-LSTM. The CNN-LSTM model achieved a training accuracy of 84%, demonstrating superior performance in capturing visual and sequential information. In contrast, the SVM-LSTM model achieved a training accuracy of 66%, indicating comparatively lower effectiveness in this context. One of the key challenges faced during the development of the system was overfitting, primarily due to computational constraints and the limited size of the dataset. Nevertheless, the model exhibited promising results through careful tuning of hyperparameters and various optimisation strategies, suggesting its potential for real-world applications. This paper also discusses the data preprocessing techniques employed, including video frame extraction, normalisation, and data augmentation, which were critical in enhancing model performance. By addressing the complexities of sign language recognition, our work advances communication accessibility for individuals relying on ISL, promoting greater inclusivity through technology.

Files

B366815020525.pdf

Files (621.3 kB)

Name Size Download all
md5:8715e93c4c5566152ede2405a7994461
621.3 kB Preview Download

Additional details

Identifiers

DOI
10.35940/ijsce.B3668.15020525/
EISSN
2231-2307

Dates

Accepted
2025-05-15
Manuscript Received on 23 April 2025 | First Revised Manuscript Received on 27 April 2025 | Second Revised Manuscript Received on 04 May 2025 | Manuscript Accepted on 15 May 2025 | Manuscript published on 30 May 2025.

References

  • Z. Hein, T. P. Htoo, B. Aye, S. M. Htet and K. Z. Ye, "Leap Motion based Myanmar Sign Language Recognition using Machine Learning", 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus), St. Petersburg, Moscow, Russia, 2021, pp. 2304-2310, DOI: http://doi.org/10.1109/ElConRus51938.2021.9396496
  • M. J. Hossein and M. Sabbir Ejaz, "Recognition of Bengali Sign Language using Novel Deep Convolutional Neural Network", 2020 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI), Dhaka, Bangladesh, 2020, pp. 1-5, DOI: http://doi.org/10.1109/STI50764.2020.9350418
  • Bekir Aksoy, Osamah Khaled Musleh Salman, Özge Ekrem, "Detection of Turkish Sign Language Using Deep Learning and Image Processing Methods", Taylor & Francis, Applied Artificial Intelligence 2021, VOL. 35, NO. 12, 952–981, DOI: https://doi.org/10.1080/08839514.2021.1982184.
  • Rehab Mustafa Duwairi, Zain Abdullah Halloush, "Automatic recognition of Arabic alphabet sign language using deep learning", International Journal of Electrical and Computer Engineering (IJECE), Vol. 12, No. 3, June 2022, pp. 2996~3004, ISSN: 2088-8708, DOI: http://doi.org/10.11591/ijece.v12i3.pp2996-3004.
  • K. Bantupalli and Y. Xie, "American Sign Language Recognition using Deep Learning and Computer Vision", 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, 2018, pp. 4896-4899, DOI: http://doi.org/10.1109/BigData.2018.8622141
  • Suharjito, H. Gunawan, N. Thiracitta and A. Nugroho, "Sign Language Recognition Using Modified Convolutional Neural Network Model", 2018 Indonesian Association for Pattern Recognition International Conference (INAPR), Jakarta, Indonesia, 2018, pp. 1-5, DOI: http://doi.org/10.1109/INAPR.2018.8627014
  • H. Muthu Mariappan and V. Gomathi, "Real-Time Recognition of Indian Sign Language", 2019 International Conference on Computational Intelligence in Data Science (ICCIDS), Chennai, India, 2019, pp. 1-6, DOI: http://doi.org/10.1109/ICCIDS.2019.8862125
  • J. Joy, K. Balakrishnan and M. Sreeraj, "SignQuiz: A Quiz-Based Tool for Learning Fingerspelled Signs in Indian Sign Language Using ASLR", in IEEE Access, vol. 7, pp. 28363-28371, 2019, DOI: http://doi.org/10.1109/ACCESS.2019.2901863
  • Q. M. Areeb, Maryam, M. Nadeem, R. Alroobaea and F. Anwer, "Helping Hearing-Impaired in Emergency Situations: A Deep Learning-Based Approach", in IEEE Access, vol. 10, pp. 8502-8517, 2022, DOI: http://doi.org/10.1109/ACCESS.2022.3142918
  • G. Jayadeep, N. V. Vishnupriya, V. Venugopal, S. Vishnu and M. Geetha, "Mudra: Convolutional Neural Network based Indian Sign Language Translator for Banks", 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India, 2020, pp. 1228-1232, DOI: http://doi.org/10.1109/ICICCS48265.2020.9121144
  • Anderson, R., Wiryana, F., Ariesta, M. C. & Kusuma, G. P., "Sign language recognition application systems for deaf-mute people: A review based on input-process-output", Procedia Comput—Sci. 116, 441–448, DOI: https://doi.org/10.1016/j.procs.2017.10.028.
  • Garg P, Aggarwal N, Sofat S (2009), "Vision-based hand gesture recognition". World Acad Sci Eng Technol 49(1):972–977, DOI: http://doi.org/10.5281/zenodo.1074855
  • Zamani M, Kanan HR (2014), "Saliency-based alphabet and numbers of American Sign Language recognition using linear feature extraction". In: 4th IEEE International eConference on Computer and Knowledge Engineering (ICCKE), pp 398–403, DOI: http://doi.org/10.1109/ICCKE.2014.6993442
  • Sun C, Zhang T, Xu C (2015), "Latent support vector machine modelling for sign language recognition with Kinect". ACM Trans Intell Syst Technol: TIST 6(2):20, DOI: http://doi.org/10.1145/2629481
  • Aryanie D, Heryadi Y (2015), "American Sign Language-based finger-spelling recognition using k-nearest neighbours' classifier". In: 3rd IEEE international conference on information and communication technology (ICoICT), pp 533–536, DOI: http://doi.org/10.1109/ICoICT.2015.7231481
  • Oyedotun OK, Khashman A (2017), "Deep learning in vision-based static hand gesture recognition". Neural Comput Appl 28(12):3941–3951, DOI: http://doi.org/10.1007/s00521-016-2294-8
  • Adithya V, Vinod PR, Gopalakrishnan U (2013), "Artificial neural network based method for Indian Sign Language recognition". In: IEEE conference on information & communication technologies (ICT), pp 1080–1085, DOI: http://doi.org/10.1109/CICT.2013.6558259
  • Khalafaoui, Y., Grozavu, N., Matei, B., Rogovschi, N. (2024). "Multimodal Machine Learning for Sign Language Prediction". In: Sontea, V., Tiginyanu, I., Railean, S. (eds) 6th International Conference on Nanotechnologies and Biomedical Engineering. ICNBME 2023. IFMBE Proceedings, vol 92. Springer, Cham. DOI: https://doi.org/10.1007/978-3-031-42782-4_26
  • Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2014), "Real-time computer vision-based Bengali Sign Language recognition". In: 17th IEEE International Conference on Computer and Information Technology (ICCIT), pp 192–197, DOI: http://doi.org/10.1109/ICCITechn.2014.7073150
  • R, Elakkiya; B, NATARAJAN (2021), "ISL-CSLTR: Indian Sign Language Dataset for Continuous Sign Language Translation and Recognition", Mendeley Data, V1, DOI: http://doi.org/10.17632/kcmpdxky7p.1
  • Rajesh B. Mapari, Govind Kharat. (2016). Performance Analysis of Different Classifiers for American Sign Language Recognition. International Journal of Soft Computing and Engineering. (Vol. 6, Issue 1, pp. 90–95). https://www.ijsce.org/wp-content/uploads/papers/v6i1/A2789036116.p df
  • Singh, S., Gupta, A. K., & Singh, T. (2019). Sign Language Recognition using Hybrid Neural Networks. In International Journal of Innovative Technology and Exploring Engineering (Vol. 9, Issue 2, pp. 1092–1098). DOI: https://doi.org/10.35940/ijitee.l3349.129219
  • Barde, U., & Ghotkar, A. (2020). Sign Language Recognition: A Survey of Techniques. In International Journal of Recent Technology and Engineering (IJRTE) (Vol. 9, Issue 2, pp. 854–857). DOI: https://doi.org/10.35940/ijrte.b3925.079220
  • R, Renjitha, D, Rubashree, V, Priya, & C, Preetha. (2020). Generating an Audio and Text for Indian Sign Language. In International Journal of Engineering and Advanced Technology (Vol. 9, Issue 3, pp. 4371–4374). DOI: https://doi.org/10.35940/ijeat.c6542.029320
  • M R, Dr. P. (2022). Sign Language Recognition System. In Indian Journal of Software Engineering and Project Management (Vol. 2, Issue 1, pp. 1–3). DOI: https://doi.org/10.54105/ijsepm.c9011.011322