Published March 30, 2024 | Version CC BY-NC-ND 4.0
Journal article Open

Deep Learning for Real-time Affective Hand Gesture Recognition in EMASPEL

Creators

  • 1. Department of Information Systems, Faculty of Computing and Information Technology, Northern Border University, Rafha, Saudi Arabia.

Contributors

Contact person:

  • 1. Department of Information Systems, Faculty of Computing and Information Technology, Northern Border University, Rafha, Saudi Arabia.
  • 2. Department of Computer Sciences Faculty of Computing and Information Technology, Northern Border University, Rafha, Saudi Arabia, National Engineering School of Tunis, LR11ES20 Analysis Design and Control of Systems Laboratory, University of Tunis El Manar, Tunis, Tunisia.
  • 3. Department of Computer Sciences Faculty of Computing and Information Technology, Northern Border University, Rafha, Saudi Arabia.

Description

Abstract: This research marks a transformative leap in personalized learning through real-time affective hand gesture recognition in EMASPEL (Emotional Multi-Agents System for Peer-to-peer E-Learning), an educational platform. Our deep learning model, a meticulously crafted ensemble of convolutional and recurrent neural networks, deciphers the unspoken language of emotions embedded within student gestures, accurately capturing both spatial and temporal patterns. This detailed emotional map empowers EMASPEL to tailor its interactions with exquisite precision, addressing frustration, nurturing curiosity, and maximizing student engagement. The impact is profound: students flourish in personalized learning environments, experiencing enhanced outcomes and a newfound connection to their educational journey. Teachers, equipped with real-time emotional insights, provide targeted support and cultivate a more inclusive, responsive classroom. Beyond gestures, we envision a future enriched by multimodal data integration, encompassing facial expressions, voice analysis, and potentially physiological sensors, to paint even richer portraits of student emotions and cognitive states. Continuous refinement through rigorous longitudinal studies will pave the way for deeper understanding and ensure responsible implementation. Ultimately, this research reimagines education as a dynamic ensemble of personalized learning, where technology serves as a bridge between teacher and student, unlocking not just academic success but a lifelong love of knowledge.

Files

F801212060324.pdf

Files (598.1 kB)

Name Size Download all
md5:78a609f4dca1d90b5a0a28310ae39a41
598.1 kB Preview Download

Additional details

Identifiers

EISSN
2277-3878
DOI
10.35940/ijrte.F8012.12060324

Dates

Accepted
2024-03-15
Manuscript received on 16 February 2024| Revised Manuscript received on 21 February 2024 | Manuscript Accepted on 15 March 2024 | Manuscript published on 30 March 2024

References

  • Alpaydin, E. (2022). Introduction to machine learning (4th ed.). MIT Press.
  • Ben Ammar, M., Neji, M., ALIMI, M.A and Gouardères, G. (2010) ''The Affective Tutoring System'', Expert Systems with Applications 37 3013-3023. Elsevier. https://doi.org/10.1016/j.eswa.2009.09.031
  • Cao, C., Xu, D., Yang, Z., & Wang, R. (2020). Real-time hand gesture recognition for human-computer interaction with a multi-scale convolutional neural network. Sensors, 20(13), 3705. doi:10.3390/s20133705 https://doi.org/10.3390/s20133705
  • Ekman, P. (2009). Unmasking the face: A guide to recognizing emotions from facial expressions (Rev. and updated ed.). Malor Books.
  • Fernández-Rodríguez, A., Martinez-Gonzalez, A., & GonzalezMancera, A. (2023). A review of affective computing applications in education. Sensors, 23(10), 5957. doi:10.3390/s23105957
  • Galati, S., Martinez, A. M., & Nobile, A. (2020). Real-time affective hand gesture recognition using a hybrid architecture. Pattern Recognition Letters, 139, 182-192. doi:10.1016/j.patrec.2020.07.022 https://doi.org/10.1016/j.patrec.2020.07.022
  • Hegde, R. S., & Kavita, T. (2022). Affective gesture recognition for educational systems: A review. International Journal of Computer Applications, 15, 1-6. doi:10.24093/ijca.1186
  • Kshirsagar, S., & Kulkarni, U. G. (2023). Real-time hand gesture recognition using computer vision for human-computer interaction: A survey. International Journal of Advanced Research in Computer Science and Software Engineering, 13(3), 876-884.
  • Li, H., Liu, Y., Wu, X., & Li, R. (2022). Real-time hand gesture recognition using transfer learning with convolutional neural networks. Electronics, 11(12), 2005. doi:10.3390/electronics11122005
  • Lopez-Martinez, A., Galati, S., Martinez, A. M., & Nobile, A. (2022). Affective human-computer interaction through real-time hand gesture recognition with dynamic neural networks. Neural Processing Letters, 48(2), 703-723. doi:10.1007/s11974-021-02805-0
  • Lu, M., Zhang, W., & Liu, Z. (2023). A survey of recent advances in hand gesture recognition. Pattern Recognition Letters, 173, 192-209. doi:10.1016/j.patrec.2023.08.004 https://doi.org/10.1016/j.patrec.2023.08.004
  • Meyer, K., Lüke, B., & Kunzendorf, E. (2022). Cultural differences in nonverbal communication and gestures: Implications for the design of gesture-based human-computer interaction systems. HumanComputer Interaction, 37(4), 1098-1142. doi:10.1080/07357206.2021.1930574
  • Pantic, M., & Rothkrantz, L. J. M. (2007). Affective computing for measuring user experience and emotions. User Modeling and User-Adapted Interaction, 16(3-4), 345-391. doi:10.1007/s11257-007- 9074-4
  • Pfau-Gray, C., Muntele, I., & Brössert, C. (2023). Affective gesture recognition in educational software: Exploring the potential of embodied learning with children. Frontiers in Computer Science, 7, 815791. doi:10.3389/fcomp.2023.815791
  • Queirolo, D., De Salvo, G., Greco, A., & Pisani, C. (2023). Deep learning for robust hand gesture recognition: A survey. Robotics and Autonomous Systems, 170, 103600. doi:10.1016/j.robot.2023.103600
  • Ruiz-Garcia, P., Sanchez-Casado, P., Fernandez-Moral, J. L., & Madrid Morales, V. (2023). Affective learning with intelligent tutoring systems: A review of the state of the art. Applied Sciences, 13(23), 12202. doi:10.3390/app132312202
  • Jaafar, J., Yusof, H. M., Hassan, S., Adtrudin, K. F., & Ahmad, R. (2019). Nexus between Emotional Intelligence (EQ-I) and Entrepreneurial Culture. In International Journal of Engineering and Advanced Technology (Vol. 8, Issue 6s3, pp. 986–992). https://doi.org/10.35940/ijeat.f1093.0986s319
  • Srinivasa, M. S., & Vijayashree, D. L. (2021). A Study on Impact of Sensitive Intelligence and Perceived Stress. In International Journal of Management and Humanities (Vol. 5, Issue 7, pp. 27–29). https://doi.org/10.35940/ijmh.g1257.035721
  • Srividya, M. S., & R, Dr. A. M. (2020). Research trends in Hand Gesture Recognition techniques. In International Journal of Recent Technology and Engineering (IJRTE) (Vol. 8, Issue 6, pp. 1059– 1064). https://doi.org/10.35940/ijrte.f7519.038620
  • Kaur, S., & Bhatla, Er. N. (2019). An Efficient Gesture Recognition with ABC-ANN Classification and Key-Point Features Extraction for Hand Images. In International Journal of Innovative Technology and Exploring Engineering (Vol. 8, Issue 10, pp. 3193–3199). https://doi.org/10.35940/ijitee.j1153.0881019
  • M R, Dr. P. (2022). Sign Language Recognition System. In Indian Journal of Software Engineering and Project Management (Vol. 2, Issue 1, pp. 1–3). https://doi.org/10.54105/ijsepm.c9011.011322