Published July 7, 2026 | Version v1
Journal article Open

DIY AI Smart Glass: Design, Development, and Evaluation of an Indigenous Multimodal Wearable Assistant

Authors/Creators

  • 1. Khyati World School, Ahmedabad, India.
  • 2. Assistant Professor, School of Design, Anant National University, Ahmedabad Gujarat.

Description

ABSTRACT

The rapid advancement of multimodal artificial intelligence has accelerated the development of wearable smart assistants capable of integrating vision, speech, and contextual reasoning. However, commercially available AI smart glasses remain expensive, proprietary, and difficult to customize, limiting their accessibility for education, research, and assistive technology development, particularly in resource-constrained environments. This study presents DIY AI Smart Glass, an indigenous low-cost multimodal wearable assistant designed to provide an open-source alternative for real-time human–computer interaction. The primary aim of this research is to develop and evaluate an affordable wearable AI platform capable of understanding spoken language, interpreting visual information, and generating contextual voice responses through an integrated multimodal framework. The proposed system employs an ESP32-S3 AI camera module equipped with an OV3660 camera, PDM microphone, and inbuilt speaker, while computationally intensive multimodal inference is performed on a host computer using a Python-based architecture integrated with a cloud-based large multimodal language model. The methodology incorporates wireless acquisition of audio and visual data, wake-word activation, speech understanding, scene interpretation, contextual reasoning, and real-time speech synthesis, enabling natural hands-free interaction through a single wearable device. Unlike conventional wearable vision systems that primarily perform object detection, the proposed framework supports both visual scene understanding and general conversational assistance within a unified architecture. Experimental validation was conducted across representative visual and non-visual interaction scenarios to evaluate functionality, response generation, and user experience. The developed prototype demonstrates reliable multimodal interaction while maintaining a significantly lower hardware cost than commercial smart glasses, thereby providing an accessible platform for assistive technologies, educational robotics, wearable artificial intelligence research, and rapid prototyping of next-generation human-centred intelligent systems.

Keywords: Multimodal Artificial Intelligence; Wearable Smart Glasses; Human–Computer Interaction; ESP32-S3; Large Language Models (LLMs).

Files

MSIJAT472026 GS.pdf

Files (1.5 MB)

Name Size Download all
md5:aafd80a225211b4539bba5da393e132f
1.5 MB Preview Download

Additional details

Software

Repository URL
https://msipublishers.com/wp-content/uploads/2026/07/MSIJAT472026-GS.pdf

References

  • Aloui, N. (2025). Towards spatial awareness: Real-time sensory augmentation with smart glasses for visually impaired individuals. Electronics, 14(17), Article 3365. https://doi.org/10.3390/electronics14173365
  • Busaeed, S., Mehmood, R., Katib, I., & Corchado, J. M. (2022). LidSonic for visually impaired: Green machine learning-based assistive smart glasses with smart app and Arduino. Electronics, 11(7), Article 1076. https://doi.org/10.3390/electronics11071076
  • Chinnapparaj, S., Mahalakshmi, E., Nadhiraj, S., & Bharathkumar, K. (2025). AIbased smart detection and vehicle control system using ESP32 and IoT. In 2025 4th International Conference on Automation, Computing and Renewable Systems (ICACRS) (pp. 66–72). IEEE. https://doi.org/10.1109/ICACRS67045.2025.11324027
  • Choudhary, S., Dhote, N., Deshpande, A. A., Sambhariya, A., & Joshi, P. K. (2023). IoT based smart glasses with facial recognition for people with visual impairments. SSRG International Journal of Electrical and Electronics Engineering, 10(9), 154– 159. https://doi.org/10.14445/23488379/IJEEE-V10I9P114
  • Gamage, B. (2024). AI-enabled smart glasses for people with severe vision impairments. ACM SIGACCESS Accessibility and Computing, (137), 1–1. https://doi.org/10.1145/3654768.3654771
  • Gohil, P. A. P. J. A., & Trivedi, R. R. (2022). Development of a remotely operated 3D printed robotic hand using electromyography. AIP Conference Proceedings, 2946(1), Article 0178508. https://doi.org/10.1063/5.0178508
  • Jeong, I., Kim, K., Jung, J., & Cho, J. (2025). YOLOv8-based XR smart glasses mobility assistive system for aiding outdoor walking of visually impaired individuals in South Korea. Electronics, 14(3), Article 0425. https://doi.org/10.3390/electronics14030425
  • Kannan, R., Shah, P., & Parikh, P. (2025). Design, simulation, and walking pattern analysis of a prosthetic leg made from recycled PET plastic for sustainable and affordable healthcare solutions. Indian Journal of Applied Research, 15(3), 45–48. https://doi.org/10.36106/ijar/0307952
  • Litayem, N. (2024). Scalable smart home management with ESP32-S3: A low-cost solution for accessible home automation. In 2024 International Conference on Computer and Applications (ICCA) (pp. 1–7). IEEE. https://doi.org/10.1109/ICCA62237.2024.10927887
  • Liu, Y., Shah, P. S., Xia, T., & Huston, D. (2025). An approach to thermal management and performance throttling for federated computation on a low-cost 3D ESP32-S3 package stack. Computers, 14(4), Article 147. https://doi.org/10.3390/computers14040147
  • Makhiddinov, M., & Cho, J. (2021). Smart glass system using deep learning for the blind and visually impaired. Electronics, 10(22), 1–30. https://doi.org/10.3390/electronics10222779
  • Moosmann, J., et al. (2025). Ultra-efficient on-device object detection on AIintegrated smart glasses with TinyissimoYOLO. In Artificial Intelligence – ICAI 2025 (pp. 262–280). Springer. https://doi.org/10.1007/978-3-031-91989-3_17
  • M, T., P, M., P, T., P, I. K., S, K., & S, K. (2026). Tinyface algorithm-based facial detection and behavior analysis for military checkpoints ESP32-S3-CAM AI vision module. In 2026 4th International Conference on Artificial Intelligence and Machine Learning Applications (AIMLA) (pp. 1–5). IEEE. https://doi.org/10.1109/AIMLA67915.2026.11522556
  • Nemlaha, E., Střelec, P., Horák, T., Kováč, S., & Tanuška, P. (2023). Suitability of MQTT and REST communication protocols for AIoT or IIoT devices based on ESP32 S3. In Advanced Hybrid Information Processing (pp. 225–233). Springer. https://doi.org/10.1007/978-3-031-21435-6_19
  • Nugroho, W., Arifianto, M. J. F., Afianto, A., Wicaksono, A., & Nursim, N. (2025). Web based IoT monitoring system for ultrasonic water flow measurement using Page 36 of 37 DOI: 10.5281/zenodo.21233518 ESP32-S3 and cloud database. Journal of Soft Computing Exploration, 6(4), 258–265. https://doi.org/10.52465/joscex.v6i4.625
  • Parikh, P., Shah, H., Vaghela, P., & Kankrecha, S. (n.d.). Design and development of a tangible machine learning-based object recognition and learning system for Indian kids suffering from Visual Agnosia [Unpublished manuscript].
  • Parikh, P., Srivastava, V., Joshi, K. D., Shah, M. A., & Sharma, A. (2026). A smart glove for Parkinson's severity detection via low-frequency motion features and ondevice machine learning. Journal of Low Frequency Noise, Vibration and Active Control. Advance online publication. https://doi.org/10.1177/14613484251414903
  • Patel, A., Parikh, P., & Shah, P. (2025). Design and development of a tangible and digital lungs exerciser for the athletes. Indian Journal of Applied Research, 15(3). https://doi.org/10.36106/ijar/7208051
  • Rossos, D., Mihailidis, A., & Laschowski, B. (2024). AI-powered smart glasses for sensing and recognition of human-robot walking environments. In 2024 10th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob) (pp. 62–67). IEEE. https://doi.org/10.1109/BioRob60516.2024.10719768
  • Salvi, S., Pahar, S., & Kadale, Y. (2021). Smart glass using IoT and machine learning technologies to aid the blind, dumb and deaf. Journal of Physics: Conference Series, 1804(1), Article 012181. https://doi.org/10.1088/1742-6596/1804/1/012181
  • Udayakumar, D., et al. (2025). Artificial intelligence-powered smart vision glasses for the visually impaired. Indian Journal of Ophthalmology, 73(Suppl 3), S492–S497. https://doi.org/10.4103/IJO.IJO_1621_24
  • Waisberg, E., Ong, J., Paladugu, P., Kamran, S. A., Zaman, N., Sarker, P., Lee, A. G., & Tavakkoli, A. (2024). Meta smart glasses—large language models and the future for assistive glasses for individuals with vision impairments. Eye, 38(6), 1036–1038. https://doi.org/10.1038/s41433-023-02842-z
  • Wang, B., et al. (2025). A systematic literature review on integrating AI-powered smart glasses into digital health management for proactive healthcare solutions. NPJ Digital Medicine, 8(1), Article 1715. https://doi.org/10.1038/s41746-025-01715-x
  • Zhang, D., Y., Li, Z., He, & Li, X. (2024). Empowering smart glasses with large language models: Towards ubiquitous AGI. In UbiComp Companion 2024 - Companion of the 2024 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 631–633). Association for Computing Machinery. https://doi.org/10.1145/3675094.3678992