Published May 24, 2023 | Version v1

Lip Reading and Reconstruction using ML

  • 1. Amal Jyothi College of Engineering, Kanjirapally

Description

Lip reading is a technique of comprehension of speech through visual interpretation of lip movements. Although lip reading is most often used by people who are deaf or hard of hearing, most people with normal hearing process some voice information from the sight of the moving mouth. In addition, understanding the language cues of lip readings can enhance the clarity of conversation in noisy environments. This paper proposes a model that identifies the impact of intermodal self- monitoring for speech reconstruction (video-audio) by taking advantage of the natural occurrence of audio and visual streams in videos. The model that has an autoregressive encoder-decoder with an attention architecture, to map directly the sequences of silent facial movements to mel-scale spectrograms for speech reconstruction, which requires no human annotation.

Files

NACORE_ P223.pdf

Files (462.9 kB)

Name Size
md5:5f9f0b82a0157a52429567a5385a5ffe
462.9 kB Preview Download