ThreeLS. Simple live speech-driven lip-sync
- 1. Universitat Pompeu Fabra
- 2. University of Oldenburg
Description
This repository contains the Unity Package, the javascript algorithm, and the scientific paper of ThreeLS. ThreeLS is a very simplistic lipsyncing that takes a very short time to set up. It uses three blend shapes for the lips: kiss, lips closed or mouth open. In terms of Facial Action Units: AU22, AU24, and AU27. It works in real-time using speech audio files or an audio stream (no need to transcribe text or phonemes). The technology is based on the differences between the spectrum of vowels and consonants.
A technical explanation and demos of it can be seen in this video (link starting at the demos): https://youtu.be/89pBiGKXpZI?t=43s. A video tutorial on how to set it up in Unity is included in the package. The video tutorial can also be seen here: https://youtu.be/wV8UIoUWIc8
Github repository (javascript only):
https://github.com/gerardllorach/threelipsync
More info and live demos:
https://gerardllorach.weebly.com/
References:
Llorach, G., Evans, A., Blat, J., Grimm, G. and Hohmann, V., 2016, September. Web-based live speech-driven lip-sync. In 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES) (pp. 1-4). IEEE.