Multimodal emotion recognition for edge devices

This thesis presents a scalable, low-cost, and privacy-conscious framework for real-time
emotion recognition using Internet of Things (IoT) technologies and machine learning.
Traditional systems often rely on high-cost, invasive hardware and are typically limited to
controlled environments. In contrast, this research demonstrates that high-performance
emotion classification can be achieved using lightweight models and aordable sensors
suitable for deployment on edge devices.
Three primary objectives were addressed: the evaluation of unimodal classiers across
text, speech, and thermal image data; the development of a reproducible data collection
framework using low-cost thermal imaging hardware; and the implementation of a realtime
Multimodal Emotion Recognition system optimised for embedded platforms. A novel
thermal facial expression dataset was introduced, collected using an IoT-based thermal
camera across six emotions and five national backgrounds. For textual emotion recognition,
traditional machine learning models combined with Term Frequency-Inverse Document
Frequency features achieved up to 90% accuracy, outperforming transformer-based
models such as Distilled Bidirectional Encoder Representations from Transformers in both
eciency and latency when deployed on edge hardware. In the thermal domain, a modied
Vision Transformer achieved 96% classification accuracy, surpassing Residual Network in
both accuracy and generalisation across hybrid datasets. The final Multimodal Emotion
Recognition system, combining thermal, speech, and text modalities using both featurelevel
and decision-level fusion, demonstrated robust real-time performance. Decision-level
fusion achieved up to 99% accuracy on structured datasets, with inference latency below
0.8 seconds and throughput exceeding 4,200 samples per second on a Raspberry Pi 4B.
These results confirm that accurate, real-time emotion recognition is achievable on
affordable hardware without compromising performance. The system developed in this
research offers a practical, deployable framework for emotion-aware computing in realworld
applications such as healthcare, education, and human-computer interaction.