Towards Emotion Recognition with 3D Pointclouds Obtained from Facial Expression Images

Facial expression-based Emotion Recognition (FER) is a critical research area within Affective Computing (AC) due to its wide-ranging applications in Human Computer Interaction (HCI), as well as its potential use in mental health assessment and fatigue monitoring. However, current FER methods predominantly rely on Deep Learning (DL) techniques trained on 2-Dimensional (2D) image data, which pose significant privacy concerns and are unsuitable for continuous, real-time monitoring. As an alternative, we propose High-Frequency Wireless Sensing (HFWS) as an enabler of continuous, privacy-aware FER, through the generation of detailed 3-Dimensional (3D) facial pointclouds via on-person sensors embedded in wearables. We present arguments supporting the privacy advantages of HFWS over traditional 2D imaging approaches, particularly under increasingly stringent data protection regulations. A major barrier to adopting HFWS for FER is the scarcity of labeled 3D FER datasets. Towards addressing this issue, we introduce a method based on the Faces Learned with an Articulated Model and Expressions (FLAME) model to generate 3D facial pointclouds from existing public 2D datasets. Using this approach, we create AffectNet3D, a 3D version of the AffectNet database. To evaluate the quality and usability of the generated data, we design a pointcloud refinement pipeline focused on isolating the facial region, and train the popular PointNet++ model on the refined pointclouds. Fine-tuning the model on a small subset of the unseen 3D FER dataset BU-3DFE yields a classification accuracy exceeding 70%, comparable to oracle-level performance. To further investigate the potential of HFWS-based FER for continuous monitoring, we simulate wearable sensing conditions by masking portions of the generated pointclouds. Experimental results show that models trained on AffectNet3D and fine-tuned with just 25% of BU-3DFE significantly outperform those trained solely on BU-3DFE. These findings highlight the viability of our data generation pipeline and support the feasibility of continuous, privacy-aware FER via wearable HFWS systems.