Multimodal, Multi-Class Bias Mitigation for Predicting Speaker Confidence

Projecting confidence during conversation or presentation is a critical skill. To effectively display confidence, speakers must employ a blend of verbal and non-verbal signals. A predictive model that leverages rich multimodal cues to measure a speaker¿½fs confidence must also mitigate biases that develop through data labelling practices, inherent imbalances in the demographic distribution, or biases introduced into the model during the training process. Fairly predicting the confidence of speakers across differing backgrounds enables more accurate and actionable feedback to a larger population of speakers. This paper introduces a set of approaches for bias mitigation for multimodal, multi-class confidence prediction of adult speakers in a work-like setting. We evaluate the extent to which bias mitigation techniques improve the performance of a multimodal confidence classifier with a dataset of 233 2-minute videos. Experimental results suggest that by bounding the loss across perceived races, genders, accents, and ages, multimodal models can significantly outperform unmitigated baselines. The implications, including automated feedback of speaker confidence, are discussed.