Executive Summary

DeepFakes, synthetic manipulations of faces produced with generative artificial intelligence, threaten the authenticity of content and expose detectors to the tough task of dealing with a multiplicity of content and great variability, compression levels and deepfake generation pipelines. Against this backdrop, this doctoral thesis investigates how errors made by DeepFake detectors relate to high-level facial attributes, and how this knowledge can be used to design more robust and interpretable systems. The work is positioned in computer vision for face forensics and addresses a central challenge in the field, namely the lack of generalization across manipulation families, compression levels, and capture conditions. The thesis contributes a scalable semi-supervised pipeline for attribute labeling, an empirical analysis that links false positives and false negatives to specific appearance factors, and a set of attribute-aware training strategies that improve reliability without sacrificing transparency. While the manuscript surveys the state of the art and reports several educational and industrial activities, the scientific core is concentrated in Chapter 4, Analysis of DF Detection through Attribute Labeling, and Chapter 5, Attribute-Aware Training Strategies. This summary emphasizes those two chapters while outlining the end-to-end approach and the resulting guidelines.

Introduction: Problem framing and research questions.

DeepFake generation has progressed quickly, which reduces the saliency of traditional visual artifacts and stresses detectors that were tuned for early datasets. As new pipelines and editing chains proliferate, model failures concentrate on specific slices of the data, for example under certain occlusions or appearance conditions. The thesis therefore asks three questions. RQ1 concerns interpretability, namely whether facial attribute labeling can make model behavior more transparent. RQ2 concerns reliability, namely which attributes correlate the most with false positives and false negatives at video level. RQ3 concerns design, namely whether detector training and evaluation can be reorganized around attribute information to improve robustness.

Datasets, preprocessing, and detection baselines.

Experiments are carried out on the FaceForensics++ corpus, focusing on pristine YouTube videos and their DeepFakes counterparts at the c40 compression tier. Videos are decoded, frames are uniformly sampled with a fixed skip factor to reduce redundancy, and faces are detected with a classical cascade. Tightly cropped face patches at (224×224) feed an image-level classifier. The reference detector is intentionally simple in order to isolate the effect of attributes. A VGG16 backbone with ImageNet initialization is used as a frozen feature extractor, followed by a compact classification head. Training, validation, and test splits are balanced by class and kept disjoint at video level. Video decisions are obtained by aggregating frame scores and by selecting the principal face track when multiple faces are present. With this configuration the system reaches high accuracy at video level, with precision and recall balanced across classes, and with stable behavior under the adopted compression and aggregation protocol. These baselines establish a transparent testbed where attribute-conditioned effects can be measured.

State of the Art.

The thesis is grounded in a structured literature review that systematizes methods, datasets, and evaluation protocols in face deepfake detection. The review follows a documented search-and-screen protocol across Scopus, IEEE Xplore, ScienceDirect and targeted snowballing, with inclusion criteria privileging reproducible evaluations on public dataset such as FaceForensics++, Celeb-DF, and DeepFake Detection Challenge Dataset (DFDC). The surveyed methods cover frame-level artifact detectors, face-centric physiological and geometric cues, and spatio–temporal pipelines, complemented by multimodal audio–visual approaches and fairness analyses. Surveys and benchmarks are used to map strengths and failure modes under compression, editing chains, and cross-dataset shift, which motivates the thesis focus on interpretability and robustness. This evidence base exposes two persistent gaps that directly shape Chapters 4 and 5: limited understanding of how high-level attributes relate to misclassifications, and a lack of training workflows that explicitly control attribute exposure. The review therefore provides both the methodological landscape for the baseline detector and the conceptual rationale for a semi-supervised attribute labeling pipeline and attribute-aware training strategies.

Industry activities in PwC.

The industrial period at PricewaterhouseCoopers Business Services Italia S.r.l. translated the research agenda into production-minded prototypes and evaluation practices. A staged sequence of projects consolidated the tooling and informed design choices later adopted in the thesis: a transfer-learning binary image classifier established a lightweight baseline for data hygiene and augmentation; a compact MNIST workflow standardized end-to-end training and reporting; a classical Olivetti face identification exercise validated reproducible splits and error inspection; finally, a video-level deepfake detector combined face cropping, pre-trained convolutional backbones, and simple temporal aggregation. These projects were developed with Python and common DL frameworks, presented in internal seminars, and used to stress operational constraints such as storage, compute budgets, compression tolerance, and threshold calibration. The lessons learned informed Chapter 4 by fixing a robust preprocessing and evaluation protocol for the attribute-enriched analysis, and guided Chapter 5 by prioritizing bias-aware curation, targeted augmentations, and attribute-conditioned training and reporting that can be adopted in enterprise settings without excessive complexity.

Analysis of DF Detection through Attribute Labeling.

Chapter 4 introduces a semi-supervised pipeline that scales facial attribute annotation with limited human effort. A seed set of fifty real videos is manually labeled for gender, hair color, hair length, ear visibility, and ethnicity, together with a set of boolean context flags. The manipulated companions inherit the same labels since identity swap keeps stable appearance traits. A per-attribute classifier, based on a compact convolutional backbone, is then trained on the seed, used to pseudo-label the remaining videos, and iteratively refined by including high-confidence predictions. This procedure yields a labeled testbed for downstream analysis without incurring prohibitive annotation costs. The chapter documents safeguards adopted to control error propagation, such as confidence thresholds for acceptance, manual spot checks on the tail of the confidence distribution, and the exclusion of attributes that collapse to a single value in the seed. The labeled corpus enables a systematic link between detector outputs and appearance factors. The analysis proceeds in three layers. First, descriptive statistics quantify error rates per attribute value, for example the share of false positives for short hair and for long hair. Second, dependence measures are computed, including group-wise differences in precision and recall, effect sizes on confusion-matrix entries, and non-parametric association coefficients for categorical variables. Third, simple interpretable models are fitted on the test predictions, for example logistic regressions that use attribute dummies to explain the probability of a misclassification, together with partial-dependence visualizations. Across these views, two factors emerge consistently. Ear visibility correlates with detector behavior, with elevated error rates when ears are occluded by hair or accessories. Hair length also influences outcomes, particularly in profiles and semi-profiles where the face contour and side hair interact with compression. These patterns persist after controlling for class balance, video aggregation, and the presence of multiple faces, which indicates that they are not artifacts of sampling alone. The chapter further reports calibration slices by attribute, showing that score distributions shift across groups. In particular, the same confidence threshold yields different precision–recall trade-offs when ears are not visible, which suggests group-aware thresholding or improved calibration as practical remedies. The multi-face setting is addressed explicitly. Since the dataset includes crowd scenes and interviews with more than one person in frame, the analysis compares three aggregation rules, namely majority voting across faces, selection of the largest face, and selection of the most persistent track over time. The last option, which approximates the main subject, reduces spurious errors introduced by bystanders, and clarifies that the attribute effects highlighted above are not driven by incidental faces. Taken together, the chapter provides an empirical map of where the baseline detector breaks, and it does so in terms that a human analyst can verify on representative clips.

Attribute-Aware Training Strategies.

Building on the evidence gathered in Chapter 4, Chapter 5 investigates how facial attributes can be used not only for post hoc analysis but also to probe the behaviour of a DeepFake detector under controlled distribution shifts. The chapter introduces an attribute-aware pipeline in which a VGG16-based classifier is trained multiple times on FaceForensics++, each time excluding one attribute value from the training data and then evaluating all models on a common, complete test set. This controlled exclusion design, combined with subgroup metrics and minimum support thresholds, provides an exploratory view of how the absence of specific attribute configurations affects accuracy, AUC and the balance between TPR and TNR. The results suggest a moderate but recurrent effect for hair_length and a more pronounced sensitivity to is_ears_visible, with the removal of visible-ear cases in training associated with a higher rate of FAKE → REAL errors. The chapter concludes with cautious operational recommendations: curating training data to ensure explicit coverage of critical attributes, monitoring performance by subgroup, and considering attribute-aware calibration or thresholding in deployment. These findings do not provide definitive bias guarantees, but they indicate that simple attribute-aware training and evaluation strategies may help to organise robustness checks and to support more transparent assessment of DeepFake detectors.

Conclusions and Future Work.

The concluding chapter revisits the three research questions through the lens of the empirical results and frames the contribution of the thesis as primarily methodological and exploratory. The work combines a standard VGG16-based DeepFake detector on FaceForensics++ with a semi-supervised ResNet18 attribute labeller, and uses the resulting annotations to study misclassification patterns and simple attribute-aware training variants. The analysis suggests that certain appearance factors, in particular hair length and ear visibility, are associated with variations in error rates and may therefore offer useful context for post hoc monitoring and bias-aware inspection, although no causal conclusions are drawn. The chapter also highlights the main limitations of the study, including the focus on a single dataset and compression level, a limited and imbalanced attribute set, reliance on univariate statistics, and the use of a deliberately simple architecture. Within these constraints, the thesis contributions can be seen as: (i) a reproducible semi-supervised pipeline for facial attribute labeling on DeepFake benchmarks; (ii) an empirical indication that attribute-conditioned error analysis may help organise the evaluation of DeepFake detectors; and (iii) a set of preliminary ideas for attribute-aware training and calibration. Possible extensions include applying the same protocol to more diverse datasets and richer attribute taxonomies, adopting multivariate analytical tools, and exploring the integration of attribute information into data curation, sampling and threshold selection to support more robust and fairness-aware detection in future work.