Intro to Py-Feat

Eshin Jolly

py-feat.org

SLAB 04/28/22

Roadmap

1. Brief Overview of Facial Expression Research

2. Automated Detection Techniques

3. Py-Feat Design Philosophy

4. Py-Feat Tutorial Walkthrough

Roadmap

1. Brief Overview of Facial Expression Research

2. Automated Detection Techniques

3. Py-Feat Design Philosophy

4. Py-Feat Tutorial Walkthrough

History

• 1969: First system for formally coding facial movements in by Swedish anatomist Carl-Herman Hjortsjö
• 1978: Evolved into Facial Action Coding System (FACS) by Paul Ekman and Wallace Friesen
• 2002: Updated version of FACS with additional codes by Ekman, Friesen, and Joseph C. Hager
• Today: Multiple FACS variants exist for babies, chimpanzees, macaques, and humans

Facial Muscles and Action Units

Major muscles of the face that form the basis of FACS (Zarins, 2014)

Action Units and Expressions

AU4: Brow Lowerer, corrugator supercilii, procerus, depressor supercilii

Smiling: zygomaticus major, orbicularis oculi, depressor labii inferioris, orbicularis oris

Roadmap

1. Brief Overview of Facial Expression Research

2. Automated Detection Techniques

3. Py-Feat Design Philosophy

4. Py-Feat Tutorial Walkthrough

Break down of facial analysis

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Break down of facial analysis

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Break down of facial analysis

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Break down of facial analysis

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Break down of facial analysis

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Break down of facial analysis

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Common Approaches

Goal: Given input images and labels (e.g. AUs, emotions, landmarks), learn a model that can predict labels from the input images

1. “Classic” Computer Vision
   • Manually specified image features
   • Histogram of Oriented Gradients (HOG)
2. Deep Learning
   • Model learns its own image features (end-to-end)
   • Convolutional Neural Networks (CNNs)

“Classic” CV: HOG Features

• Intuition: local image intensity changes and edge orientations provide a useful descriptor of object appearance and shape
• Extract features using the gradient orientation in multiple regions of an image; output is a histogram per image region
• Output features -> simple classifier (e.g. SVM, RF, Logistic Regression)

How to extract HOG Features

 

Angela Merkel HOG-ified

Deep Learning: CNNs

• Given some training labels (e.g. presence of AU), model automatically learns what features are important for prediction (end-to-end training)

Example 1. Deep Region Multi-label Learning (DRML): Learns informative face regions for AU detection along with covariance of AUs across images

Example 2: JAA-Net: One of the first models to jointly learn facial alignment and AU detection

Caveats

• Models often learn biases that exist in the dataset
• Image datasets vary wildly on quality and representativeness
• Models are basically never given contextual information
• In practice DL models tend to be overfit to the data reported in papers despite state-of-the-art results

Always inspect model predictions and images when you use automated techniques!

Roadmap

1. Brief Overview of Facial Expression Research

2. Automated Detection Techniques

3. Py-Feat Design Philosophy

4. Py-Feat Tutorial Walkthrough

Philosophy: Usability-based design

• State-of-the-art models + user-experience
  • Best models often don’t share code or don’t maintain it
  • Best models historically absorbed into closed-source commerical applications
  • No existing library for full-suite of tools (preprocessing, analyzing, visualizing)
• Make your life as a researcher easier
  • Don’t need to be a machine-learning expert
  • Write less boilerplate code
  • Useful high-level functions (e.g. plotting, regression, etc)
• Let’s you work at the “right” level of abstraction

Py-Feat glues all this together for you!

1. Face Detection

2. Facial Landmark Detection + Alignment

3. Face + Head Pose Estimation

4. Action Unit Detection

5. Emotion Detection

Our resident deep learning expert, who engineered and trained these models from scratch!

Training and Validation

• Mix of datasets used in previous literature
• Posed and elicited facial expressions
• One or more FACS expert annotations for action units
• Elicitation task labels or manual annotations for emotions
• Curated and in-the-wild images (vary in luminance, pose, etc.)
• 4k - 1.1M images/frames per dataset
• Wide range of ages, gender, ethnicity (but definitely skews Caucasian)

Example Emotion Prediction Performance

F1 Scores predicting each emotion label on unseen data. Higher numbers darker colors = better performance. Chance = .14 (1/7)

Philosophy: Community Resource

• Open source
  • View and contribute to code and download pre-trained models
• Comprehensive tutorials (with more on the way)
  • Basic tutorials with zero experience required
  • Advanced tutorials for complete analyses
• Easily extensible with new models

Current Work In Progress

• Improve usability (feedback welcome!)
  • Library and code updates
  • More tutorials and guides
• Robustness checks
  • Orientation
  • Luminance
  • Occlusion
• New more generalizable custom AU detector under development by our resident deep learning wizard

Roadmap

1. Brief Overview of Facial Expression Research

2. Automated Detection Techniques

3. Py-Feat Design Philosophy

4. Tutorial Walkthrough

Broad Module Overview

1. Detector
   • Single model class that “glues” together multiple detectors
   • Main tool for processing images and videos
2. Fex
   • Single data class that stores the output of Detector(s)
   • Just a fancier pandas dataframe with additional methods for advanced statistical analysis and processing
3. plotting and utils functions
   • For visualizing and animating detections and analyses

Detectors

from feat import Detector

detector = Detector()

Detectors

from feat import Detector

detector = Detector() # <-- Wrapper around face, landmark, emotion, au, etc models

Detectors

from feat import Detector

detector = Detector() # <-- Wrapper around face, landmark, emotion, au, etc models

detector.detect_image('my_img.jpg') # <-- Perform all types of detection on an image

Detectors

from feat import Detector

detector = Detector() # <-- Wrapper around face, landmark, emotion, au, etc models

detector.detect_image('my_img.jpg') # <-- Perform all types of detection on an image

detector.detect_video('my_video.mp4') # <-- Perform all types of detection on video (frames)  

Fex Dataclass

from feat import Fex

fex = Fex()

Fex Dataclass

from feat import Fex

fex = Fex() # <-- Special dataframe that stores and operates on detector output

Fex Dataclass

from feat import Fex

fex = Fex() # <-- Special dataframe that stores and operates on detector output

# Most often you'll create it from a Detector...

Fex Dataclass

from feat import Detector, Fex

detector = Detector()
detections = detector.detect_image('./imgs/single_face.jpg') # <-- Returns a Fex dataclass!

Fex Dataclass

from feat import Detector, Fex

detector = Detector()
detections = detector.detect_image('./imgs/single_face.jpg') # <-- Returns a Fex dataclass!

detections.head()

	frame	FaceRectX	FaceRectY	FaceRectWidth	FaceRectHeight	FaceScore	x_0	x_1	x_2	x_3	...	Roll	Yaw	anger	disgust	fear	happiness	sadness	surprise	neutral	input
0	0	196.976837	140.997742	173.810486	257.639343	0.999681	192.864593	191.586715	192.874618	197.394792	...	-1.903955	4.869262	0.000369	0.000026	0.000485	0.986996	0.000046	0.01201	0.000068	./imgs/single_face.jpg

1 rows × 173 columns

Now we can do other stuff like:

# Run fMRI style GLM over AUs
detections.regress(X=my_design_matrix, y=detections.aus)

# Run stimulus-ISC analysis for a single AU
detections.isc(col="AU01", method="pearson")

Visualization

from feat import plot_face

# 20 dimensional vector of AU intensities
action_units = np.zeros(20)

Visualization

from feat import plot_face

# 20 dimensional vector of AU intensities
action_units = np.zeros(20)
ax = plot_face(au=action_units, title='Neutral')

# Increase AU1 intensity: inner brow
action_units[0] = 3
ax = plot_face(au=action_units, title='Raised inner brow')

Animation

from feat import animate_face

# Just pass in a FACS AU id

Animation

from feat import animate_face

# Just pass in a FACS AU id
animation = animate_face(AU=1, start=0, end=3)

# Create 2 AU vectors to interpolate between
neutral = np.zeros(20)
smiling = neutral.copy()
smiling[0], smiling[8] = 3, 3

animation = animate_face(
    start=neutral,
    end=smiling,
    muscles={"all": "heatmap"},
)

Summary figures

detections.plot_detections(poses=True)

Comprehensive analysis tools

Conceptual replication of Watson et al (2020): Participants received good or bad news while their facial expressions were filmed. Used the original videos and ran them through py-feat to compare conditions:

(a) t-test comparing AUs across conditions; (b) “decoding beta-map” classifying good vs bad news from AUs; (c) visualizing the decoder weights on a template face

Documentation and Tutorials

Thanks!

How Good Are Humans at detecting Facial Expressions?

Mean True Positive recognition performance of humans vs classifiers

 

ROC curves for same comparison (dashed line is chance)

Dupre et al, 2020, PLoS One

Current Default Detectors

1. Face Detection: RetinaFace
2. Facial Landmark Detection + Alignment: MobileNet
3. Face + Head Pose Estimation: img2pose
4. Action Unit Detection: SVM
5. Emotion Detection: ResMaskNet