Evolution of UMAP Embeddings in ResNet-50: Hypernym Bias and Neural Collapse

This video visualizes the evolution of UMAP embeddings of ResNet-152 penultimate-layer features throughout training. The training settings follow the Neuronal Collapse framework proposed by Papyan et al. (2020).

🔹 Left plot: UMAP projection of training set embeddings (50 samples per class, 50,000 total).
🔹 Right plot: UMAP projection of validation set embeddings (50 samples per class, 50,000 total).
🔹 Color mapping: Represents hyponymy (hierarchical) distance relative to the anchor class: "tabby cat."

Key Observations

1. Early Training Phase: The network initially learns to organize features based on high-level category relations (hypernyms), such as distinguishing artifacts from animals.

2. Emerging Subclusters:  Through training, manifold complexity increases, subclusters emerge within major categories..

3. Neural Collapse Emergence:  In later training stages, the training set embeddings converge into class-specific clusters, losing hierarchical information.

4. Generalization: Neural collapse is observed only on the training set, while the validation set retains some hierarchical structure.

These findings provide insights into Hypernym Bias, as explored in our paper:
Malashin R. , Yachnaya V, Mullin A. "Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy." Accepted at the 28th International Conference on Artificial Intelligence and Statistics (AISTATS) (2025) .