MNIST-C: A Robustness Benchmark for Computer Vision

We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions applied to the MNIST test set for benchmarking out-of-distribution robustness in computer vision. Through several experiments and visualizations we demonstrate that our corruptions significantly degrade performance of state-of-the-art computer vision models while preserving the semantics of the image. In contrast to the popular notion of adversarial robustness, our corruptions do not seek to achieve worst-case performance but are instead designed to be broad and diverse, capturing multiple failure modes of modern models. In fact, we find that several previously published adversarial defenses \emph{significantly degrade} robustness as measured by MNIST-C. We hope that our benchmark serves as a useful tool for future work in designing systems that are able to learn robust feature representations that capture the true semantics of the input.