Towards Standardizing AI Bias Exploration

Creating fair AI systems is a complex problem that includes the assessment of context-dependent bias concerns. Existing research and programming libraries express specific concerns as measures of bias that they aim to constraint or mitigate. In practice, one should explore a wide variety of (sometimes incompatible) measures before deciding which warrant corrective action, but this is hindered by their narrow scope. In this work, we present a mathematical framework that splits literature measures of bias into building blocks and creates new combinations to cover a wide range of fairness concerns, such as classification or recommendation differences across multiple multivalue sensitive attributes (e.g., many genders and races, and their intersections). We show how this framework generalizes existing concepts and extract building blocks from popular literature measures. We finally implement our framework as a Python library, called FairBench, that facilitates systematic exploration of potential bias concerns.