Gender Trouble in Language Models: A Critical Audit Guided by Gender Performativity Theory

Language models deployed in high-impact sectors like healthcare, education, and law risk perpetuating discrimination against marginalized groups. Existing efforts to reduce biases in models struggle to remove complex prejudices beyond the surface level. Here, we propose a novel framework for evaluating how language models encode and represent gender, rooted in decades of sociological research on gender and language. We identify three key requirements: avoiding essentialism, ensuring meaningful embeddings for all gender identities, and eliminating harmful stereotypes. Testing these requirements on multiple prominent language models, we reveal persistent patterns of gender essentialism, inadequate representations of nonbinary and transgender identities, and harmful pathologizing stereotypes. These findings highlight the need for critical engagement with concepts like gender when auditing language models for their representations. Addressing these issues is crucial to preventing harmful outcomes in high-risk applications, such as biased medical diagnoses, misinformed educational assessments, and weakened legal protections for marginalized communities.