Tested but not Evaluated: Social Science and the Assessment of AI in Public Programs

Tested but Not Evaluated is a call for social scientists — evaluators, behavioral economists, sociologists and other researchers trained in causal inference, social construct measurement, ethics, and stakeholder engagement — to be integrated into AI testing and evaluation as a standard practice. Organizations providing AI-driven tools, systems and services — what I label here “AI-driven programs” — need better evaluation standards and practices in AI system development and deployment, and social scientists, technologists, data governance functions, product developers and many others must work together to develop such practices. Given the scope and scale of public service program impacts, new standards and methods are more urgent than ever in testing AI-driven programs in public policy. We need social scientists to claim a role — an ethical, human-centered role — in the AI eval space. Tested but Not Evaluated will contribute to efforts to build a new, multidisciplinary approach to AI evaluation and measurement in public policy: one that helps us understand the impacts of AI-driven systems and programs on society.