A Precarious Task: Science Indicators, Research Assessment, and Science Policy in the United States

This dissertation examines the evolution of science indicators in the United States from the 1970s to the present. It traces how indicators have been constructed, interpreted, and adopted across three historical periods and considers how they have been used to make claims about research quality and value. The analysis is based on inductive thematic analysis of approximately 200 documents, including federal reports, congressional hearings, policy memoranda, and scholarly literature.

The dissertation makes three primary contributions. First, it identifies the operationalization of evaluative concepts into indicators as a key site where governance is enacted. Second, it documents a growing distance between indicators and the phenomena they indicate. While this distance is acknowledged early in the development of science indicators, it becomes more difficult to track as indicators become embedded in infrastructures and begin to travel across contexts. Third, it distinguishes between research quality and research value as analytically distinct but historically conflated concepts, demonstrating the extent to which their persistent conflation has complicated research assessment and science policy for over fifty years.

The findings of this dissertation support several recommendations. First, evaluative terms should be explicitly identified and defined prior to their operationalization. Second, evaluators should remain attentive to the distinction between research quality and research value, recognizing that they often speak to different audiences. Third, the space between an indicator and the phenomenon it should be acknowledged and actively protected. Finally, the use of indicators as evaluative tools should be approached with humility, recognizing their capacity to shape behavior and reorganize complex sociotechnical systems in ways that extend beyond their original intent.

These contributions take on particular significance in the current moment. Recent policy developments - notably the Executive Order on “Gold Standard Science” - signal a more explicit intervention into the terms of evaluation themselves. Understood within the longer history traced in this dissertation, such developments can be understood as intensifications of prior dynamics, in which control is exercised through the definition, operationalization, and enforcement of evaluative terms.