Delivering an effective National Data Library

The UK National Data Library: Technical White Paper Challenge, initiated by Wellcome and the Economic and Social Research Council (ESRC), sought visionary approaches to enhancing access to UK public sector datasets. Evaluated by an expert panel, this paper was recognised as a distinguished and successful response to the call, offering innovative insights for creating a National Data Library that benefits both the research community and the public good.

A clear vision is a necessary design constraint for the National Data Library (NDL).

During the 2024 election, the Labour Party manifesto described that the NDL will “bring together existing research programmes and help deliver data-driven public services, whilst maintaining strong safeguards and ensuring all of the public benefit”. Science and Technology minister Peter Kyle has said the NDL will be a “data access policy and a library and exchange service” and will deliver “AI-powered public services that [citizens] deserve”.

This ambiguous language has caused confusion. As a result, there seems to be (at least) two competing visions for what the NDL could do:

1. Enable improved use of government-held data for research.
2. Enable improved operational data access across public services.

The difference between data for research and data for operations is a significant one, technically, legally and commercially. In health, the NHS Federated Data Platform is designed to connect data from across NHS organisations to assist healthcare professionals in real-time, operational decision-making. There’s also a need for a much wider set of data infrastructures to ensure health data is safely used for research.

The needs of researchers and operational data users are unlikely to be met by a singular intervention. We recommend the UK Government and other organisations looking to influence its design clarify their vision for the NDL, documenting this as a clear, tightly-bound problem statement. This will help bridge the chasm between a high-level political vision and technical execution.

In this Technical White Paper, we’re choosing to pursue a vision of a NDL that enables improved discovery of government-held data for research. This is in line with the Challenge’s description of the NDL’s purpose to “make public sector datasets more accessible to researchers and enable future science to thrive”.

(The UK has significant improvements to make on operational data access, though. Estonia’s X-Road is an example of modern cross-government data sharing to deliver more efficient public services. This is sometimes referred to as the ‘data exchange layer’ in Digital Public Infrastructure circles.)