A National Data Library: the Modular Approach

The UK National Data Library: Technical White Paper Challenge, initiated by Wellcome and the Economic and Social Research Council (ESRC), sought visionary approaches to enhancing access to UK public sector datasets. Evaluated by an expert panel, this paper was recognised as a distinguished and successful response to the call, offering innovative insights for creating a National Data Library that benefits both the research community and the public good.

Previous attempts at delivering national data infrastructure have relied - unwisely - on moving all the data to a single box, and giving all delivery work to huge single teams managed within large single organisations. That model is bad for privacy, scalability, accountability, and delivery. Those projects have gone badly.

Here we set out a new model: a network of single function components, stitched together into a working data service by the National Data Library.

This model is better because it:

• Follows best practice in delivery of large data and digital projects;
• Ensures clarity, transparency, and accountability for each component and team;
• Drives best use of existing tools and teams;
• Leaves the best legacy for future work;
• Ensures scalability;
• Facilitates resilience;
• Lets you pick the best team for each component.

This paper summarises: the context; the reasons why recent government data platforms have failed; and the best technical design of an NDL, including the components and the overarching network. It then sets out a series of organisational approaches to help government define the cultures and market incentives to drive success. It ends with a concrete proposal for a small project to illustrate this way of working.