Published April 5, 2024 | Version v1
Dataset Open

SchemaPile: A Large Collection of Relational Database Schemas

  • 1. ROR icon University of Amsterdam
  • 2. ROR icon University of California, Berkeley

Description

Access to fine-grained schema information is crucial for understanding how relational databases are designed and used in practice, and for building systems that help users interact with them. Furthermore, such information is required as training data to leverage the potential of large language models (LLMs) for improving data preparation, data integration and natural language querying.

Existing single-table corpora such as GitTables provide insights into how tables are structured in-the-wild, but lack detailed schema information about how tables relate to each other, as well as metadata like data types or integrity constraints. On the other hand, existing multi-table (or database schema) datasets are rather small and attribute-poor, leaving it unclear to what extent they actually represent typical real-world database schemas. 

In order to address these challenges, we present SchemaPile, a corpus of 221,171 database schemas, extracted from SQL files on GitHub. It contains 1.7 million tables with 10 million column definitions, 700 thousand foreign key relationships, seven million integrity constraints, and data content for more than 340 thousand tables. We conduct an in-depth analysis on the millions of schema metadata properties in our corpus, as well as its highly diverse language and topic distribution. In addition, we showcase the potential of SchemaPile to improve a variety of data management applications, e.g., fine-tuning LLMs for schema-only foreign key detection, improving CSV header detection and evaluating multi-dialect SQL parsers. We publish the code and data for recreating SchemaPile and a permissively licensed subset SchemaPile-Perm. 

GitHub repo: http://github.com/amsterdata/schemapile

Files

Files (228.8 MB)

Name Size Download all
md5:35e5296aced1bbb2a60f3f37912bf6a3
152.8 MB Download
md5:c1d305adda48e64f216f82563c138ff2
27.0 MB Download
md5:d26631c189f03d7f845008b4727ffad0
48.9 MB Download