Dataset Open Access

SOTorrent Dataset

Baltes, Sebastian; Dumani, Lorik

Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of code snippets and free-form text on a wide variety of topics. Like other software artifacts, questions and answers on SO evolve over time, for example when bugs in code snippets are fixed, code is updated to work with a more recent library version, or text surrounding a code snippet is edited for clarity. To be able to analyze how content on SO evolves, we built SOTorrent, an open dataset based on the official SO data dump. SOTorrent provides access to the version history of SO content at the level of whole posts and individual text or code blocks. It connects SO posts to other platforms by aggregating URLs from text blocks and comments, and by collecting references from GitHub files to SO posts. Our vision is that researchers will use SOTorrent to investigate and understand the evolution of SO posts and their relation to other platforms such as GitHub.

If you use this dataset in your work, please cite our MSR 2018 paper (BibTex) or our MSR 2019 mining challenge proposal.

The dataset is based on the official Stack Overflow data dump released 2018-09-05 (https://archive.org/details/stackexchange) and the Google BigQuery GitHub data set queried 2018-09-23 (https://cloud.google.com/bigquery/public-data/github). Please read the license files (LICENSE.md) before using the dataset.
Files (91.4 GB)
Name Size
1_create_database.sql
md5:f86e6dd62cb67c24d3bfe7786e06c311
6.5 kB Download
2_load_so_from_xml.sql
md5:335bae9dfa717e8770a3163ffe9766c7
1.2 kB Download
3_create_indices.sql
md5:34e0448795c8a5d721cdfaad0e5a60ee
488 Bytes Download
4_create_sotorrent_tables.sql
md5:503d42f43fc93be4951a90045e8b25bc
6.4 kB Download
5_create_sotorrent_user.sql
md5:d21d6f09ed4e3b139a995a0952e40aad
495 Bytes Download
6_load_sotorrent.sql
md5:73808cd585cbd8fa83b335de11410a0e
4.6 kB Download
7_load_postreferencegh.sql
md5:08dca95b38d5738e13e4a907a6043409
476 Bytes Download
8_create_sotorrent_indices.sql
md5:c20591fd8506997bec3b84eaf72c0f6f
1.5 kB Download
Badges.xml.gz
md5:5c9eca00455798b52c1d2350e230f10a
299.6 MB Download
Comments.xml.gz
md5:b86e623b5b7193a0162c2eb42a9e350f
5.8 GB Download
CommentUrl.csv.gz
md5:1a9da581afb6f437d338cebe698bcbfb
415.7 MB Download
LICENSE.md
md5:a229fa1ff538512f5001d5e27ab8b193
38.4 kB Download
PostBlockDiff.csv.gz
md5:d9583fca96e8a9ee44202546644edf3b
9.5 GB Download
PostBlockVersion.csv.gz
md5:ce1627e6d9d541898f8bbc21fb656d6c
20.9 GB Download
PostHistory.xml.gz
md5:135fa29ce42cd7e6124fed45a06667d7
31.4 GB Download
PostLinks.xml.gz
md5:d5c341e84b27c7380411d85032788e1a
98.1 MB Download
PostReferenceGH.csv.gz
md5:ffa65c18f935601ad4a33b7eb945bf75
354.9 MB Download
Posts.xml.gz
md5:d956c65639f7007077f22d53f03850e7
17.7 GB Download
PostVersion.csv.gz
md5:0555b6b1065ed00588bef3f865c946ea
1.0 GB Download
PostVersionUrl.csv.gz
md5:6ce3d76521b6a6af176f89762ab0b362
1.4 GB Download
README.md
md5:45e75aab2bb3bb51f2efa7b2e9bee4a3
1.4 kB Download
Tags.xml.gz
md5:fb80db0991af196d5ea26cf1895f39e4
1.0 MB Download
TitleVersion.csv.gz
md5:32360229a756f1a3cd1d5e0ae9203eab
749.1 MB Download
Users.xml.gz
md5:10daec62708b1bc9adae18f215f18c19
599.7 MB Download
Votes.xml.gz
md5:ca5b4acbd9a57496a64f6f8565db0584
1.3 GB Download
1,193
4,741
views
downloads
All versions This version
Views 1,193686
Downloads 4,7412,172
Data volume 20.8 TB11.9 TB
Unique views 963575
Unique downloads 747556

Share

Cite as