Dataset Open Access

SOTorrent Dataset

Baltes, Sebastian; Dumani, Lorik

Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of code snippets and free-form text on a wide variety of topics. Like other software artifacts, questions and answers on SO evolve over time, for example when bugs in code snippets are fixed, code is updated to work with a more recent library version, or text surrounding a code snippet is edited for clarity. To be able to analyze how content on SO evolves, we built SOTorrent, an open dataset based on the official SO data dump. SOTorrent provides access to the version history of SO content at the level of whole posts and individual text or code blocks. It connects SO posts to other platforms by aggregating URLs from text blocks and comments, and by collecting references from GitHub files to SO posts. Our vision is that researchers will use SOTorrent to investigate and understand the evolution of SO posts and their relation to other platforms such as GitHub.

If you use this dataset in your work, please cite our MSR 2018 paper (BibTex) or our MSR 2019 mining challenge proposal.

The dataset is based on the official Stack Overflow data dump released 2018-12-02 (https://archive.org/details/stackexchange) and the Google BigQuery GitHub data set queried 2018-12-09 (https://cloud.google.com/bigquery/public-data/github). Please read the license files (LICENSE.md) before using the dataset.
Files (95.6 GB)
Name Size
1_create_database.sql
md5:e079bf0648e49e77aee5053a7170df4f
6.6 kB Download
2_load_so_from_xml.sql
md5:aeab3d449ec3f39a5c372475ef85bdf8
1.2 kB Download
3_create_indices.sql
md5:4bf83b1ea7fbe7c5eeffb403c52bd402
488 Bytes Download
4_create_sotorrent_tables.sql
md5:dc71eaf388f5a7bf1815dada6ee90352
6.7 kB Download
5_create_sotorrent_user.sql
md5:e719b7d430dcbfc41e9149fd7be2cbbc
495 Bytes Download
6_load_sotorrent.sql
md5:cea4330cba80e2bec7a66dcc6270c0d3
4.6 kB Download
7_load_postreferencegh.sql
md5:a747635b8d2361ab07f16460f6436054
544 Bytes Download
8_load_ghmatches.sql
md5:053033fbbc897272c6e98bd877d49d50
457 Bytes Download
9_create_sotorrent_indices.sql
md5:d3ee212014e05ba9fb924bce92c28c85
1.5 kB Download
Badges.xml.gz
md5:781bd4cc1da8f0518011fa6ac0590c59
310.3 MB Download
Comments.xml.gz
md5:2fb09435346d796cc7b08d8148ea771d
6.0 GB Download
CommentUrl.csv.gz
md5:032869b4f42458855048e29edbf59d59
429.5 MB Download
GHMatches.csv.gz
md5:5121b1659ede2e4f6ce77233cdcd973f
1.3 GB Download
LICENSE.md
md5:dd690a9239427df0ccf2369c97285125
38.4 kB Download
PostBlockDiff.csv.gz
md5:bd9a25996f530391b0dfd1f63273c1dd
9.9 GB Download
PostBlockVersion.csv.gz
md5:78cba7e35e6169ea4b230174a195ace9
21.5 GB Download
PostHistory.xml.gz
md5:c3473175aa700784561043a4b420176f
32.3 GB Download
PostLinks.xml.gz
md5:3673fd8815d409b8471734de3d01fbc0
101.8 MB Download
PostReferenceGH.csv.gz
md5:61e0fe853e2b33c1063d54d96a805c3b
384.9 MB Download
Posts.xml.gz
md5:ff7cbf0c93cf8efabdd9512324d8a30b
18.2 GB Download
PostVersion.csv.gz
md5:fd8e444a05d25ffa7d473f17643c6f70
1.0 GB Download
PostVersionUrl.csv.gz
md5:271e15da37247800bfa5e09441e5d830
1.4 GB Download
README.md
md5:76e1d6694a2f7862dd5361fd1a9a1074
1.7 kB Download
Tags.xml.gz
md5:45ade28bffb60ef9b05d0b5af62e599a
1.1 MB Download
TitleVersion.csv.gz
md5:7f23f3308b1def66b32d4e5b5dff9995
771.6 MB Download
Users.xml.gz
md5:0b0fb49c71a001582219db42394a5976
633.1 MB Download
Votes.xml.gz
md5:42ae552fbdadb4bd4e02e26ec57fa070
1.4 GB Download
2,157
7,856
views
downloads
All versions This version
Views 2,157378
Downloads 7,8561,101
Data volume 37.6 TB5.1 TB
Unique views 1,702290
Unique downloads 1,358255

Share

Cite as