Collected dataset for DEBAR

We share our two collected datasets and evaluation results online.

The first dataset is a set of 9 buggy architectures collected by existing studies. The buggy architectures come from two studies: eight architectures were collected by a previous empirical study on TensorFlow bugs (Github/StackOverflow-IPS-id.pbtxt) and one architecture was obtained from the study that proposes and evaluates TensorFuzz (TensorFuzz.pbtxt).

The second dataset contains 48 architectures from a large collection of research projects in TensorFlow Models repository. Overall, our second dataset contains a great diversity of neural architectures like CNN, RNN, GAN, HMM, and so on. Please note that we have no knowledge about whether the architectures in this dataset contain numerical bugs when collecting the dataset.

For every architecture in two datasets, we extract the computation graph by using a TensorFlow API. Each extracted computation graph is represented by a Protocol Buffer file, which provides the operations (nodes) and the data flow relations (edges).