WONDERBREAD: A Benchmark + Dataset for Business Process Management (BPM) Tasks

Paper: WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks

Background

The WONDERBREAD dataset contains 2,928 human demonstrations of 598 web navigation workflows across 6 types of BPM tasks. These tasks measure the ability of a model to generate accurate documentation, assist in knowledge transfer, and improve the efficiency of workflows.

Please see our website for more details: https://wonderbread.stanford.edu/

Quick Start

To start, download debug_demos.zip (1 GB). It contains a subset of 24 demonstrations which can give you a sense of how the dataset is structured.

To reproduce the paper, download gold_demos.zip (33 GB). It contains 724 demonstrations corresponding to the 162 "Gold" tasks which were used for all the evaluations in the original paper.

To obtain the full dataset, download demos.zip (133 GB). This contains all 2,928 demonstrations and can be used for training, fine-tuning, and evaluating models.

Dataset Structure

The dataset contains several files, defined below.

1. Raw Data (useful for training/fine-tuning/evaluation)
   1. debug_demos.zip (1 GB) -- a subset of only 24 demonstrations taken from the full dataset. Useful to get a sense of the dataset and for debugging.
   2. gold_demos.zip (22 GB) -- a subset of only 724 demonstrations corresopnding to the 162 "Gold" tasks. This is the dataset that was used for all evaluations in the original WONDERBREAD paper.
   3. demos.zip (133 GB) -- all 2,928 demonstrations across 598 tasks. Useful for training your own models.
2. Modality-Specific Subsets of Raw Data (useful for specific types of training/fine-tuning/evaluation)
   
   1. All Demos
      1. demos_sop_only.zip (4 MB)-- only the SOP .txt files for all 2,928 demonstrations
      2. demos_sop_and_trace_only.zip (770 MB)-- only the SOP .txt files and action trace .json files for all 2,928 demonstrations
      3. demos_sop_and_trace_and_screenshots_only.zip (22 GB)-- only the SOP .txt files and action trace .json files and screenshot images for all 2,928 demonstrations
   2. "Gold" Demos
      1. gold_demos_sop_only.zip (1 MB)-- only the SOP .txt files for the 724 demonstrations in the "Gold" tasks.
      2. gold_demos_sop_and_trace_only.zip (190 MB) -- only the SOP .txt files and action trace .json files for the 724 demonstrations in the "Gold" tasks.
      3. gold_demos_sop_and_trace_and_screenshots_only.zip (6 GB) -- only the SOP .txt files and action trace .json files and screenshot images for the 724 demonstrations in the "Gold" tasks
3. Evaluation (useful for evaluation)
   1. qa_dataset.csv -- contains all 120 questions and ground truth answers used in the "Knowlege Transfer" evaluation.
      
   2. df_rankings.csv -- contains the rankings of all "Gold" tasks used in the "SOP Ranking" evaluation.
      
4. Metadata (can be safely ignored)
   1. Process Mining Task Demonstrations.xlsx -- maps human annotators to specific demonstrations; also contains "Gold" task rankings used in the "SOP Ranking" evaluation.
   2. metadata.json -- maps Google Drive URLs to Google Drive Folder IDs to demonstration names
   3. df_valid.csv -- tracks assets associated with each demonstration