AutumnBench Public Benchmark

AutumnBench Dataset Release

This dataset accompanies the paper “Benchmarking World-Model Learning with Environment-Level Queries.”

It contains the benchmark release for AutumnBench, including task prompts, ground-truth answers, and environment programs used for evaluation.

The release includes a `prompts/` folder containing the task prompts for each AutumnBench environment and challenge task. Files are named {env_name}_{task_name}.json, where task_name is one of mfp, cd, or planning.

It also includes an `answers/` folder containing the ground-truth answers for each task. For mfp, each file gives the index of the correct answer option (0-indexed). For cd, each file gives the condition after which the change occurs. For planning, each file gives the optimal number of steps required to reach the target.

The `programs/` folder contains the Autumn programs for the original environments, as well as the modified environments used in the change-detection task. The Autumn interpreter for running the programs is available in the GitHub repository: https://github.com/BasisResearch/Autumn.cpp. 

Finally, the release includes a shared `color_dict.yaml` file that maps numeric identifiers to color names.

This release is intended to support reproducibility and further analysis of the AutumnBench benchmark described in “Benchmarking World-Model Learning with Environment-Level Queries.”