Introduction

A Benchmark for Parameter-Identifiable Physical Reasoning

LLMPhy-TraySim is a synthetic benchmark for parameter-identifiable physical reasoning, designed to rigorously evaluate whether modern large language models (LLMs) and vision-language models (VLMs) can move beyond pattern recognition to recover physical parameters and generalize them across dynamical settings.

The central objective of LLMPhy-TraySim is to assess a model’s ability to (i) infer intrinsic physical properties of a dynamical system and (ii) predict event outcomes under novel configurations where the underlying physical parameters remain invariant but the scene layout and external perturbations change. Unlike traditional perception benchmarks, this formulation explicitly tests causal and transferable reasoning, rather than memorization of visual patterns.

Specifically, LLMPhy-TraySim tests whether models can:

• Recover actual physical parameters from observed dynamics
• Transfer these parameters to novel scene configurations
• Predict physical outcomes under new scenarios.

The benchmark explicitly separates physical parameter identification from downstream compositional reasoning, enabling rigorous evaluation of mechanistic generalization.

Why LLMPhy-TraySim?

Modern foundation models excel at perception and language understanding, but reliable deployment in real-world systems requires:

• Mechanistic reasoning
• Parameter identifiability
• Physically grounded prediction
• Transfer under intervention

LLMPhy-TraySim provides a controlled digital-twin-style benchmark to evaluate these capabilities in a systematic and reproducible manner.

At a Glance

• Total number of sequences: 101
• Dataset file size: 1.7 GB (zipped), 1.91 GB (extracted)

Files in this folder

README.md

LLMPhy-TraySim/: dataset content.

The main dataset is organized as:

llmphy_twophase_9_101_101.npy # ground truth details of all the examples.

example_10 # (an example from 101 examples)

  └──

    phase_1

    ├── cam1/ # scene dynamics video frames in top-down-view

    ├── cam2/ # scene dynamics video frames in side-view

    ├── cam3/ # scene dynamics video frames in side-view

    ├── cam4/ # scene dynamics video frames in side-view

    ├── cam1.mp4  # video from the frames in cam1/

    ├── cam2.mp4  # video from the frames in cam2/

    ├── cam3.mp4

    ├── cam4.mp4

    ├── example_10_view_1.png   # top-down view first frame

    ├── example_10_view_1_last.png # top-down view last frame

    ├── example_10_view_2.png

    ├── example_10_view_2_last.png

    ├── example_10_view_3.png

    ├── example_10_view_3_last.png

    ├── example_10_view_4.png

    ├── example_10_view_4_last.png

    └── obj_traj_10_gt.txt # ground truth instance discretized trajectories

    -----------------------

    ├── example_10.py # ground truth physical parameters and scene construction code (not to be used for inference).

    phase_2

    ├── example_10_view_1.png # phase_2 first image frame -- input for phase 2 inference.

    ├── example_10_view_2.png # phase_2 ground truth first frame in view 2 -- input for phase 2 inference.

    ├── example_10_view_3.png # phase_2 ground truth first frame in view 3 -- input for phase 2 inference

    ├── example_10_view_4.png  # phase_2 ground truth first frame in view 3 -- input for phase 2 inference

    -----------------------

    ├── obj_traj_10_gt.txt   # phase_2 ground truth trajectory -- not to be used during inference.

    ├── example_10.py   # phase_2 ground truth code -- not to be used during inference.

    ├── example_10_view_1_last.png  # phase_2 ground truth last frame -- not to be used during inference.

    ├── example_10_view_2_last.png # phase_2 ground truth last frame in view 2 -- not to be used during inference.

    ├── example_10_view_3_last.png # phase_2 ground truth first frame in view 3 -- not to be used during inference.

    ├── example_10_view_4_last.png # phase_2 ground truth first frame in view 3 -- not to be used during inference.

In phase_1:

• We provide all the 202 frames from 4 camera views in the folders phase_1/cam_Y for Y in 1,2,3,4. Note that cam_1 is the top-down view, while others are side views from 90 degree separated angles.
• We also separately provide the first and last frame from the four views as example_X_view_Y.png and example_X_view_Y_last.png respectively.
• The ground truth physical parameters used and the code to execute to recreate the sequence are provided in example_X.py.
• The ground truth trajectories of the centroids of the various objects in the scene are provided in obj_traj_X_gt.txt.
• We also provide mp4 formatted video sequence for each sequence.

In phase_2:

• We provide the first and last frame, example_X_view_Y.png and example_X_view_Y_last.png.
• The ground truth code is provided in example_X.py
• The ground truth trajectories of the centroids of the various objects in phase_2 are given in obj_traj_X_gt.txt

Task Details

LLMPhy-TraySim is divided into two sequential phases:

Phase 1: Physical Parameter Identification

Input

• A video sequence showing:
  • A single object instance (e.g., bottle, wineglass, martini glass) from each class
  • A pusher impacting the tray at a given velocity

Objective

Infer the physical parameters of each object class:

• Sliding friction
• Damping
• Stiffness
• Inertia

This phase evaluates whether models can extract mechanistic physical representations purely from observed motion. You may use the obj_traj_X_gt.txt as input to the model, e.g., an LLM.

Phase 2: Compositional Scene Reasoning & Prediction

Input

• A single static image showing the initial configuration of the scene.
• Multiple object instances placed on the tray
• A pusher approaching at a new velocity (different from Phase 1)

Required Reasoning Steps

The model must:

1.     Infer scene layout

o   Identify object classes

o   Determine spatial arrangement

2.     Reconstruct the scene in the simulator

3.     Assign physical parameters

o   Use estimates obtained in Phase 1

4.     Simulate forward dynamics

o   Run simulation for 200 steps

5.     Predict stability outcomes

o   Select which objects remain upright from five candidates

This phase tests:

• Compositional generalization
• Transfer of inferred physics
• Multi-body reasoning under distribution shift

Task Summary

Models evaluated on LLMPhy-TraySim must:

• Infer physical parameters from video dynamics (Phase 1)
• Perform scene parsing from a static image (Phase 2)
• Transfer physical knowledge to new perturbations
• Predict multi-body collision outcomes
• Select stable object instances under quantitative evaluation

Additional Details About the Tasks

The key steps in the two phases and the evaluation metrics are summarized below.

Step 1: Physical Parameter Estimation (Phase 1)

• Load phase 1 video dynamics for example  from camera views  (you may use one or many views). You may use the object centroid trajectory sequence obj_traj_X_gt.txt.
• Estimate physical parameters of each object class using the model:
  • PhysicalParameters ← model(phase_1/example_X/cam_Y)
• Store estimated parameters:
  • Sliding friction
  • Damping
  • Stiffness
  • Inertia

Step 2: Scene Layout Inference (Phase 2)

• Load the first frame image (you may use any or all view(s) of the first image).
  • phase_2/example_X/cam_Y/example_X_view_Y.png
• Estimate object instance locations within a 3×3 grid and identify object classes:
  • SceneLayout ← model(phase_2/example_X/cam_Y/example_X_view_Y.png)

Step 3: Simulation-Based Outcome Prediction

• Get the pusher velocity corresponding to example  phase 2:
  • pusher_velocity ← llmphy_twophase_9_101_101.npy (called db below)
• Generate simulation code using:
  • code ← model(PhysicalParameters, SceneLayout, pusher_velocity)
• Ensure generated Python code conforms to the simulator interface (analogous to phase_2/example_0.py however for example_X.)
• Execute simulator for 200 time steps:
  • output ← simulator(code)
• Extract predicted upright object instances from simulator output.

Step 4: Evaluation

• Retrieve answer options for example X from db (llmphy_twophase_9_101_101.npy)
  • answer_options ← db[X]['phase_2']['answer_options']
• Retrieve ground-truth answers:
  • ground_truth ← db[X]['phase_2']['answers']
• Compare predicted upright objects with ground truth.
• Compute performance using Intersection-over-Union (IoU):
  • IoU = |Predicted ∩ GroundTruth| / |Predicted ∪ GroundTruth|
• Report mean IoU across all examples.

Other Resources

The code associated with the approach will be released separately on GitHub.

Citation

If you use the LLMPhy-TraySim dataset in your research, please cite our contribution:

@inproceedings{Cherian2026TraySim,
  author    = {Cherian, Anoop and Corcodel, Radu and Jain, Siddarth and Romeres, Diego},
  title     = { {LLMPhy}: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines},
  booktitle = { Twenty-Ninth Annual Conference on Artificial Intelligence and Statistics (AISTATS)}
  year      = {2026},
}

License

The LLMPhy-TraySim dataset, excluding Python files, is released under CC-BY-SA-4.0.
Created by Mitsubishi Electric Research Laboratories (MERL), 2026
SPDX-License-Identifier: CC-BY-SA-4.0

The Python files in LLMPhy-TraySim dataset (LLMPhy-TraySim/example_*/phase_*/example_*.py) are released under MIT License. 
SPDX-License-Identifier: MIT