Evaluation Dataset for ChemGraph: An Agentic Framework for Computational Chemistry Workflows

This dataset provides scripts, reference data, and evaluation tools for benchmarking ChemGraph, an LLM-based molecular simulation framework. It includes outputs generated by four different language models: GPT-4o-mini, Claude-3.5-haiku, Qwen2.5-14B, and GPT-4o. 

Main Files and Descriptions

- data_from_pubchempy.json: Structured chemical information obtained from PubChemPy. Serves as an input dataset for each experiment

- manual_workflow.json: A manually constructed reference workflow representing true tool call sequences and outputs. Used for benchmarking LLM results.

- llm_workflow_[...].json: A JSON file containing tool-use outputs generated by different LLMs. Includes additional metadata such as model name, timestamps and system prompt.

** Update history:

• October 7th, 2025:
  • Uploaded the ChemGraph source code release associated with the manuscript.
• October 6th, 2025:
  • Uploaded plotting data in the manuscript: evaluation_plot_data.json.
• August 29th, 2025:
  • Expanded benchmark from 260 to 360. 
  • Reran all evaluations. Added GPT-4o multi-agent evaluation.