Published October 25, 2026 | Version r03

Optimization and Discovery in AI Scientist Systems: Tracing Evidence, Claims, and Decisions

Authors/Creators

  • 1. WeDataLab

Description

Scientific discovery depends not only on obtaining a better result but also on deciding what the result establishes and what to investigate next. We trace those decisions across twenty-one reported AI Scientist configurations using Servo's functional roles and the decision vocabulary of partially observable Markov decision processes and Bayesian experimental design. Five source-checked accounts anchor the analysis. ScientistTwo reports improvement on five of six metrics but withholds credit from the proposed multicore mechanism after ablation: the performance finding stands while the contribution claim does not. Adam's growth-curve evidence updates a logical model, while the cited twenty-hypothesis account does not specify an information-based test ordering; PiEvo states a distinct rule for selecting surrogate queries about candidate principles. Coscientist's separate episodes resolve an implementation error and optimize reaction conditions against precollected condition-yield data. Robin's drug ranking informs human selection for an initial assay, while its subsequent RNA-seq proposal seeks evidence about the remaining transcription question. These are differences in what an observation licenses and what the next action is for, not stages of scientific maturity. Our contribution is a source-grounded determination of those links, their actors, and their unresolved warrants within the reported accounts. It neither estimates discovery performance nor treats information gain as a universal objective; an automated catalogue map supplies context rather than validation.

This manuscript is a revised and expanded version of "Formalizing AI Scientist Systems: A Component Framework and Theoretical Analysis," published at CAISc 2026. The title and content have been revised. The present version has not undergone a separate peer review.

Conference version: https://openreview.net/forum?id=ItSw72nR4c

Files

arxiv_submit.pdf

Files (587.3 kB)

Name Size
md5:20b945729b28489babe39c56d0020eca
234.0 kB Preview Download
md5:a4e3e5da8d78b74382a2cd312bb1e74e
353.2 kB Preview Download

Additional details

Related works

Is new version of
Conference paper: https://openreview.net/forum?id=ItSw72nR4c (URL)