CloudGraph: Evaluation Dataset for Graph-Grounded Verification of LLM-Generated Root Cause Analysis in Kubernetes

Evaluation dataset accompanying an MSc study of whether the dependency graph
that an orchestrated system already maintains can be used to distinguish a true
root-cause explanation from a fabricated one.

A graph-based verifier (Graph-Provenance Claim Scoring) is compared against a
self-consistency baseline. Both score the same claims from the same generation,
so the comparison isolates the mechanism.

DESIGN

A complete 3x6 factorial: three microservice systems (Online Boutique, Sock
Shop, Train Ticket) crossed with six fault families (CPU, memory, disk, network
delay, packet loss, socket exhaustion), every cell filled exactly once. Each
scenario is run under three retrieval conditions (none, raw, hybrid), giving 54
runs.

CONTENTS

18 scenarios, 54 runs, 1,950 scored claims, 1,057 LLM calls, zero fallbacks or
timeouts. Of the 1,950 claims, 93 (4.8%) carry a correctness label: 36
consistent and 57 contradicted. The graph verifier flagged 79.3% of claims
unsupported; the self-consistency baseline flagged 53.0%.

  logs/       54 gzipped run logs — the raw evidence. Every figure derives
              from these.
  results/    claims.csv (one row per scored claim), MANIFEST.json, and two
              analysis documents.
  traces/     Nine narrative walkthroughs of individual runs.
  scenarios/  The 18 scenario definitions.
  scripts/    build_claims_csv.py rebuilds claims.csv from the logs;
              label_claim_correctness.py is the deterministic labeller.

REPRODUCIBILITY

claims.csv is regenerated from the raw logs by a committed script and is
deterministic: the same logs always produce the same file. Correctness verdicts
come from a deterministic Python labeller executed during each run. No step is
manual.

The logs themselves cannot be reproduced byte for byte, because generation runs
at temperature 0.8. Repeating one scenario-condition cell on an unchanged
configuration moved verifier concordance by up to 25.7 percentage points, so
single cells are uninformative and only pooled figures should be used.

SCOPE AND LIMITATIONS

No infrastructure or host-cluster data is present: scenario telemetry is seeded
from the upstream benchmark file, retrieval is scoped by scenario identifier,
and benchmark data is torn down between runs. No inferential statistics are
computed — one sample per cell does not support them, and all figures are
descriptive. Only 4.8% of generated claims can be adjudicated against the
benchmark metadata at all, which is itself reported as a finding rather than
treated as a defect. The affected service is supplied by the benchmark, so the
task is fault diagnosis rather than root-cause localisation.

PROVENANCE

Scenario telemetry is derived from RCAEval RE2 (MIT licensed, doi:10.5281/zenodo.14590730), which should be cited alongside this dataset.