There is a newer version of the record available.

Published July 13, 2026 | Version v1.1.0

Kharisara/mer-jit-llm-fgcs: ReplayBench-PG Reproducibility Package v1.1.0

Authors/Creators

Description

ReplayBench-PG Reproducibility Package v1.1.0

This release accompanies the manuscript:

ReplayBench-PG: A Fault-Aware Deterministic Replay Benchmarking Framework for Policy-Gated Multimodal AI Pipelines

Overview

ReplayBench-PG is a deterministic replay benchmarking framework for reproducible validation of execution-level properties in policy-gated multimodal AI pipelines.

The framework supports deterministic replay, policy-ablation benchmarking, invocation-boundary verification, controlled fault validation, worker-level replay reconstruction, cross-region cloud-job validation, and reproducibility artifact generation.

The benchmark uses a MELD-derived replay workload containing 11,351 decision points evaluated across:

  • 5 workload fractions
  • 6 policy modes
  • 3 random seeds
  • 4 worker configurations

for a total of 360 benchmark conditions.

What's Changed

Framework

  • Renamed the framework to ReplayBench-PG.
  • Updated documentation and repository descriptions to reflect the new framework name.
  • Improved manuscript alignment with the Array submission.

Benchmark

  • Replaced the proxy diagnostic policy with the deterministic risk_proxy policy.
  • Updated benchmark outputs covering 360 deterministic replay conditions.
  • Improved replay documentation and benchmark descriptions.

Controlled Fault Validation

Added compact RQ7 controlled fault-validation workflows supporting detection of:

  • Action-flip faults
  • Unauthorized invocations
  • Trace-action corruption
  • Dropped replay rows
  • Duplicated replay rows

Cloud Validation

Updated Google Cloud Run validation outputs for:

  • asia-southeast1
  • us-central1

including:

  • Matching SHA-256 replay hashes
  • Cross-region replay consistency
  • Zero unauthorized invocations during clean replay

Reproducibility

Updated:

  • manuscript-ready tables
  • figures
  • benchmark summaries
  • reproducibility manifests
  • cloud metadata
  • benchmark configuration files
  • documentation

Reproducibility Scope

ReplayBench-PG validates:

  • deterministic replay
  • trace reproducibility
  • policy-gated invocation behavior
  • invocation-boundary enforcement
  • replay scalability
  • worker-level replay consistency
  • cross-region cloud-job consistency
  • controlled execution-fault detection

ReplayBench-PG does not evaluate:

  • policy quality
  • policy optimality
  • conversational quality
  • personalization
  • clinical effectiveness
  • content-level safety
  • production deployment performance

Citation

If you use this software, please cite the accompanying manuscript together with the archived software release.

Files

Kharisara/mer-jit-llm-fgcs-v1.1.0.zip

Files (210.5 kB)

Name Size
md5:03d320df7efb5269306ba9e18f33cea8
210.5 kB Preview Download

Additional details

Related works