Kharisara/mer-jit-llm-fgcs: ReplayBench-PG Reproducibility Package v1.1.0
Authors/Creators
Description
ReplayBench-PG Reproducibility Package v1.1.0
This release accompanies the manuscript:
ReplayBench-PG: A Fault-Aware Deterministic Replay Benchmarking Framework for Policy-Gated Multimodal AI Pipelines
Overview
ReplayBench-PG is a deterministic replay benchmarking framework for reproducible validation of execution-level properties in policy-gated multimodal AI pipelines.
The framework supports deterministic replay, policy-ablation benchmarking, invocation-boundary verification, controlled fault validation, worker-level replay reconstruction, cross-region cloud-job validation, and reproducibility artifact generation.
The benchmark uses a MELD-derived replay workload containing 11,351 decision points evaluated across:
- 5 workload fractions
- 6 policy modes
- 3 random seeds
- 4 worker configurations
for a total of 360 benchmark conditions.
What's Changed
Framework
- Renamed the framework to ReplayBench-PG.
- Updated documentation and repository descriptions to reflect the new framework name.
- Improved manuscript alignment with the Array submission.
Benchmark
- Replaced the proxy diagnostic policy with the deterministic risk_proxy policy.
- Updated benchmark outputs covering 360 deterministic replay conditions.
- Improved replay documentation and benchmark descriptions.
Controlled Fault Validation
Added compact RQ7 controlled fault-validation workflows supporting detection of:
- Action-flip faults
- Unauthorized invocations
- Trace-action corruption
- Dropped replay rows
- Duplicated replay rows
Cloud Validation
Updated Google Cloud Run validation outputs for:
- asia-southeast1
- us-central1
including:
- Matching SHA-256 replay hashes
- Cross-region replay consistency
- Zero unauthorized invocations during clean replay
Reproducibility
Updated:
- manuscript-ready tables
- figures
- benchmark summaries
- reproducibility manifests
- cloud metadata
- benchmark configuration files
- documentation
Reproducibility Scope
ReplayBench-PG validates:
- deterministic replay
- trace reproducibility
- policy-gated invocation behavior
- invocation-boundary enforcement
- replay scalability
- worker-level replay consistency
- cross-region cloud-job consistency
- controlled execution-fault detection
ReplayBench-PG does not evaluate:
- policy quality
- policy optimality
- conversational quality
- personalization
- clinical effectiveness
- content-level safety
- production deployment performance
Citation
If you use this software, please cite the accompanying manuscript together with the archived software release.
Files
Kharisara/mer-jit-llm-fgcs-v1.1.0.zip
Additional details
Related works
- Is supplement to
- Software: https://github.com/Kharisara/mer-jit-llm-fgcs/tree/v1.1.0 (URL)
Software
- Repository URL
- https://github.com/Kharisara/mer-jit-llm-fgcs