Published May 30, 2026 | Version 1.0
Technical note Open

forge-harness: Engineering Methods for Robust AI Collaboration Harnesses

Authors/Creators

Description

This paper presents four complementary harness engineering methods addressing the primary failure modes in AI collaboration harnesses: steel-quench (adversarial structural validation), source-grounding-audit (phantom claim
detection), harvest-loop (session-to-harness self-evolution), and sim-conductor (pre-deployment transfer validation). Applied to forge-harness itself: 10 structural defects resolved (4 S-grade, 4 A-grade), phantom claim rate reduced from 6.4% to 0% (3/47 → 0/44), 100% skill reachability confirmed across 4 external personas, 80% HIGH-grade external contribution absorption rate.

Files

forge_harness_v1.0.pdf

Files (735.7 kB)

Name Size Download all
md5:e5e989c3a20688589bf61e72c316b55e
735.7 kB Preview Download

Additional details

References
Preprint: arXiv:2604.14228 (arXiv)
Preprint: arXiv:2605.18747 (arXiv)
Preprint: arXiv:2604.21003 (arXiv)
Preprint: arXiv:2603.28052 (arXiv)
Preprint: arXiv:2605.28764 (arXiv)
Preprint: arXiv:2605.22733 (arXiv)
Preprint: arXiv:2605.27922 (arXiv)
Preprint: arXiv:2605.25665 (arXiv)
Preprint: arXiv:2605.26302 (arXiv)
Preprint: arXiv:2605.27575 (arXiv)
Preprint: arXiv:2605.28065 (arXiv)
Preprint: arXiv:2605.23904 (arXiv)
Preprint: arXiv:2604.25850 (arXiv)
Preprint: arXiv:2605.26112 (arXiv)
Preprint: arXiv:2605.29861 (arXiv)

Dates

Updated
2026-05-29
Adds prompt-regression, mcp-circuit-breaker, and token-budget-gate skill domains. Expands independent architectural convergence evidence from 3 to 6 implementations (SwarmHarness, HarnessAPI, Harness-Bench). Updates skill count to 28.
Updated
2026-05-30
Nine independent implementations (up from six) converge on the same outer-loop architecture — SkillOpt (arXiv:2605.23904), AHE (arXiv:2604.25850), and Scaling the Harness (arXiv:2605.26112) each independently formalized patterns forge-harness had already operationalized (synthesizer gate, regression blindspot, stale-but-confident detection). Three new gap skills added from PR #32: edit-manifest (prediction-verification loop), memory-hygiene (stale-but-confident detection), VCS-Layer Gate Enforcement (git pre-commit hook + marker file pattern). Skill count: 30 (26 fh-meta + 4 fh-commons). Quantitative summary table consolidated. Explicit positioning vs. performance optimization systems.
Updated
2026-05-30
Eleven independent implementations converge on the same outer-loop architecture — two new convergence points: Ptah (arXiv:2605.29861, stage-wise multi-agent verification convergent with 3-axis auto-gate) and Anthropic Dynamic Workflow (parallel sub-agent orchestration at scale, convergent with agent-composer Wave architecture). sim-conductor updated to task-adaptive persona selection (3-tier sourcing: installed plugins → built-in role directives → external fetch; scale 3–16 parallel agents). Model-agnostic harness layer positioning added (§5.4): Base mode (Sonnet) sufficient for standard validation; Amplified mode (Opus orchestrator + Sonnet executors) extends to Dynamic Workflow-scale fan-out without changing the validation contract. Table 1 convergence count corrected to 11.
Updated
2026-05-30
Two new convergence points (total: 11): Ptah [arXiv:2605.29861] — stage-wise multi-agent verification convergent with 3-axis auto-gate; Anthropic Dynamic Workflow — parallel sub-agent orchestration at scale, convergent with agent-composer Wave architecture. sim-conductor: task-adaptive persona selection (installed plugins → built-in fallback → external fetch), scale 3–16 parallel agents. Model-agnostic positioning (§5.4): Base (Sonnet) for standard validation; Amplified (Opus orchestrator + Sonnet executors) for Dynamic Workflow-scale fan-out — same validation contract either way. Pipeline architecture diagrams added (Figures 1–2).

Software

Repository URL
https://github.com/chrono-code/forge-harness
Development Status
Active

References

  • Chen, X. et al. "98.4% of Claude Code is Harness Infrastructure." VILA-Lab, 2026. arXiv:2604.14228
  • Sylph.AI. "The Last Harness You'll Ever Build." 2026. arXiv:2604.21003
  • Ning, X. et al. "Meta-Engineering Frameworks for AI Agent Harnesses: Two-Pass Validation and Four-Way Arbitration." UIUC, 2026. arXiv:2605.25665
  • Liu, S. et al. "AgingBench: Measuring Harness Knowledge Degradation Across Sessions." 2026. arXiv:2605.26302
  • Min, K. et al. "Agyn: Signal-Driven Harness Adaptation for Infrastructure-as-Code." 2026. arXiv:2605.27575
  • Park, J. et al. "SpatialBench: Joint Evaluation of Model-Harness Pairs in Agent Performance." 2026. arXiv:2605.28065
  • Jose, E. "SwarmHarness: Skill-Based Task Routing via Decentralized Incentive-Aligned AI Agent Networks." Western Michigan University, 2026. arXiv:2605.28764
  • Jose, E. "HarnessAPI: A Skill-First Framework for Unified Streaming APIs and MCP Tools." Western Michigan University, 2026. arXiv:2605.22733
  • Yao, Y. et al. "Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows." Peking University / Qihoo360, 2026. arXiv:2605.27922
  • Zhang, X. et al. "SkillOpt: Skill Optimization via Selection-Split Validation and Rejected-Edit Feedback." Microsoft Research, 2026. arXiv:2605.23904
  • Wang, Y. et al. "Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses." Fudan University, 2026. arXiv:2604.25850
  • Gu, S. et al. "Scaling the Harness in Agentic AI." UC Berkeley, 2026. arXiv:2605.26112
  • Perez, E. et al. "Red Teaming Language Models with Language Models." 2022. arXiv:2202.03286
  • Bai, Y. et al. "Constitutional AI: Harmlessness from AI Feedback." 2022. arXiv:2212.08073
  • Maynez, J. et al. "On Faithfulness and Factuality in Abstractive Summarization." ACL 2020.
  • Ji, Z. et al. "Survey of Hallucination in Natural Language Generation." ACM Computing Surveys, 2023.
  • Shinn, N. et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." NeurIPS 2023.
  • Madaan, A. et al. "Self-Refine: Iterative Refinement with Self-Feedback." NeurIPS 2023.
  • "Code as Agent Harness." 43 authors. 2026. arXiv:2605.18747
  • Stanford IRIS Lab. "Meta-Harness: Automated Harness Optimization via Outer-Loop Execution." 2026. arXiv:2603.28052
  • Christi, R. harness-evolver. MIT License. GitHub: https://github.com/raphaelchristi/harness-evolver, 2026.
  • Zhang, C. et al. "Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation." Renmin University of China, 2026. arXiv:2605.29861