Harbor: A framework for evaluating and optimizing agents and models in container environments

What's Changed

• Publish local launch tasks through the existing publisher by @scvance in https://github.com/harbor-framework/harbor/pull/3184
• fix(environments): name the actual build-context dir in the missing-definition error by @hellno in https://github.com/harbor-framework/harbor/pull/2854
• fix(codex): preserve web-search IDs in ATIF by @ayushnangia in https://github.com/harbor-framework/harbor/pull/2972
• fix(computer-1): send prompt_cache_key so OpenAI runs hit the cache by @neverSettles in https://github.com/harbor-framework/harbor/pull/3190
• fix(viewer): show all rewards in jobs list hover tooltip by @thealchen in https://github.com/harbor-framework/harbor/pull/3201
• fix(rewardkit): json_key_equals must not treat a missing key as null by @Rome-1 in https://github.com/harbor-framework/harbor/pull/3182
• fix(computer-1): execute native Anthropic, Bedrock, and Gemini batches by @erikqu in https://github.com/harbor-framework/harbor/pull/3204
• feat: stream trajectories and sandbox editor over SSH by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3171
• docs: explain streaming logs and sandbox files by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3172
• fix(multi-step): don't leak prior step's /tests and /logs/verifier into the next agent phase by @EachSheep in https://github.com/harbor-framework/harbor/pull/1961
• fix: avoid OverflowError in environment_content_hash for files >4GB by @axiom-of-choice in https://github.com/harbor-framework/harbor/pull/3011
• Add --use-static-ip flag for hosted launches by @scvance in https://github.com/harbor-framework/harbor/pull/3213
• Add Code Puppy agent (based on Pydantic AI) by @mpfaffenberger in https://github.com/harbor-framework/harbor/pull/3199
• Warn when separate verifiers exclude configured mounts by @ayushnangia in https://github.com/harbor-framework/harbor/pull/3193
• Add first-class Muse Code installed agent support by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3166
• Group RewardKit criteria by Python file by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/2947
• feat: declare and validate agent skills and MCP capabilities by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3220
• refactor: share SSH setup helpers by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3228
• chore: limit reviewer mentions to rewardkit by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3221
• feat(tensorlake): enforce compose network policy at the sandbox level by @cooleel in https://github.com/harbor-framework/harbor/pull/3225
• docs: document agent MCP and skills capabilities by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3248
• docs: replace stream screenshot with demo video by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3250
• docs: add v0.23.0 changelog highlights by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3251
• docs: correct prompt_template_path parameter name by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3252
• docs: fix leaderboard and network policy links by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3253
• feat: support Modal streaming by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3229
• docs: use Mintlify trees for task directory examples by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3242
• docs: add upgrade and nightly installation instructions by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3243
• docs: highlight optional environment directory for prebuilt images by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3245
• docs: link MCP and skills compatibility to agent capabilities by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3246
• refactor: move Windows capability into _validate_agent_capabilities by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3255
• fix(viewer): keep trial tab bar in view when switching tabs by @thealchen in https://github.com/harbor-framework/harbor/pull/3233
• Port Rewardkit docs to Mintlify by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3234
• docs: expand sidebar navigation groups by default by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3256
• docs: render sidebar sections as fixed headings by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3265
• refactor(cli): move add and remove under dataset with hidden aliases by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3261
• docs: redirect legacy documentation to Mintlify by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3259
• Bump harbor-rewardkit to 0.2.1 by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3270
• docs: correct hosted launch and custom-agent guidance by @scvance in https://github.com/harbor-framework/harbor/pull/3241
• docs: add docs MCP sidebar link by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3266
• fix: read goose thinking blocks from the thinking field by @alran in https://github.com/harbor-framework/harbor/pull/3274
• fix: preserve executable status in task archives by @scvance in https://github.com/harbor-framework/harbor/pull/3282
• fix(viewer): prevent sticky tabs from covering stream controls by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3284
• Add --launch support to trial and job regrade by @scvance in https://github.com/harbor-framework/harbor/pull/3237
• fix: install only missing agent system dependencies by @scvance in https://github.com/harbor-framework/harbor/pull/3277
• fix: record Hub dataset downloads by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3291
• fix (stream): write stream.json after agent setup by @cooleel in https://github.com/harbor-framework/harbor/pull/3257
• fix(antigravity-cli): load MCP servers declared in task.toml on agy >= 1.1.27 by @xa8zz in https://github.com/harbor-framework/harbor/pull/3095
• Update Rewardkit docs for per-file scoring by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3269
• Reject non-finite RewardKit scores by @ayushnangia in https://github.com/harbor-framework/harbor/pull/3218
• fix(rewardkit): don't score a missing CSV column as an empty match by @no-hup in https://github.com/harbor-framework/harbor/pull/3272
• Fail the trial when a GKE pod is gone instead of returning exit code 1 by @mgeorgaklis in https://github.com/harbor-framework/harbor/pull/3215
• Honor task build timeout for Modal Compose startup by @scvance in https://github.com/harbor-framework/harbor/pull/3305
• fix: close Daytona client before event loop shutdown by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3302
• docs: move News into the docs and migrate existing announcements by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3286
• docs: add June–September 2026 news announcements by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3287
• docs: ask fork PRs to allow maintainer edits by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3309
• feat: --effort by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3304
• fix: classify launch failures from harness events only by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3216
• feat: add a common disable_web_search agent option by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3308
• feat(antigravity-cli): add error markers, options by @yaoshengzhe in https://github.com/harbor-framework/harbor/pull/3307
• fix(beam): update sandbox integration by @luke-lombardi in https://github.com/harbor-framework/harbor/pull/3306
• fix(rewardkit): clean up judge processes on cancellation by @ayushnangia in https://github.com/harbor-framework/harbor/pull/3300
• Improve viewer evaluation tables, filters, and trial navigation by @alexgshaw in https://github.com/harbor-framework/harbor/pull/3311
• feat(rewardkit): add JEV judge and rubric criteria by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3325
• fix(hermes): setup-timeout floor, oneshot export, drop the 90-turn cap and compression override by @teknium1 in https://github.com/harbor-framework/harbor/pull/3321
• feat(cwsandbox): merge the wandb environment into cwsandbox by @matthoare117-wandb in https://github.com/harbor-framework/harbor/pull/3085
• feat(tensorlake): add streaming support for tensorlake sandboxes by @cooleel in https://github.com/harbor-framework/harbor/pull/3281
• fix(pi): support max thinking level by @scvance in https://github.com/harbor-framework/harbor/pull/3345
• docs(runta): restore Runta in Mintlify documentation by @SASUKE40 in https://github.com/harbor-framework/harbor/pull/3342
• feat(agents): add codex and opencode as simulated-user targets by @joebaumann in https://github.com/harbor-framework/harbor/pull/3343
• feat(pi-agent): add MCP, turn limits, and ATIF trajectories by @stackviolator in https://github.com/harbor-framework/harbor/pull/3317
• fix(pi): preserve thinking levels for custom endpoints by @scvance in https://github.com/harbor-framework/harbor/pull/3352
• docs: note streaming sandbox use for debugging by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3347
• docs: add missing streaming sandbox providers by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3348
• docs: remove legacy docs by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3354
• docs: document Pi capabilities in Mintlify by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3367
• fix(docker): add SELinux label support for bind mounts by @not-stbenjam in https://github.com/harbor-framework/harbor/pull/1991
• feat: harbor run diff by @kobe0938 in https://github.com/harbor-framework/harbor/pull/2430
• feat(trial): expose resolved timeout properties by @0oshowero0 in https://github.com/harbor-framework/harbor/pull/3388
• fix(viewer): show the RewardKit judge model by @ayushnangia in https://github.com/harbor-framework/harbor/pull/3375
• fix(exec): accept --plugin and --plugin-kwarg by @YanggWendy in https://github.com/harbor-framework/harbor/pull/3370
• fix(ack): remove temp Docker config dir after crane image check by @Vedant-Agarwal in https://github.com/harbor-framework/harbor/pull/2158
• feat(runta): support runtime disk size by @xieydd in https://github.com/harbor-framework/harbor/pull/3341
• fix(swe-agent): install coreutils for stdbuf by @tu-trinh-scale in https://github.com/harbor-framework/harbor/pull/3396
• fix(hf-sandbox): create task workdir before setup by @aganezov in https://github.com/harbor-framework/harbor/pull/3365
• feat: add prime environment provider by @xeophon in https://github.com/harbor-framework/harbor/pull/3377
• fix(hyperbrowser): honor Compose and configure image builder resources by @Dingway98 in https://github.com/harbor-framework/harbor/pull/3353
• feat: support organization sharing during publish by @scvance in https://github.com/harbor-framework/harbor/pull/3411
• feat: support sharing hosted jobs during launch by @scvance in https://github.com/harbor-framework/harbor/pull/3412
• feat(viewer): add Pareto and Outcomes views and refresh analysis controls by @alexgshaw in https://github.com/harbor-framework/harbor/pull/3399
• fix(openhands-sdk): pass the instruction through a file instead of argv by @asaraog in https://github.com/harbor-framework/harbor/pull/3420
• feat(environments): add Mosaic Sandbox environment by @nitinrao in https://github.com/harbor-framework/harbor/pull/2870
• feat(prime): support Docker Compose task stacks on Prime VMs by @hallerite in https://github.com/harbor-framework/harbor/pull/3429
• feat(codex): add opt-in hook trust bypass option by @tempoden in https://github.com/harbor-framework/harbor/pull/3402
• fix: remove stale IANA link check from allowlist example by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3437
• fix(hermes): route openai/<model> to Hermes's openai-api provider by @mszummer in https://github.com/harbor-framework/harbor/pull/3426
• feat(rewardkit): judge samples, guard, and structured criterion results by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3326
• fix(rewardkit): stop command descendants on timeout by @ayushnangia in https://github.com/harbor-framework/harbor/pull/3390
• fix(claude-code): drop background-task env vars Claude Code no longer reads by @YanggWendy in https://github.com/harbor-framework/harbor/pull/3430
• Preserve Claude subagent IDs in ATIF steps by @psurkov in https://github.com/harbor-framework/harbor/pull/3434
• Add the Smol Machines environment with streaming by @BinSquare in https://github.com/harbor-framework/harbor/pull/3346
• Add Goose reasoning effort support by @alran in https://github.com/harbor-framework/harbor/pull/3360
• Reject unsupported SSE transport for Goose by @alran in https://github.com/harbor-framework/harbor/pull/3362
• feat(swe-agent): expose MCP server configurations by @tu-trinh-scale in https://github.com/harbor-framework/harbor/pull/3432
• feat(swe-agent): expose reasoning effort by @tu-trinh-scale in https://github.com/harbor-framework/harbor/pull/3438
• Support immutable revisions for SWE-agent installation by @tu-trinh-scale in https://github.com/harbor-framework/harbor/pull/3094
• Translate custom provider hosts for Goose by @alran in https://github.com/harbor-framework/harbor/pull/3363
• Add Goose runtime controls by @alran in https://github.com/harbor-framework/harbor/pull/3364
• fix(rewardkit): keep a Codex judge grading when a stream error was recovered by @asaraog in https://github.com/harbor-framework/harbor/pull/3458
• fix(rewardkit): clean up cancelled judge setup by @ayushnangia in https://github.com/harbor-framework/harbor/pull/3374
• feat(devin): expose task MCP servers by @tu-trinh-scale in https://github.com/harbor-framework/harbor/pull/3485
• fix(antigravity-cli): let Harbor enforce the agent timeout by @mgeorgaklis in https://github.com/harbor-framework/harbor/pull/3497
• shorten documentation URLs and preserve legacy redirects by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3505
• fix(cli): accept import paths in task start-env --env by @kar137 in https://github.com/harbor-framework/harbor/pull/3455
• Fix separate verifier image precedence and fallback test uploads by @alexgshaw in https://github.com/harbor-framework/harbor/pull/3258
• fix(codex): record structured tool outputs as text and images in ATIF by @YanggWendy in https://github.com/harbor-framework/harbor/pull/3467
• fix(trial): preserve separate verifier environment start timeouts by @Charlie-Wang-03 in https://github.com/harbor-framework/harbor/pull/3491
• Docker: support GPU allocation via Compose device reservations by @Elfsong in https://github.com/harbor-framework/harbor/pull/3504
• feat(rewardkit): add cost estimation for agent and llm judges by @mreza-kiani in https://github.com/harbor-framework/harbor/pull/3501
• docs: update README examples by @kobe0938 in https://github.com/harbor-framework/harbor/pull/3516
• fix(environments): skip FIFOs and device nodes when extracting transfer archives by @hallerite in https://github.com/harbor-framework/harbor/pull/3500
• fix(daytona): fail broken snapshot builds fast instead of timing out by @jadenfix in https://github.com/harbor-framework/harbor/pull/3494
• Simplify RewardKit tests and improve coverage by @benediktstroebl in https://github.com/harbor-framework/harbor/pull/3514

New Contributors

• @hellno made their first contribution in https://github.com/harbor-framework/harbor/pull/2854
• @thealchen made their first contribution in https://github.com/harbor-framework/harbor/pull/3201
• @Rome-1 made their first contribution in https://github.com/harbor-framework/harbor/pull/3182
• @EachSheep made their first contribution in https://github.com/harbor-framework/harbor/pull/1961
• @axiom-of-choice made their first contribution in https://github.com/harbor-framework/harbor/pull/3011
• @mpfaffenberger made their first contribution in https://github.com/harbor-framework/harbor/pull/3199
• @alran made their first contribution in https://github.com/harbor-framework/harbor/pull/3274
• @xa8zz made their first contribution in https://github.com/harbor-framework/harbor/pull/3095
• @yaoshengzhe made their first contribution in https://github.com/harbor-framework/harbor/pull/3307
• @teknium1 made their first contribution in https://github.com/harbor-framework/harbor/pull/3321
• @SASUKE40 made their first contribution in https://github.com/harbor-framework/harbor/pull/3342
• @joebaumann made their first contribution in https://github.com/harbor-framework/harbor/pull/3343
• @stackviolator made their first contribution in https://github.com/harbor-framework/harbor/pull/3317
• @not-stbenjam made their first contribution in https://github.com/harbor-framework/harbor/pull/1991
• @0oshowero0 made their first contribution in https://github.com/harbor-framework/harbor/pull/3388
• @YanggWendy made their first contribution in https://github.com/harbor-framework/harbor/pull/3370
• @xieydd made their first contribution in https://github.com/harbor-framework/harbor/pull/3341
• @tu-trinh-scale made their first contribution in https://github.com/harbor-framework/harbor/pull/3396
• @aganezov made their first contribution in https://github.com/harbor-framework/harbor/pull/3365
• @xeophon made their first contribution in https://github.com/harbor-framework/harbor/pull/3377
• @asaraog made their first contribution in https://github.com/harbor-framework/harbor/pull/3420
• @nitinrao made their first contribution in https://github.com/harbor-framework/harbor/pull/2870
• @hallerite made their first contribution in https://github.com/harbor-framework/harbor/pull/3429
• @tempoden made their first contribution in https://github.com/harbor-framework/harbor/pull/3402
• @mszummer made their first contribution in https://github.com/harbor-framework/harbor/pull/3426
• @psurkov made their first contribution in https://github.com/harbor-framework/harbor/pull/3434
• @BinSquare made their first contribution in https://github.com/harbor-framework/harbor/pull/3346
• @kar137 made their first contribution in https://github.com/harbor-framework/harbor/pull/3455
• @Charlie-Wang-03 made their first contribution in https://github.com/harbor-framework/harbor/pull/3491
• @Elfsong made their first contribution in https://github.com/harbor-framework/harbor/pull/3504
• @jadenfix made their first contribution in https://github.com/harbor-framework/harbor/pull/3494

Full Changelog: https://github.com/harbor-framework/harbor/compare/v0.23.0...v0.24.0