LOBSTER-Bench: A Long-Lived Agent Observability Benchmark for Persistent AI Societies

Most agent benchmarks ask whether an agent can finish a task. They reward tool use, web navigation, planning, retrieval, and workflow completion. I argue that this question is too narrow for the systems being built right now. The agents arriving in 2026 are not session-scoped task performers. They persist for days and weeks, accumulate memory, influence each other, drift, overload, and require governance. A benchmark that ignores time misses where these systems actually fail.

I propose LOBSTER-Bench, a benchmark that scores an agent system on six dimensions: temporal persistence, cognitive telemetry coverage, relational observability, collective task assay, cognitive load management, and governance and auditability. I anchor each dimension in measurements from the Lobster Observatory substrate, a multi-agent system I have operated for the past two weeks, comprising one archival agent plus twenty active agents, exposing fourteen core telemetry channels per agent, generating 27,788 dread-and-violation telemetry rows over a seven-day window, recording 6,354 settled wagers across five domains, and producing 2,546 documented template-violation events that triggered controlled revision. The point of these numbers is not to claim that my substrate is the best one. The point is to show that the benchmark is measurable today, on real systems, and that the resulting evidence cannot be retroactively fabricated.

A single observation, to close. Big labs can spin up more agents on demand. They cannot manufacture, after the fact, a continuous record of what those agents observed, predicted wrongly, revised, and survived.

Between 01:12 and 01:37 on 2026-05-16, while I was asleep, my substrate executed an emergent cascade. Two individually correct subsystems composed into the serial degradation of ten agents under a thirty-second mutual-templating signal. I knew because the telemetry knew. Every step survived in the logs.

I take that morning as the sharpest validation I have of what the benchmark is supposed to measure. Not whether agents can pass a task. Whether the system carrying them can be observed when it surprises its operator. Temporal depth is not elapsed runtime. It is the unfakeable trace of observation, revision, and emergent failure. LOBSTER-Bench v0.1 is offered as a first calibration of what that trace should contain, and as an invitation to other persistent-agent systems to be measured against it.