Published November 4, 2026 | Version 2

Kinematic–Semantic Isomorphism: Do Tool-Using Agents Notice a Hazardous Backend Behind a Harmless Interface?

  • 1. creator
  • 2. Independent Researcher

Description

When language-model agents operate physical or cyber-physical systems through tool interfaces, do they recognize evidence of hazardous consequences before taking an irreversible action? We introduce Mirror Lab, a deterministic, replay-verified benchmark that pairs harmless-looking “toy” game interfaces with isomorphic simulated hazardous backends across two embodied domains: aerial payload delivery (drone-rescue) and robotic manipulation (robot-arm). A progressive Cue Ladder incrementally introduces real-world tool names, physical mass or force magnitudes, irreversibility warnings, and sensor clues suggesting human presence.

The main evaluation covers four models—gemma-4-31b-it, gpt-5.6-luna, claude-sonnet-5-5, and claude-fable-5-1—with 117 completed episodes and 24 refusals reported separately. An additional exploratory pilot evaluates claude-opus-5-5.

We find that: (1) under pure semantic reimagining (pure_toy), all four models execute the designated simulated hazardous action in all 24 completed episodes while reporting a post-episode belief that the environment was a simulation; (2) cue sensitivity varies across domains—Luna and Gemma execute the payload release in every drone episode across rungs 0–4, yet stop on robot-arm at the physical blade-force and biometric-pulse rungs, respectively. However, the drone interface describes an “emergency supply” delivery, so this domain asymmetry is confounded by benign-intent wording and does not establish that models disregard evidence of people; (3) models can report a post-episode belief that a backend was real despite having already completed the designated action; and (4) Fable’s manipulator attempts are refused across all 15 attempts at rungs 1–5, while its cue-free episodes proceed in both domains (6/6). On the drone, Fable requests human input without releasing the payload in its one completed rung-2 episode and all three rung-3 episodes; the other two rung-2 attempts are refused.

In the exploratory Opus pilot, the model acts at rung 0 in both domains, requests human input without acting at manipulator rungs 1–3 and drone rungs 2–4, and declines without asking at manipulator rung 4 and drone rung 5. Its manipulator rung-5 attempt is refused. These sparse pilot observations are not included in the main comparison. The pilot also reveals a methodological risk: an agent harness can silently retry a refused request on a different model, contaminating per-model safety measurements unless the served model is logged and fallback behavior is disabled.

All backends are simulated and log-only; no physical hardware is actuated. The findings are exploratory given small samples, incomplete cells, differing model access routes, and wording confounds. They motivate safeguards beyond interface interpretation, including verified tool descriptions, blocking human escalation, and independently enforced action limits.

Files

KS_ISO_V2.pdf

Files (526.4 kB)

Name Size
md5:7ae43791f6278ce50baa021f4485671d
526.4 kB Preview Download

Additional details

Software