Let's Build a Ship: Why complex systems fail politely - and how humans can keep them afloat

Abstract

Modern systems increasingly assist humans in reasoning and decision-making, yet their failures are often subtle rather than catastrophic. Instead of clear errors, these systems exhibit polite continuation in the presence of contradiction: rules are acknowledged, corrections are agreed, and the same mistakes quietly recur.

Such behaviour is confusing and destabilising for human operators, even when no intent exists. Low-stakes, everyday interactions illustrate that the underlying failure is not rule ignorance but loss of state coherence combined with the absence of explicit mechanisms to stop, invalidate, and reassert authority.

Understanding and correction are distinct operations, and verbal agreement without structural correction is a dominant failure mode in probabilistic, context-bounded systems. Explicit externalisation plays a critical role in maintaining coherence in complex systems:

• preserving partial records and explicitly marking uncertainty as acts of safety rather than bureaucracy;
• creating stable reference points when memory, context, or continuity can no longer be trusted;
• tolerating incompleteness while resisting silent repair;
• allowing contradictions to be surfaced and authority to be reasserted;
• enabling controlled stand-down when coherence is lost;
• providing durable artefacts that support recovery and prevent repeated rediscovery.

Together, these practices enable recovery and trust in environments where continuation is otherwise favoured over correctness.