The Non-Delegable Core: Designing Legitimate Oversight for Agentic AI

As artificial intelligence systems become increasingly autonomous and assume oversight roles over other AI systems, traditional models of governance are rapidly eroding. This paper introduces the concept of the non-delegable core—governance functions that must remain under human authority not because AI lacks technical capability, but because democratic legitimacy requires it. We identify an Accountability-Capability Paradox, where AI systems' very success in surpassing human capacity undermines our ability to oversee them meaningfully, and propose the Human-AI Governance (HAIG) framework—a dimensional model that reconceives oversight along three axes: decision authority, process autonomy, and accountability configuration. Rather than defaulting to recursive AI-monitoring-AI hierarchies that obscure responsibility and invite failure, HAIG establishes adaptive trust thresholds to maintain human comprehensibility and control where it matters most. We illustrate HAIG-enabled anticipatory, flexible, and stakeholder-responsive governance scenarios in critical domains like medical triage, autonomous vehicles, and content moderation. The paper concludes with policy recommendations and institutional innovations—including AI audit courts and algorithmic juries—that support hybrid governance systems capable of sustaining democratic legitimacy in the age of agentic AI.