Published November 20, 2026 | Version v1

Retiring the Agent Persona: Fail-Closed, Catalogue-Constrained Process Orchestration as a Successor to Vertical LLM Agents: An Experience Report and Evaluation

Description

Enterprise automation built on large language models has predominantly followed the persona-agent pattern: a named agent persona owns a business domain, reasons through unconstrained prose, and executes tools in an iterative prompt loop. Recent empirical evaluations demonstrate that this pattern suffers from framework-level failure modes, non-deterministic control flow, and unprincipled governance. Consequently, systems engineering has begun shifting toward deterministic orchestration architectures.

 

In this experience report, we document the architectural retirement of seven vertical persona agents within a production-scale multi-tenant enterprise platform. We replaced these agents with a deterministic-first process engine that fundamentally inverts the control loop. A deterministic compiler maps natural-language user goals onto a registered catalogue of capabilities. The large language model operates solely as a bounded, untrusted fallback whose structural proposals are validated against the catalogue; any unrecognized capability or execution failure automatically demotes to an audited human task.

 

Our contributions are threefold:
  1. An empirical migration case study and defect post-mortem detailing a declarative skill registry breakdown (where 23 of 29 capability pointers silently failed to resolve) along with its startup-reconciliation remedy.
  2. The architecture of catalogue-constrained goal compilation enforcing a verifiable safety invariant: the orchestration engine cannot emit or execute a capability outside its verified catalogue.
  3. An evaluation of this safety invariant and its operational flexibility cost: 8/8 hostile-input cases contained (including strict zero-call containment on unverified tenant context), 24 representative enterprise goals compiled with zero hallucinated capabilities at a measured human demotion rate of 0.109, and a 271/276 passing test suite verified against the frozen prototype snapshot.
Artifact Note: Gathered against frozen research tag V2-Prototype. Repository archived as research-only. Accompanying systems architecture baseline: Console V4 (July 2026).

Files

PAPER.pdf

Files (367.3 kB)

Name Size
md5:f322d9e549777008e59aba7009c1fd18
367.3 kB Preview Download

Additional details

Related works

Continues
Preprint: 10.5281/zenodo.24399624 (DOI)

References

  • Zhang, L., et al. (2025). *PioAgent: Deterministic Finite-State Control for Reliable LLM-Based Conversational Systems*. arXiv preprint arXiv:2501.08921.
  • Sterling, M., & Vance, H. (2025). *Demoting the Master Control Program: Deterministic Supervision of LLM Fleets for Safe Software Construction*. In *Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE)*, pp. 412–424.
  • Kitzelmann, E., et al. (2025). *Symbolic Probabilistic Languages: Unifying Stochastic Generation with Formal Verification*. In *Advances in Neural Information Processing Systems (NeurIPS 38)*
  • Russell, S., & Norvig, P. (2020). *Artificial Intelligence: A Modern Approach* (4th ed.). Pearson
  • Wooldridge, M. (2009). *An Introduction to MultiAgent Systems* (2nd ed.). John Wiley & Sons
  • Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014). Power to the People: The Role of Humans in Interactive Machine Learning. *AI Magazine*, 35(4), 105–120
  • Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In *Advances in Neural Information Processing Systems (NeurIPS 33)*, pp. 9459–9474
  • Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. In *International Conference on Learning Representations (ICLR)*.
  • Park, J. S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. In *ACM Symposium on User Interface Software and Technology (UIST)*, pp. 1–22.
  • Packer, C., et al. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv preprint arXiv:2310.08560.
  • Hong, S., et al. (2024). MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In *International Conference on Learning Representations (ICLR)*.
  • Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. In *Advances in Neural Information Processing Systems (NeurIPS 36)*.
  • Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. In *Advances in Neural Information Processing Systems (NeurIPS 36)*
  • Edge, D., et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv preprint arXiv:2404.16130.
  • Guo, Z., et al. (2024). LightRAG: Simple and Fast Retrieval-Augmented Generation. arXiv preprint arXiv:2410.05779.
  • Anthropic. (2024). *Model Context Protocol (MCP) Specification*. Technical Report.
  • Microsoft. (2024). *Semantic Kernel: Enterprise Orchestration Framework*. GitHub repository.
  • Django Software Foundation. (2024). *Django: High-Level Python Web Framework*. Version 5.0.
  • Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design Science in Information Systems Research. *MIS Quarterly*, 28(1), 75–105.
  • Inala, J. P., et al. (2024). Policy-as-Prompt: Constraining Foundation Model Behavior via Program Synthesis. In *ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)*.
  • Zhang, J., et al. (2024). ShieldAgent: Automated Guardrails and Enforcement for Multi-Agent Systems. arXiv preprint arXiv:2408.12034.
  • Significant Gravitas. (2023). *AutoGPT: An Autonomous GPT-4 Experiment*. GitHub repository.
  • Li, G., et al. (2023). CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In *Advances in Neural Information Processing Systems (NeurIPS 36)*.
  • Wang, G., et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv preprint arXiv:2305.16291.
  • Wu, Q., et al. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv preprint arXiv:2308.08155.
  • Moura, J., et al. (2024). *CrewAI: Framework for Orchestrating Role-Playing Autonomous AI Agents*. Documentation and Technical Reports.
  • LangChain. (2024). *LangGraph: Building Stateful, Multi-Actor Applications with LLMs*. Technical Documentation.