Published November 14, 2026 | Version 3.0

Logical AI OS: A Training-Free Governance Layer Outside the Model for Instruction Following — Benefits, Costs, Failures, and the Calibration of the Measuring Instrument Itself

Authors/Creators

  • 1. Independent Researcher

Description

Small language models often fail explicit user requests (length, number of items, format, required or forbidden words, grammatical person, answer language). We measure a training-free governance layer placed outside the model: it reads the requests with rules, checks the answer mechanically, repairs it when a request is not kept, and hands back the original answer when the repair fails. The loop exists in prior work; our contribution is measuring its benefits, costs and failures, and reporting how the measuring instrument itself was calibrated.

Results on four 4-bit Qwen-family models (3B, 7B, 14B, one 7B reasoning model) and 329 Chinese prompts:

1. Specification level: answers that keep every request the layer read rise significantly (all comparisons p<0.0001), by 34 to 66 of 329 under the corrected checker and 52 to 92 after a length-control fix. A generic redo message brings no significant gain.

2. User level (exploratory): on 90 repair-triggered prompts, two AI annotators found the layer produced an acceptable answer where the bare model did not more often than the reverse (49:7; 18:12, not significant).

3. Costs: 43-66% more generation time. Whole-answer rewriting makes a sizeable share of usable answers unusable.

4. Failures: rule-based reading reaches about 70% and 84% of a two-annotator ceiling on unseen phrasings. Unsatisfiable requests make the reasoning model loop. Model self-reports are unreliable (18 of 25 "fixed" items were not fixed).

5. The instrument: over half of the original checker's "violated" verdicts on structural requests were wrong. After correction the main result holds, and one comparison is downgraded.

Conclusions are limited to this Chinese material, the Qwen family and sizes up to 14B.

The governance implementation is not released. Data, model answers, per-item verdicts, annotations and the standalone checker are public: https://github.com/blackwing04/Logical-AI-OS (tag v3.0-data).

Abstract (Chinese)

  • 使用者對語言模型提出的明文要求(字數、條數、格式、必含或禁用的詞、人稱、作答語言等),小型模型經常沒有守住。本文量測一個不訓練模型、放在模型外面的治理層:以規則讀出使用者的要求,機械核對答案,沒守住時修補,修不好時交回原答案。這個迴圈本身已有前作;本文的工作是量它的好處、代價與失效,並公開量測工具自己被校準的過程。

    在四個 Qwen 系模型(3B、7B、14B 與一個 7B 推理模型,皆 4-bit 量化)與 329 篇中文題目上:

    1. 規格層:治理層讀到的要求全部守住的篇數顯著增加(各組比較皆 p<0.0001),修正後的量尺下增加 34 至 66 篇,修正控長並重跑後 52 至 92 篇。不帶要求內容的通用重做訊息沒有帶來顯著增加。

    2. 使用者層(探索性):在 90 篇觸發修補的題目上,治理層做出標註者可以接受、而裸模型沒有的答案,多於反過來的情形(兩位 AI 標註者分別為 49 比 7 與 18 比 12,後者不顯著)。

    3. 代價:每篇多花 43% 至 66% 的生成時間;整篇重寫會把相當比例原本可用的答案改成不可用。

    4. 失效:規則讀題在沒見過的說法上,對雙標註天花板約為 70% 與 84%;無法滿足的要求會讓推理模型打轉;模型的自述不可靠(說「已改」的 25 條中 18 條其實沒改到)。

    5. 量尺:原核對器在結構類要求上判「違反」者過半不實;修正後主要結論不變,有一項比較因此降級。

    結論只在這一批中文材料、Qwen 系、模型規模最高 14B 的條件下成立。

    治理層的實作不公開。資料、模型答案、逐篇判定、標註與獨立版核對器公開於:https://github.com/blackwing04/Logical-AI-OS (tag v3.0-data)。

Files

logical_ai_os_v3.pdf

Files (559.0 kB)

Name Size
md5:0b3f9526102988f0f5c77283a1c4e82d
559.0 kB Preview Download

Additional details

Identifiers

Related works

Continues
Technical note: 10.5281/zenodo.23226740 (DOI)
Is supplemented by
Software: https://github.com/blackwing04/Logical-AI-OS/tree/v3.0-data (URL)
References
Preprint: arXiv:2410.06458 (arXiv)