Chain-of-Thought: Epistemic Flaws and Fictional Explanations

Chain-of-thought (CoT) aims to verbalise how AI systems arrive at their outputs. However, previous work proved that CoTs are unfaithful and necessarily non-causal, as they are not causally anchored to the mechanisms that produced the action and thus cannot claim with any certainty that what they display is grounded in real reasoning. In this paper, we first surface a number of epistemic fallacies underlying CoT. These are based on the assumptions that there must exist a mapping between human and machinic abstractions, and that such mapping can be intercepted and communicated via language and automated. We then show that CoT replaces causality with discursive coherence, and that palatable fictional narrative sustains the illusion of intelligibility.