Flash-Jacobian: Local Operator Analysis Reveals Depth-Dependent Geometry and Nonlinear Factuality in LLMs

The Linear Representation Hypothesis—the assumption that high-level concepts are encoded as linear directions in the representation space of large language models (LLMs)—underpins much of current interpretability research, including sparse autoencoders, representation engineering, and probing classifiers. We present Flash Jacobian, a scalable GPU system for extracting cluster-representative Jacobians (CRJs)—piecewise first-order operators that summarize local layer dynamics over token populations—and use it to probe the validity of this assumption across three architectures (Qwen-3.5-4B, Llama-3.2-3B, Phi-3-mini). We report three findings. (1) A universal depth-dependent geometry: local CRJ fidelity follows a U-shaped (bell-shaped) profile across all three models, peaking at middle layers and collapsing at late layers due to gate anisotropy rather than centroid distance. (2) Semantic entanglement in causal directions: clusters extracted by Flash Jacobian are descriptively clean (capturing morphological roles such as surname prefixes and function words), but causal interventions along their principal directions act primarily as boundary-token suppressors rather than semantic activators, with a $400\times$ imbalance in probability mass change. (3) Nonlinear factuality: linear probes trained on hidden states achieve chance-level AUC ($\approx 0.50$) for hallucination detection on HaluBench across all three models, whereas nonlinear classifiers operating on Flash Jacobian's geometric features (trajectory summaries, residual mismatch, gate statistics) achieve AUC $> 0.99$ within each model—but fail to transfer across models (pooled AUC $\approx 0.50$). Together, these results indicate that key representational structures in LLMs are nonlinearly entangled and model-specific, demonstrating that the linear assumption is fundamentally insufficient for the causal and factual phenomena studied here.