Large-Scale Evaluation of MaxEntRAG-Flow: Incremental Evidence Structures for Real-Time Context Compression and Joint Probabilistic Graph Retrieval in Long-Context LLMs

This preprint evaluates MaxEntRAG-Flow as a context-compression mechanism for long-context retrieval-augmented generation. MaxEntRAG-Flow was originally formulated as an LLM-free, source-anchored sparse retrieval framework for cost-efficient graph-augmented RAG. This version studies its applicability to conversational and continuously updated retrieval settings, where update work can be performed off the user-facing path while retrieval latency, prompt length, and interaction cost directly affect user experience.

The paper evaluates the method across LongBench, ZeroSCROLLS, MuSiQue, LooGLE, and NaturalQuestions using a local Qwen-3B-Instruct setup. It reports full probability-threshold sweeps from τ = 0.1 to τ = 1.0, comparisons against Oracle, prompt truncation, sliding-window, BM25-style VectorRAG, and fixed Top-3 retrieval baselines, as well as update latency, retrieval latency, token reduction, budget-sweep behavior, and structural ablation results.

The main finding is that the source-anchored probabilistic evidence structure can substantially reduce retrieved context size and retrieval latency while preserving useful QA performance across several long-context benchmarks. The results also show benchmark-specific transition behavior: some datasets exhibit stable compression regions, while others show sharp context-collapse regimes at higher probability thresholds. The paper positions this behavior as a threshold-controlled connectivity effect in the underlying term-span co-occurrence graph.

This release includes the preprint manuscript, source files, figures, and supporting experimental tables used in the evaluation.