SynthTrace: A Real-Time Pipeline for LLM Source Transparency and Overreliance Reduction

SynthTrace is a four-stage AI pipeline that classifies whether a Large Language Model's answer draws from information the user explicitly provided, or from patterns absorbed during training. That distinction is invisible in most AI interfaces today, and it has a direct impact on how much a user should trust a given response.

The core problem this work addresses is overreliance: users accepting AI-generated claims without scrutiny, even when those claims are hallucinated or factually wrong. Rather than trying to make the model more accurate, SynthTrace asks whether giving users basic source transparency information would cause them to calibrate their trust more appropriately.

A pilot simulation study of 100 claims found that it does. Blind trust in hallucinated AI answers averaged 62.4 out of 100 when no source information was shown. When SynthTrace surfaced a low credibility score and flagged the parametric source, that figure dropped to 31.2, a 50% relative reduction in misplaced trust. Overall average trust calibration shifted by +34.4 points across all claims.

The pipeline runs in real time via API and is compatible with any underlying LLM. A live working demo is available at https://synthtrace.vercel.app and the full source code is at https://github.com/MALIKSAAD-dev/synthtrace

This deposit includes the full research paper, the raw benchmark dataset (100 queries with per-query trust scores and source classifications), the complete prompt log for all test cases, and the exact script used to generate the dataset so the methodology is fully reproducible.