Semantic Distributed Caching Architecture for Large Language Models: A Multi-Layer Approach

Large Language Model (LLM) inference is a major computational and economic bottleneck in AI-powered applications. Traditional caches rely on exact string matching and cannot exploit the high rate of semantically equivalent queries. This paper presents a Three-Tier Semantic Distributed Caching Architecture for LLM question-answering systems. The architecture combines: (1) a local user-level LRU cache for personalized patterns, (2) a distributed Redis cluster with semantic hash indexing for cross-user deduplication, and (3) a persistent knowledge graph store with vector-based nearest neighbor search as semantic fallback. The AM-MQM multi-vector question matching algorithm is integrated to decide cache eligibility with 96% semantic accuracy. Deployed on the AnswerMe platform with 1.2 million monthly queries, the system achieves a 78% cache hit rate, reduces API costs by 91%, cuts average latency from 2.3 seconds to 180 milliseconds for cache hits, and maintains consistency across a multi-model arena (GPT-4, Claude, DeepSeek). We analyze cache warming, invalidation policies, and cost–latency trade-offs for production deployment.