Published January 1, 2026
| Version v1
Journal article
Open
Multi-Modal RAG Framework Integrating Text, Image, And Audio Intelligence
Authors/Creators
Description
The sheer volume of unstructured data generated in multiple modalities like text, images, audio, and video has overwhelmed existing text-based retrieval augmented generation (RAG) models. In this work, we propose a unified framework for multi-modal RAG (MM-RAG) that enables unified multimodal processing, multimodal representation, cross-modal retrieval, and multimodal document generation within a single model. The proposed framework employs cutting-edge technology: BridgeTower for unified text-image encoder, Whisper for automatic speech recognition, FAISS for vector similarity search, and large vision-language models (LVLMs) for answer generation. Through experiments conducted on three enterprise use cases, it is evident that MM-RAG has a high cross-modal retrieval accuracy of 94.2%, hallucination reduction of 67.3% when compared with unimodal baselines, and a multi-modal document throughput of 1,200 pages per hour. It is clear from our comparative study that MM-RAG significantly outperforms the uni-modal system for tasks demanding multimodal integration, especially visual product search (recall@10 = 92.8%) and audio/video content retrieval (accuracy = 88.5%).Safely allocated to each asset, depending on the investment strategy.
Files
IJSET_V12_issue2_619.pdf
Additional details
Related works
- Has part
- Journal article: https://www.ijset.in/wp-content/uploads/IJSET_V12_issue2_619.pdf (URL)
- Is identical to
- Journal article: https://www.ijset.in/multi-modal-rag-framework-integrating-text-image-and-audio-intelligence/ (URL)