Supplementary Dataset and Benchmark Logs: From Semantic Retrieval to Conversational Agent

This repository contains the supplementary materials and experimental data supporting the research article: "From Semantic Retrieval to Conversational Agent: A Web-Based RAG Architecture for Interactive System Dynamics Modeling".

The dataset is divided into two primary components: the source model environment (search space) and the raw experimental benchmarks evaluating retrieval performance across different levels of user expertise and conversational search strategies.

1. Model Corpus, Queries, and Scenarios

This section contains the definitions, domain classifications, and configurations used to build the semantic search environment and simulate user interactions.

• System Dynamics Models: Contains the extracted, curated, and serialized structural definitions of 63 System Dynamics models. These models cover diverse application domains, including Ecology, Macroeconomics, Smart Cities, Agriculture, and Epidemiology.

• User Queries and Intents: A dataset contrasting authentic broad novice search intents (e.g., "Show me health-related models") with theoretically perfect, expert-formulated structured queries requiring specific domain vocabulary.

• Benchmark Scenarios: 37 standardized benchmark scenarios engineered to evaluate cross-disciplinary semantic and lexical search performance across the system.

2. Experimental Benchmarks

The benchmark execution logs provide a quantitative comparative analysis of different retrieval paradigms, running on a local AI ecosystem with direct CPU inference.

• File: conversational_rag_benchmark_metrics.csv: This file contains the aggregate Information Retrieval metrics (Precision@5, Recall@5, MRR, and nDCG@5) calculated for the 37 test scenarios.

• File: inference_latency_logs.csv: Documents the execution timestamps and hardware latency logs for the local ONNX inference engine, tracking the multi-turn conversational delays.

• File: ablation_study_p_values.csv: Contains the statistical hypothesis testing (paired t-tests) results validating the significance of the agentic retrieval improvements.

• File: contextless_retrieval_test.csv: Contains the isolated experimental data evaluating the impact of conversational memory (Method F).

Evaluated Methodologies (Ablation Study) The benchmark data tests the following six retrieval paths:

• Method A: Broad Intent (Direct Retrieval Baseline) using standard single-turn semantic search.

• Method B: Agentic Refinement (Real Multi-Turn Agent Path) representing the complete conversational architecture.

• Method C: Expert Upper Bound (Direct Retrieval) serving as the theoretical maximum performance for semantic search.

• Method D: Apache BM25 (Lexical over Expert Query) testing exact keyword matching.

• Method E: Expert Query via Agent (Single Agent Turn) to assess system robustness against over-complication.

• Method F: Refinement Query (Contextless Direct Retrieval) isolating the final agent query from the conversational history.

Key Finding - Retrieval Accuracy: The data demonstrates that replacing the static search baseline (Method A) with the Agentic Orchestrator (Method B) improves the nDCG@5 metric from 0.107 to 0.438, a >300% performance increase.

Key Finding - Lexical vs. Semantic Dynamics: The benchmarks reveal that under optimal conditions with expert queries, exact lexical matching (Method D) provides superior top-tier ranking, achieving an MRR of 0.878 compared to semantic methods.

Key Finding - Computational Latency: The logs document the expected latency overhead of local CPU processing; a complete multi-turn exploratory session (Method B) averages roughly 59.6 seconds, whereas structurally complete expert queries (Method E) execute in about 50.1 seconds.