A Benchmark for Citation-Grounded Geographic QA

GroundedGeo v1.0.0

First official release of GroundedGeo, a benchmark for citation-grounded, freshness-aware, and conflict-aware geographic QA.

Dataset

• 200 queries across 5 hard-case buckets (40 each)
• 147 dev / 53 frozen test split
• 100% evidence coverage: every query includes gold_evidence with URLs + citation spans
• 85 multi-source queries for testing source preference & conflict handling

Frozen Benchmark Results (Test Split)

| System | Accuracy | Conflict Handled | |--------|----------|------------------| | Closed-Book LLM | 17.0% | 11.1% | | Naïve RAG | 79.2% | 11.1% | | Official-First RAG | 94.3% | 100.0% | | Conflict-Aware RAG | 92.5% | 88.9% |

Key Finding

Naïve RAG reaches 79.2% accuracy but fails on conflicting sources (11.1% conflict-handled). Adding official-source ranking improves overall accuracy to 94.3% and raises conflict handling to 100%.

Artifacts

• data/groundedgeo_v1.0.json — Full dataset (schema + metadata)
• data/groundedgeo_v1.0.jsonl — HuggingFace format
• data/groundedgeo_v1.0.csv — Spreadsheet view
• results/final_results.json — Frozen test metrics
• results/error_analysis.json — Error analysis
• paper/paper.tex — LaTeX paper source
• paper/paper_tables.tex — LaTeX tables

Reproducibility

Run baselines + evaluation from the repo (eval/) using the included splits and configs.

Citation

@misc{pandya2025groundedgeo,
  title={GroundedGeo: A Benchmark for Citation-Grounded Geographic QA},
  author={Pandya, Nidhi},
  year={2025},
  url={https://github.com/nidhip1611/GroundedGeo}
}

Links

• HuggingFace: nidhip1611/groundedgeo
• GitHub: nidhip1611/GroundedGeo