A dataset to assess Microsoft Copilot Answers in the Context of Swiss, Bavarian and Hesse Elections.

This dataset allows to assess the emerging challenges posed by Generative Artificial Intelligence, when doing Active Retrieval  Augmented Generation (RAG), especially when summarizing trustworthy sources on the internet. As a case study, we focus on Microsoft Copilot, an innovative software that integrates Large Language Models (LLMs) and Search Engines (SE) making advanced AI accessible to the general public. The core contribution of this paper is the presentation of the largest public database to date of RAG responses to user prompts, collected during the 2023 electoral campaigns in the Swiss, Bavaria and Hesse. This dataset was compiled with the assistance of a group of experts who posed realistic voter questions and conducted fact-checking of Microsoft Copilot's responses. It contains prompts and answers in English, German, French and Italian. All the collection happened during the electoral  campaign, between 21 August 2023 and 2 October 2023. The paper makes available the full set of 5,561 pairs of prompts and answers, including the quoted URLs for the source referenced in the answers. In addition to the dataset itself,  we provide 1374 answers labeled by a group of experts who rated the accuracy of the answers in providing factual information. This resource is intended to facilitate further research into the performance of LLMs in the context of elections, defined as "high-risk scenario" by the Digital Service Act (DSA).