LLMs Languages Least Moderated: Testing Cross-National Moderation in the context of the EU and the US Elections on Chatbots

AI Forensics had previously exposed that Microsoft Copilot's answers to simple election-related questions contained factual errors 30% of the time. In collaboration with Nieuwsuur, we uncovered how chatbots can recommend and support the dissemination of disinformation as a campaign strategy. Following those investigations as well as a request for information from the European Commission, Microsoft and Google introduced “moderation layers" to their chatbots so that they refuse to answer election-related prompts.

This dataset was produced as part of project "LLMs: Languages Least Moderated" at the 2024 Digital Methods Summer School and Data Sprint, which AI Forensics facilitated to allow participants to evaluate and compare the effectiveness of these safeguards in different scenarios. In particular, we investigated the consistency with which electoral moderation was triggered, depending the language of the prompt and the electoral context.