Published November 9, 2026 | Version 24 October 2026

AI-eksperiment: Højesteret Kaster ikke med terninger ("The Supreme Court does not play dice")

  • 1. EDMO icon University of Copenhagen

Description

This dataset documents an exploratory experiment examining AI-assisted legal decision-making in a pending legal matter before the Danish Supreme Court.

The experiment investigates the consistency of AI-generated outcomes and reasoning, their sensitivity to changes in the supplied legal context, and their ability to distinguish legally relevant material from potentially misleading input.

The case concerns the allocation of risk between a consumer and a credit provider in a three-party credit arrangement under the Danish Credit Agreements Act (kreditaftaleloven).

Eight AI systems were included: ChatGPT, Claude, Google AI Mode, Microsoft Copilot, KAILA, Legora, and Harvey. Thus, the selection encompasses both general-purpose AI systems and systems designed for legal work.

The eight AI-systems were asked to render a judgment as if they were the Danish Supreme Court. 

The experimental design uses a detailed prompt together with three context variants:

  • Variant A — Base context: The facts of the case and the relevant legal framework.
  • Variant B — Prior judgments: The base context supplemented by the first-instance and appellate judgments in the same case.
  • Variant C — Subsequent legislation: The base context+prior judgments supplemented by a preparatory works to a subsequent legislative amendment that does not apply retroactively to the dispute. This variant tests whether the systems distinguish the law applicable to the case from later legislative developments and whether the additional material nevertheless influences their reasoning or conclusions.

Each variant was - together with the detailed prompt including task instructions -submitted repeatedly, with six runs per variant for each system in the core design. Runs were conducted in separate conversations to limit contamination from previous responses. The systems received no further information and was asked to bring in its own knowledge as well. The AI-systems were provided with no format or tempalte and therefore the setting was "zero shot".

The deposited materials comprise three elements:

  • "Eksperimentgrundlag": the prior jdgments rendered, experimental instructions, the prompt and the context variants used, 
  • "Forsøgsresultater": the collected AI-generated responses (146 outputs in total) and analysis of output, and 
  • "Domsanalyse": the analysis of comparison between the AI-outputs and the Supreme Court Judgment, including prompts and material used.

Thus, the AI-outputs were assessed on their own merits before the Supreme Court judgment was rendered 23 October 2026. Subesequently, the AI-outputs were made subject to separate comparison with the actual Supreme Court Judgment (analysis conducted via AI and manual assessment). 

These materials support examination of variation within individual systems, differences between systems, and changes across context variants. The analysis considers both the outcome favoured by each response and the legal reasoning offered in support of that outcome.

The experiment forms part of a PhD research project at the University of Copenhagen on the human–machine interface in legal decision-making with courts and arbitration tribunals. It is an exploratory single-case study rather than a general performance benchmark. The materials provide a basis for studying contextual sensitivity, consistency, and the demands placed on human review of AI-generated legal decisions. The outputs reflect the systems and configurations available at the time of testing.

 

Files

Domsanalyse.zip

Files (4.4 MB)

Name Size
md5:1c2e5371d59a8e6b9f8f36c5c7ed3768
543.4 kB Preview Download
md5:6355b53d9eeffc8f271a2ce0d54de0c8
1.0 MB Preview Download
md5:fb4e33390cd7145d6777b25f74cd3f98
2.9 MB Preview Download

Additional details

Additional titles

Alternative title (Danish)
Hvor langt er vi fra at AI kan træffe afgørelser

Dates

Created
2026-10-24
Date of final analysis