TurkishLegalBench: A Comprehensive Multi-Task Benchmark Suite for Turkish Legal NLP
Description
TurkishLegalBench is a large-scale, multi-task benchmark suite specifically designed for Natural Language Processing (NLP) in the Turkish legal domain. The dataset consists of 38,009 authentic legal documents curated from Turkish high courts. It covers seven distinct legal tasks categorized under three pillars: Classification (TurkVerdict, TurkVenue, TurkCanon), Extraction (TurkChronos, TurkCite), and Reasoning (TurkCoherence, TurkAudit). This resource aims to bridge the gap in low-resource legal NLP for Turkish and provides a standardized evaluation framework for researchers.
Methods
The Tasks (The 3 Pillars)
We organize the benchmark into three pillars representing different levels of legal cognition:
I. The Gavel (High-Level Classification)
| Task | Description | Metric | Size (Train/Dev/Test) |
| TurkVerdict | Predict the judgment outcome (e.g., Affirmation, Reversal) from the case rationale. | Macro-F1 | 18k / 2.2k / 2.2k |
| TurkVenue | Identify the competent court chamber (Daire) based on case facts (36 classes). | Macro-F1 | 17.7k / 2.5k / 5.0k |
| TurkCanon | Classify legislative documents into types (Law, Regulation, Decree, etc.). | Macro-F1 | 6.4k / 0.9k / 1.8k |
II. The Quill (Information Extraction)
| Task | Description | Metric | Size (Train/Dev/Test) |
| TurkChronos | Identify the decision year of a case amidst distractor dates. | Accuracy | 19.4k / 2.7k / 5.5k |
| TurkCite | Extract citations (Law No. & Article No.) from unstructured text (NER). | Entity F1 | 10.9k / 1.5k / 3.1k |
III. The Scale (Legal Reasoning)
| Task | Description | Metric | Size (Train/Dev/Test) |
|---|---|---|---|
| TurkCoherence | Natural Language Inference (NLI) to check if the reasoning supports the verdict. | Macro-F1 | 4.9k / 0.7k / 1.4k |
| TurkAudit | Detect "legal hallucinations" and anachronistic citations (e.g., citing a 2016 law in 2010). | Weighted-F1 | 7k / 1k / 2k |
Benchmark Results
We evaluated baseline and domain-adapted models across all 7 tasks using the Test Set. The table below reports the primary metric for each task (Macro-F1 for classification, Accuracy for Chronos, and Entity-F1 for NER).
| Model | Verdict (m-F1) |
Venue (m-F1) |
Canon (m-F1) |
Chronos (Acc) |
Cite (F1) |
Coherence (m-F1) |
Audit (W-F1) |
| BERTurk | 75.3 | 82.0 | 92.9 | 88.8 | 45.8 | 53.1 | 97.4 |
| Legal-BERT (Eng) | 68.8 | 62.9 | 92.2 | 90.0 | 37.5 | 45.9 | 95.8 |
| XLM-RoBERTa | 72.0 | 64.0 | 52.5 | 84.9 | 41.9 | 33.0 | 96.8 |
| DistilBERT-TR | 70.6 | 62.5 | 87.1 | 82.6 | 41.6 | 33.2 | 97.4 |
| Longformer | 70.1 | 62.1 | 91.6 | 83.7 | 58.4 | 33.0 | 97.0 |
| BERT-TR-128k | 79.4 | 81.1 | 92.6 | 92.0 | 49.3 | 64.3 | 98.3 |
License
Under this license, you are free to:
- Share — copy and redistribute the material in any medium or format.
- Adapt — remix, transform, and build upon the material.
Under the following terms:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made.
- NonCommercial — You may not use the material for commercial purposes.
- ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
Note: The underlying raw texts (court decisions and laws) are public records. This license applies to the curated benchmark, annotations, and structured dataset created by the authors.
Citation
If you use TurkishLegalBench (data, code, or models) in your research, please cite our paper:
@article{erkan2026turkishlegalbench,
title={TurkishLegalBench: A Comprehensive Multi-Task Benchmark Suite for Turkish Legal NLP},
author={Erkan, Mehmet Ali and Yozgatl{\i}gil, Ceylan},
year={2026}
}
Files
TurkAudit.zip
Files
(252.1 MB)
| Name | Size | |
|---|---|---|
|
md5:183335423763df2a8691d2ccad32beb8
|
17.0 MB | Preview Download |
|
md5:5c83ee6df0e8203b945546aa808cbb00
|
62.6 MB | Preview Download |
|
md5:571d73e57e954ec6d860e31004c0de02
|
48.4 MB | Preview Download |
|
md5:4879bea0b19e81ebfa53fe8d894f57b2
|
43.5 MB | Preview Download |
|
md5:b3009f7e4ef82849bad674ec9e262534
|
12.4 MB | Preview Download |
|
md5:4b33802660a0ce3c6fb18a030fbeaef8
|
40.7 MB | Preview Download |
|
md5:419e09d73acc481d66d927ff46db4a71
|
27.6 MB | Preview Download |
Additional details
Software
- Repository URL
- https://github.com/mrkn7/TurkishLegalBench
- Programming language
- Python