Pakistan Supreme Court and Islamabad High Court Judgment and Headnote Dataset for Abstractive Legal Text Summarization

Overview

This dataset contains paired judicial records and human-written headnotes collected for research on abstractive legal text summarization. The records were obtained from the Supreme Court of Pakistan and the Islamabad High Court. Each dataset item contains a court judgment and its corresponding headnote, allowing machine learning models to learn how lengthy judicial documents can be converted into shorter and more informative summaries.

The dataset was prepared as part of the research presented in the paper titled Text Summarization from Judicial Records using Deep Neural Machines. The work focuses on the use of transformer-based neural models for summarizing long legal documents from Pakistan.

Purpose of the dataset

Court judgments contain important legal facts, arguments, references, findings, and decisions. These documents can be lengthy and may require considerable time to read and analyse. A concise headnote can help a reader understand the main facts, legal issues, applicable laws, reasoning, and final outcome of a case.

The dataset was developed to support the design and evaluation of systems that can automatically generate summaries of Pakistani court judgments. It can be used to train models that receive a complete judgment as input and generate a shorter legal headnote as output.

The associated research explains that legal text summarization can help lawyers, judges, researchers, and court personnel obtain an overview of a judgment without initially reading the entire document. It may also support faster access to relevant legal information.

Dataset sources

The judicial records and their corresponding headnotes were manually downloaded from the following courts:

Supreme Court of Pakistan

The Supreme Court of Pakistan is the highest appellate court in the country. The dataset includes judgments and corresponding headnotes obtained from publicly accessible judicial records.

Islamabad High Court

The Islamabad High Court dataset contains judgments and corresponding headnotes from cases decided within the jurisdiction of the Islamabad High Court.

The paper reports that judgments and headnotes were collected from records covering the period from 1991 to the time of dataset preparation. The manual collection process was used to reduce duplicate documents because a single judgment could appear in more than one legal journal or source.

Dataset size

The dataset contains a total of 429 judgment and headnote pairs.

Direct inspection of the uploaded archive shows the following distribution:

Every judgment has a corresponding headnote file with the same numerical identifier.

Directory structure

The archive is organised into three main data splits:

Training

This directory contains the examples used for model training.

Validation

This directory contains the examples used for model selection, parameter adjustment, and performance checking during development.

Testing

This directory contains examples intended for the final evaluation of trained summarization models.

Each split contains two subdirectories:

case-description

This directory contains the longer judicial text used as the source input for a summarization system.

hd

This directory contains the corresponding human-written headnote used as the reference summary.

The dataset therefore follows a paired input and output structure:

Input: Judicial case description or judgment text

Target output: Corresponding human-written legal headnote

File format

All records are provided as plain text files using the .txt format.

A judgment file and its matching headnote can be connected through their numerical file identifier. For example, a judgment file ending in a particular number corresponds to the headnote file carrying the same number.

The plain text structure makes the dataset suitable for use with Python, PyTorch, TensorFlow, Hugging Face Transformers, and other natural language processing frameworks.

Data preparation process

The judgments and headnotes were downloaded manually. Manual collection was selected to help avoid duplicate judgments that may have been published in different journals or repositories.

The headnote was separated from the main judgment text so that the judgment could be used as the source document and the headnote could be used as the target summary.

The preparation process also included:

• Separating judgments from their corresponding summaries
• Converting documents into paragraphs and sentences
• Applying regular expressions for legal abbreviations
• Improving sentence segmentation
• Manually checking the processed text for errors
• Dividing the records into training, validation, and testing data

The paper reports that the Maltese Language Software Services sentence splitter was used during paragraph and sentence processing. Regular expressions were then applied to handle domain-specific abbreviations and legal sentence structures. Manual checking was performed to identify processing errors.

Nature of the summaries

The target summaries are legal headnotes associated with the original judgments. These headnotes provide a condensed representation of the case and may include:

• Names of the parties
• Court and judge information
• Case number
• Date of decision
• Relevant legislation
• Legal provisions
• Main factual background
• Questions considered by the court
• Important legal principles
• Findings of the court
• Final case outcome

Unlike a basic extractive summary, a legal headnote may reorganise or restate information from the judgment. It therefore provides a suitable target for abstractive summarization research.

The paper describes the reference headnotes as human-written summaries and compares them with summaries generated by the trained model. The examples on page 5 show that a generated headnote can follow the expected legal structure while remaining fluent and factually relevant.

Text length characteristics

The associated study reports that the source documents have a median length of approximately 1,933 tokens. The 98th percentile of the input documents is approximately 6,101 tokens.

The target summaries have a median length of approximately 374 tokens. The 90th percentile of the summary length is approximately 385 tokens.

These characteristics make the dataset suitable for research involving long document summarization. The study used an input limit of 8,192 tokens and an output limit of 512 tokens during model fine-tuning.

Associated modelling approach

The dataset was evaluated using Longformer Encoder Decoder models designed to process long input sequences.

The associated research compared:

• led-base-8192
• legal-led-base-8192

The legal model was based on legal-led-base-16384, which had previously been trained using legal documents. It was further fine-tuned for the Pakistani judicial summarization dataset using a maximum input length of 8,192 tokens and a maximum output length of 512 tokens.

The use of a legal domain model helped reduce problems caused by legal terminology and uncommon vocabulary. The Longformer attention mechanism was also suitable for processing documents longer than those normally supported by standard transformer architectures.

Benchmark results

The models were evaluated using ROUGE scores.

For the Supreme Court of Pakistan and Islamabad High Court dataset, the legal-led-base-8192 model achieved:

• ROUGE-1: 53.11
• ROUGE-2: 32.12
• ROUGE-L: 34.09

The general led-base-8192 model achieved:

• ROUGE-1: 48.95
• ROUGE-2: 25.27
• ROUGE-L: 31.22

These results indicate that a model previously trained using legal domain material performed better than a model trained primarily using general news data.

Potential applications

This dataset may be used for:

• Abstractive legal text summarization
• Long document summarization
• Training transformer-based summarization models
• Legal natural language processing
• Automatic headnote generation
• Transfer learning in the legal domain
• Sequence-to-sequence learning
• Evaluation of Longformer models
• Legal document indexing
• Judicial information retrieval
• Court judgment analysis
• Legal research support systems
• Development of legal question answering systems
• Creation of searchable judicial databases
• Comparative research involving courts and legal jurisdictions

Intended users

The dataset may be useful for:

• Natural language processing researchers
• Machine learning researchers
• Legal informatics researchers
• Computational linguists
• Legal technology developers
• Law students and academics
• Judicial analytics researchers
• Researchers studying Pakistani legal documents
• Developers of document summarization systems

Limitations

The dataset represents judgments collected from the Supreme Court of Pakistan and the Islamabad High Court. It should not be treated as a complete representation of all courts, tribunals, case types, languages, or legal jurisdictions in Pakistan.

The records are primarily in English. Urdu judgments and multilingual court documents are not clearly represented in the released archive.

The quality and style of the headnotes may vary because they originate from different judgments and reporting contexts. Some headnotes may be more detailed than others.

Legal judgments can exceed the input length supported by a model. Documents longer than the selected token limit may require truncation, segmentation, or another long document processing strategy.

Generated summaries may contain incomplete sentences when an output token limit is reached. The paper identifies output length as one of the limitations of the proposed approach.

ROUGE scores measure word overlap between a generated summary and a reference summary. A generated summary may be factually correct while receiving a low ROUGE score because it uses different wording or sentence structure. Human evaluation by legal experts may therefore be needed for a complete assessment of summary quality.

Ethical and responsible use

The dataset contains public judicial material that may include personal names, case details, addresses, organisations, allegations, and other potentially sensitive information.

Users should handle the records responsibly. The dataset should not be used to harass individuals, create misleading legal profiles, make automated decisions about people, or present generated summaries as official legal advice.

Any system trained using this dataset should be evaluated for factual accuracy. Important legal details should be checked against the complete judgment before being used in research, professional practice, or public communication.

Citation

Users of this dataset should cite the associated conference paper:

Ayesha Sarwar, Seemab Latif, Rabia Irfan, Adnan Ul-Hasan, and Faisal Shafait. Text Summarization from Judicial Records using Deep Neural Machines. 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME. IEEE, 2022. DOI: 10.1109/ICECCME55909.2022.9987795.