MBBMD: Media Bias Bias-Mitigated Dataset

MBBMD (Media Bias Bias-Mitigated Dataset) is a dataset designed for bias detection in Spanish-language news articles. The dataset is structured hierarchically into two annotation levels, enabling research in binary bias classification as well as fine-grained bias categorization.

This dataset has been created to support Natural Language Processing (NLP) research, bias detection models, and media studies by offering a structured and annotated corpus of news articles covering diverse political perspectives and a range of bias types.

MBBMD consists of 100 news articles sourced from multiple Spanish-language media outlets, annotated using a perspectivist approach (LeWiDi) at the document level. The dataset is divided into two main phases, each containing training, testing, and control subsets.

MBBMD is structured into two levels of analysis:

1. Level 1: Document-level binary classification
   This level determines whether a news article is biased or not, based on majority vote agreement among annotators. It includes percentage scores reflecting the degree of agreement.

2. Level 2: Multilabel bias classification
   This level categorizes bias into five specific types: intentional bias, spin bias, statement bias, coverage bias, and gatekeeping bias. Each type includes binary majority vote annotations and percentage agreement scores from annotators.

To enhance annotation robustness, Counterfactual Data Augmentation (CDA) techniques were applied to a subset of the dataset. These modifications involve outlet swaps, entity swaps, and terminological changes, allowing for the assessment of how these factors influence annotator perceptions of bias.

Dataset File Structure and Field Descriptions

The Multilevel Bias Detection Dataset for Spanish Media (MBBMD) is organized into two main directories, corresponding to the two annotation phases:

• Phase 1 (Document-level annotations): Focused on binary and multilabel bias classification at the article level.
• Phase 2 (Sentence-level annotations): Provides fine-grained sentence-level bias annotations.

Each directory contains three subsets:

• train: The primary dataset for training models.
• test: A reserved subset for model evaluation.
• control: Contains articles modified using Counterfactual Data Augmentation (CDA) to analyze annotation robustness.

The dataset files are stored in TSV (Tab-Separated Values) format, encoded in UTF-8, ensuring compatibility with most data processing tools.

Phase 1: Document-Level Annotations

Files:

• phase_1/train_phase1.tsv
• phase_1/test_phase1.tsv
• phase_1/control_phase1.tsv

Fields in Phase 1 files

Each row represents a full news article annotated for bias.

• topic: The topic of the article (e.g., Pedro Sánchez Investiture, Barcelona Amnesty Protest).
• text_id: A unique identifier for each article.
• title: The headline of the article.
• text: The full body of the news article.
• is_biased_majority_vote: Binary label (1 = biased, 0 = not biased), determined by the majority vote of annotators.
• is_biased_%: The percentage of annotators who classified the article as biased.

Phase 2: Multilabel Bias Classification at the Document Level

Files:

• phase_1/train_phase2.tsv
• phase_1/test_phase2.tsv
• phase_1/control_phase2.tsv

Fields in Phase 2 files

Each row corresponds to a news article, but with bias annotations broken down into five specific categories.

• topic: Topic of the article.
• text_id: Unique identifier for the article.
• title: Headline of the article.
• text: Full text of the article.
• is_intentional_bias_majority_vote: Binary label (1 = present, 0 = absent) indicating whether intentional bias is detected.
• is_spin_bias_majority_vote: Binary label indicating the presence of spin bias.
• is_statement_bias_majority_vote: Binary label indicating statement bias.
• is_coverage_bias_majority_vote: Binary label indicating coverage bias.
• is_gatekeeping_bias_majority_vote: Binary label indicating gatekeeping bias.
• is_intentional_bias_%: Percentage of annotators who identified intentional bias.
• is_spin_bias_%: Percentage of annotators who identified spin bias.
• is_statement_bias_%: Percentage of annotators who identified statement bias.
• is_coverage_bias_%: Percentage of annotators who identified coverage bias.
• is_gatekeeping_bias_%: Percentage of annotators who identified gatekeeping bias.

These fields enable multilabel classification, allowing researchers to analyze different dimensions of bias simultaneously.