Attribution Bias in Literary Style Evaluation: Comparing Human and AI Perceptions of Authorship

This dataset contains the complete experimental data for a large-scale study investigating attribution bias in literary style evaluation. The research compares how human evaluators (N=556) and AI language models (N=13) assess identical literary content differently based on perceived authorship (human-authored vs. AI-generated).

The study employed a three-condition experimental design using Raymond Queneau's "Exercises in Style" as literary stimuli across 30 distinct writing styles. Participants and AI models evaluated the same content under blind, open-label, and counterfactual conditions, revealing systematic attribution bias where identical text receives different quality assessments based solely on authorship labels.

Dataset Contents

This repository includes raw experimental data, quality-filtered datasets, AI model simulation logs, and processed analysis-ready files spanning the complete research pipeline. 

The compressed data folder (data.zip) should be extracted into this cloned GitHub repository (drop the unzipped folder at the same level of the analysis folder) to enable full replication of analyses through the provided Jupyter notebooks: 

style_and_prejudice/
├── analysis/
│   ├── 01_data_quality_analysis.ipynb
│   ├── 02_human_bias_analysis.ipynb
│   └── ...
├── data/  # ← copy here!
│   ├── literary_materials/
│   ├── logs/
│   └── responses/
└── README.md

Folder Descriptions

• responses/ --> core response data organized by evaluator type and processing stage:

==> input for notebook #1 (01_data_quality_analysis.ipynb) 

• • human_evaluators/raw/ --> original participant data (fully anonymized!) collected through web platform
    • questionnaire.csv --> demographics, attention checks, and screening responses
    • responses.csv --> all participant evaluations across experimental conditions

-------------------------------------------------------------------------------------------------------

==> generated by notebook #1 (01_data_quality_analysis.ipynb), used by notebook #2 (02_human_bias_analysis.ipynb)

• • human_evaluators/processed/ --> quality-filtered human participant datasets
    • valid_participants_dataset.csv --> participants meeting inclusion criteria (N=556)
    • excluded_participants_dataset.csv --> excluded participants with reasons

-------------------------------------------------------------------------------------------------------

==> generated by notebook #3 (03_run_ai_simulation.ipynb), used by notebook #4 (04_ai_bias_analysis.ipynb)

• • ai_evaluators/raw/ --> AI model simulation data
    • ai_participant_simulation_20250805_113946.csv --> complete AI evaluations across all models  

-------------------------------------------------------------------------------------------------------

==> generated by notebooks #2 (02_human_bias_analysis.ipynb) & #4 (04_ai_bias_analysis.ipynb), used by notebook #5 (05_comparative_analysis.ipynb)

• • processed_for_comparison/ --> final analysis-ready datasets for cross-evaluator comparison
    • human_responses_processed.csv --> processed human evaluation data
    • human_summary_for_comparison.csv --> Human bias summary statistics
    • ai_responses_processed.csv --> AI evaluation data formatted for comparison
    • ai_summary_for_comparison.csv --> AI bias summary statistics
    • human_experiment_stats.json & ai_condition_stats.json --> experimental condition metadata

-------------------------------------------------------------------------------------------------------

• logs/ --> processing logs of AI evaluator simulations:
  • ai_evaluators/ --> execution logs from AI simulation
    • ai_participant_simulation_20250805_113946.log --> API calls, and processing steps

-------------------------------------------------------------------------------------------------------

• literary_materials/
  • The experimental stimuli file in this folder (experimental_stimuli.xlsx) contains structured literary content across 30 style categories, enabling full replication pipeline. The dataset includes the GPT-4 generated stories alongside content from other contemporary language models, providing researchers flexibility to experiment with alternative literary materials. For complete replication of the original evaluation study, the placeholder literary content should be replaced with corresponding excerpts from Raymond Queneau's Exercises in Style (2012 New Directions edition, see here). The dataset enables complete reproduction of findings while respecting copyright restrictions on literary materials through placeholder text that demonstrates experimental methodology without redistributing protected content.

-------------------------------------------------------------------------------------------------------
Individual files are also provided separately for direct access to specific datasets.

Funding and Support

This work was supported by the Princeton Language and Intelligence (PLI) Seed Grant Program. Research conducted by Wouter Haverals and Meredith Martin, at Princeton University's Center for Digital Humanities.