Attribution Bias in Literary Style Evaluation: Comparing Human and AI Perceptions of Authorship

This dataset contains the complete experimental data for a large-scale study investigating attribution bias in literary style evaluation across two complementary experimental designs. Study 1 compares how human evaluators (N=556) and AI language models (N=13) assess identical literary content (the original Queneau vs. GPT-4-generated versions of his exercises) differently based on perceived authorship (human-authored vs. AI-generated). Study 2 expands the content generation across 14 models, then examining bias patterns when evaluators judge material from every AI creator in the experimental matrix.

Both studies employed a three-condition experimental design using Raymond Queneau's "Exercises in Style" as literary stimuli across 30 distinct writing styles. Participants and AI models evaluated the same content under blind, open-label, and counterfactual conditions, revealing systematic attribution bias where identical text receives different quality assessments based solely on authorship labels.

Dataset Contents

This repository includes raw experimental data, quality-filtered datasets, AI model simulation logs, and processed analysis-ready files spanning the complete research pipeline for both experimental studies. Human participant response data can be found here in de-identified form (N=2,780 evaluations from 556 participants), with personally identifiable information removed in accordance with Princeton University IRB protocol #18320.

The compressed data folder (data.zip) should be extracted into this cloned GitHub repository (drop the unzipped folder at the same level of the analysis folder) to enable full replication of analyses through the provided Jupyter notebooks: 

style_and_prejudice/
├── analysis/
│   ├── 01_data_quality_analysis.ipynb
│   ├── 02_human_bias_analysis.ipynb
│   ├── 03_run_ai_simulation.ipynb            # Study 1: single creator, 13 AI evaluators
│   ├── 03b_run_ai_simulation_expanded.ipynb  # Study 2: cross-model design, 14x14 creators
│   ├── 04_ai_bias_analysis.ipynb             # Study 1: analysis
│   ├── 04b_ai_bias_analysis_extended.ipynb   # Study 2: analysis
│   ├── 05_comparative_analysis.ipynb         # Study 1: human vs. AI evaluator bias
│   ├── 06_surface_feature_confounds.ipynb   # Surface feature analysis (Study 1)
│   └── 07_llm_criterion_inversion_annotation.ipynb  # Criterion inversion coding & analysis
├── data/  # ← COPY HERE!
│   ├── literary_materials/
│   ├── responses/
│   ├── logs/
├── criterion_inversions_coding/
└── README.md

Folder Descriptions

• responses/ --> core response data organized by evaluator type and processing stage:

==> input for notebook #1 (01_data_quality_analysis.ipynb) 

• • human_evaluators/raw/ → original participant data (fully anonymized!) collected through web platform
    • questionnaire.csv → demographics, attention checks, and screening responses
    • responses.csv → all participant evaluations across experimental conditions

-------------------------------------------------------------------------------------------------------

==> generated by notebook #1 (01_data_quality_analysis.ipynb), used by notebook #2 (02_human_bias_analysis.ipynb)

• • human_evaluators/processed/ → quality-filtered human participant datasets
    • valid_participants_dataset.csv → participants meeting inclusion criteria (N=556)
    • excluded_participants_dataset.csv → excluded participants with reasons

-------------------------------------------------------------------------------------------------------

==> generated by notebook #3 and #3b (03_run_ai_simulation.ipynb, 03b_run_ai_simulation_expanded.ipynb), used by notebook #4 and #4b (04_ai_bias_analysis.ipynb, 04b_ai_bias_analysis_extended.ipynb)

• • ai_evaluators/raw/ → AI model simulation data
    • Study 1 (single AI-generator): ai_participant_simulation_20250805_113946.csv → complete AI evaluations across all 13 models (13 models × 30 styles × 3 conditions, 3 separate runs)
    • Study 1 robustness replications:
      • ai_participant_simulation_20250915_210237-awardwinningai-t07.csv → alternative labeling ("award-winning AI")
      • ai_participant_simulation_20250916_195700-symmetricallanguage-t07.csv → symmetric labeling (t=0.7)
      • ai_participant_simulation_20250917_211949-symmetricallanguage-t00.csv → symmetric labeling, deterministic (t=0)
    • Study 2 (cross-model design): ai_participant_simulation_expanded.csv → cross-model evaluations (14 evaluators × 14 creators × 30 styles × 3 conditions, 1 run)

-------------------------------------------------------------------------------------------------------

==> generated by notebooks #2 (02_human_bias_analysis.ipynb) & #4 (04_ai_bias_analysis.ipynb), used by notebook #5 (05_comparative_analysis.ipynb)

• • processed_for_comparison/ → final analysis-ready datasets for cross-evaluator comparison
    • human_responses_processed.csv → processed human evaluation data
    • human_summary_for_comparison.csv → Human bias summary statistics
    • ai_responses_processed.csv → AI evaluation data formatted for comparison
    • ai_summary_for_comparison.csv → AI bias summary statistics
    • human_experiment_stats.json & ai_condition_stats.json → experimental condition metadata

-------------------------------------------------------------------------------------------------------

• logs/ --> processing logs of AI evaluator simulations:
  • ai_evaluators/ → API execution logs from AI simulation
  •  Study 1:
    • ai_participant_simulation_20250805_113946.log
    • ai_participant_simulation_20250915_210237-awardwinningai-t07.log
    • ai_participant_simulation_20250916_195700-symmetricallanguage-t07.log
    • ai_participant_simulation_20250917_211949-symmetricallanguage-t00.log
  • Study 2:
    • ai_participant_simulation_expanded.log

-------------------------------------------------------------------------------------------------------

• literary_materials/
  • Study 1: experimental_stimuli.xlsx → GPT-4 generated stories (30 styles)
  • Study 2: experimental_stimuli_expanded.xlsx → multi-model generated stories (14 AI creators × 30 styles)

The experimental stimuli files contain structured literary content across 30 style categories, enabling full replication pipeline. The datasets include AI-generated stories from multiple contemporary language models, providing researchers flexibility to experiment with alternative literary materials. For complete replication of the original evaluation study, the placeholder literary content should be replaced with corresponding excerpts from Raymond Queneau's Exercises in Style (2012 New Directions edition, see here). The dataset enables complete reproduction of findings while respecting copyright restrictions on literary materials through placeholder text that demonstrates experimental methodology without redistributing protected content.

-------------------------------------------------------------------------------------------------------

==> used by notebook #7 (07_llm_criterion_inversion_annotation.ipynb)

• criterion_inversions_coding/ → systematic coding of AI evaluator rationales for criterion inversion analysis
  • llm_coders/ → compressed JSONL annotation files from three independent LLM coder models
    • criterion_inversion_annotations__openai__gpt-5.2.jsonl.zip → 5,880 paired rationale annotations
    • criterion_inversion_annotations__openai__gpt-4.1.jsonl.zip → 5,880 paired rationale annotations
    • criterion_inversion_annotations__anthropic__claude-sonnet-4.6.jsonl.zip → 5,880 paired rationale annotations
      
      Each record contains structured annotations for one paired rationale (open-label + counterfactual) using a six-criterion codebook (C1–C6): criterion presence, valence toward Version A and Version B (+1/0/−1), and grounding evidence snippets. All three coders annotated all pairs independently at temperature 0, blind to experimental conditions. Decompress before use.
      
      
  • human_audit/ → human inter-coder reliability data (98 paired rationales, 2 independent coders)
    • human_audit_coderA.csv → annotations from human coder A (98 pairs)
    • human_audit_coderB.csv → annotations from human coder B (98 pairs)
    • human_audit_combinedAB.csv → merged file with both coders' annotations side by side (196 rows)
      
      Two researchers independently coded a stratified sample of 98 paired rationales using the same codebook and blinding protocol as the LLM coders. These files serve as the human reference set for validating the LLM coding pipeline.

Individual files are also provided separately for direct access to specific datasets.

Funding and Support

This work was supported by the Princeton Language and Intelligence (PLI) Seed Grant Program. Research conducted by Wouter Haverals and Meredith Martin, at Princeton University's Center for Digital Humanities.