Published November 15, 2026
| Version v0.1
Dataset
Open
RA-Eval v0.1: A Chinese AI-Flavor Evaluation Framework with Calibrated LLM-Judge-Annotated Preference Pairs
Description
RA-Eval v0.1 — the evaluation framework and preference dataset accompanying the paper "Grounding Language Models in Speech Does Not Fix 'AI Flavor': Calibrated LLM Judges, Goodhart Traps, and a Preference-Tuning Fix" (arXiv preprint, 2026).
CONTENTS
1) ra-eval-v0.1-code-data.zip — benchmark code (feature library, human blind-eval protocol, calibrated LLM-judge pipeline), derived data (feature reference statistics, preference soft labels), result documents, and the paper draft. License: MIT (code) / CC-BY-4.0 (annotations).
2) ra-eval-v0.1-corpus.zip — human reference corpus: 86 Chinese college-entrance-examination (gaokao) essays plus judge-calibration sets with answer keys. License: non-commercial research use only, attribution required, takedown policy in LICENSE-data.txt. Copyright remains with the original anonymous student authors.
3) model_v5.tar.gz — merged 0.6B model weights (DPO trained on calibrated-judge preference labels, with an SFT anchor) and the LoRA adapter. Base model: Qwen3-0.6B (Apache-2.0); the fine-tuned weights are redistributed under the same license.
LICENSE MAPPING (this record lists three licenses)
- ra-eval-v0.1-code-data.zip — code: MIT (see LICENSE-MIT.txt inside the zip); annotations: CC-BY-4.0 (see LICENSE-CCBY-annotations.txt)
- ra-eval-v0.1-corpus.zip — human reference corpus: CC-BY-NC-ND-4.0 plus takedown policy (see LICENSE-data.txt inside the zip)
- model_v5.tar.gz — Apache-2.0
KEY RESULTS
Judge-labeled DPO reaches a 0.833 [0.699, 0.972] held-out blind win rate against the base model; repetition loops drop to zero (0.114 to 0.000); the length collapse induced by judge length preference is characterized end-to-end and repaired with an RPO-style anchor (weight w = 0.1).
DATA STATEMENT
The full-text human corpus is distributed for non-commercial research use only, with attribution and a takedown policy: any rights holder may request removal (contact: qiandongwu123@qq.com) and the texts will be removed from all distributions. Derived statistics carry no restriction. The feature library is Chinese-specific and validated on Chinese text; the three-layer evaluation structure and the judge protocol are language-agnostic.
Files
ra-eval-v0.1-code-data.zip
Additional details
Related works
- Is referenced by
- Preprint: 10.48550/arXiv.2611.19840 (DOI)