Published November 15, 2026 | Version v0.1

RA-Eval v0.1: A Chinese AI-Flavor Evaluation Framework with Calibrated LLM-Judge-Annotated Preference Pairs

Authors/Creators

  • 1. Independent Researcher

Description

RA-Eval v0.1 — the evaluation framework and preference dataset accompanying the paper "Grounding Language Models in Speech Does Not Fix 'AI Flavor': Calibrated LLM Judges, Goodhart Traps, and a Preference-Tuning Fix" (arXiv preprint, 2026).
 
CONTENTS
 
1) ra-eval-v0.1-code-data.zip — benchmark code (feature library, human blind-eval protocol, calibrated LLM-judge pipeline), derived data (feature reference statistics, preference soft labels), result documents, and the paper draft. License: MIT (code) / CC-BY-4.0 (annotations).
 
2) ra-eval-v0.1-corpus.zip — human reference corpus: 86 Chinese college-entrance-examination (gaokao) essays plus judge-calibration sets with answer keys. License: non-commercial research use only, attribution required, takedown policy in LICENSE-data.txt. Copyright remains with the original anonymous student authors.
 
3) model_v5.tar.gz — merged 0.6B model weights (DPO trained on calibrated-judge preference labels, with an SFT anchor) and the LoRA adapter. Base model: Qwen3-0.6B (Apache-2.0); the fine-tuned weights are redistributed under the same license.
 
LICENSE MAPPING (this record lists three licenses)
 
- ra-eval-v0.1-code-data.zip — code: MIT (see LICENSE-MIT.txt inside the zip); annotations: CC-BY-4.0 (see LICENSE-CCBY-annotations.txt)
- ra-eval-v0.1-corpus.zip — human reference corpus: CC-BY-NC-ND-4.0 plus takedown policy (see LICENSE-data.txt inside the zip)
- model_v5.tar.gz — Apache-2.0
 
KEY RESULTS
 
Judge-labeled DPO reaches a 0.833 [0.699, 0.972] held-out blind win rate against the base model; repetition loops drop to zero (0.114 to 0.000); the length collapse induced by judge length preference is characterized end-to-end and repaired with an RPO-style anchor (weight w = 0.1).
 
DATA STATEMENT
 
The full-text human corpus is distributed for non-commercial research use only, with attribution and a takedown policy: any rights holder may request removal (contact: qiandongwu123@qq.com) and the texts will be removed from all distributions. Derived statistics carry no restriction. The feature library is Chinese-specific and validated on Chinese text; the three-layer evaluation structure and the judge protocol are language-agnostic.

Files

ra-eval-v0.1-code-data.zip

Files (2.3 GB)

Name Size
md5:83495d3e54b0db6685b39034e8aadaa9
2.3 GB Download
md5:e37d5b12095cb95562284c363caefe25
393.0 kB Preview Download
md5:24869201b3257d984373223f8c8e67b3
153.7 kB Preview Download

Additional details

Related works

Is referenced by
Preprint: 10.48550/arXiv.2611.19840 (DOI)