Datasheets for Reward Models:
A Seven-Layer Pipeline for Auditable Preference Data
Gediminas Sadaunykas · AimRank, Vilnius · gedas@aimrank.io
Working draft v0.4 · 2026-04-28 · industry whitepaper / arXiv preprint candidate · comments welcome.
Judge figures are from the 2026-04-28 run of claude-haiku-4-5 on AWS Bedrock. Production has judged with Qwen3.5-397B-A17B on OVH AI Endpoints since 2026-07-09 and has not been re-measured.
Abstract
Preference data (the pairwise human judgments behind every aligned LLM) is collected today by US-incorporated labelling vendors who do not surface per-annotator reliability, do not publish their qualification protocols, and do not produce the methodology documentation that EU AI Act Annex IV will require of high-risk AI providers from 2 December 2027 (Annex III high-risk) and 2 August 2028 (Annex I product-embedded), per Reg. (EU) 2026/1744.
We present a seven-layer measurement pipeline: qualification gate, active sampling, multi-rater coverage, per-annotator κ, Bayesian aggregation, drift detection, auto-generated data card, together with §7.1, an Annex IV traceability map for preference-data infrastructure. We validate the methodology against three external benchmarks (RewardBench, HelpSteer2, UltraFeedback) and ship a κ-weighted DPO loss with synthetic-lift evidence across 5 configurations.
Reproducibility: all metrics in §6 are recomputable from the JSONL with the methodology code in backend/api/services/iaa_metrics.py, part of the reproduction package available on request. Scripts pinned with seed=42.
Sections
§7.1: EU AI Act Annex IV traceability table
Each item of Regulation (EU) 2024/1689 Annex IV(2)(b)–(h) is quoted verbatim and mapped to a specific pipeline artefact: Layer 1 qualification, Layer 4 per-annotator κ, Layer 7 data card, §5 κ-weighted DPO loss, §8 six-metric calibration battery.
A buyer's compliance officer drops the table into their conformity-assessment file with a citation. They don't regenerate the evidence.
Read / download
arXiv submission: in progress for cs.CL / cs.AI / cs.LG / stat.ML / cs.CY cross-listing. CC-BY-4.0 license. Citable DOI link will appear here once the moderation pass clears.
Reproducibility + companion docs
The companion code and notes are not public yet. The reproduction package is available on request.
iaa_metrics.pyCohen κ, Krippendorff α, Fleiss κ, per-annotator κ implementations with unit testssimulate_kappa_weighted_lift.pysweep-validated synthetic lift for the κ-weighted DPO loss (5/5 configs positive)PAPER_POLISH_NOTES.mdwhat shipped per polish pass; v0.5 backlog- /benchmarksbenchmark numbers, method and cost