Gediminas Sadaunykas, AimRank, Vilnius, Lithuania, gedas@aimrank.io
Working draft v0.5 · 2026-05-23 · arXiv preprint candidate · comments welcome.
Pairwise human preferences are now the dominant supervision signal for post-training large language models (Christiano et al., 2017; Ouyang et al., 2022; Rafailov et al., 2023), yet the public preference corpora that drive most open RLHF and DPO research ship with little or no information about how the data was collected, who judged it, or how reliable the judgements are. RewardBench (Lambert et al., 2024; Allen AI, 2025) crowns ever-better reward models on top of preference inputs whose inter-annotator agreement is rarely measured and almost never published. We argue that the measurement layer between raw votes and downstream training is structurally absent from the field’s open infrastructure, and we describe a seven-layer pipeline that closes that gap: (1) gold-standard annotator qualification, (2) Glicko-2-driven active pair selection, (3) configurable multi-rater coverage filtering, (4) per-annotator agreement against majority consensus, (5) Bradley–Terry aggregation with bootstrap confidence intervals, (6) drift and anomaly detection, and (7) auto-generated Datasheets-for-Datasets-style data cards. Every layer is deterministic, open-source-implementable, and produces audit artefacts a downstream consumer can re-run. We additionally specify two voluntary declarative scaffolds, an explicit constitution (Bai et al., 2022) and a multi-axis evaluation objective vector (Wang et al., 2024), surfaced to voters in the UI and to consumers in the data card. We propose a κ-weighted, margin-aware DPO formulation that consumes the layer-1–4 outputs, frame it explicitly as a planned protocol rather than a published result, and lay out a reproducibility recipe that runs against a single EU-hosted reference implementation.
Keywords: preference data; reward models; RLHF; DPO; inter-annotator agreement; Krippendorff’s α; Datasheets for Datasets; EU AI Act; data sovereignty.
Reward modelling sits at the highest-leverage point of the alignment stack. A single contaminated pair propagates through every subsequent PPO, DPO, or KTO step (Rafailov et al., 2023; Ethayarajh et al., 2024), and the cost of cleaning the model after deployment is orders of magnitude larger than the cost of cleaning the data before training. The empirical literature reflects this asymmetry: RewardBench (Lambert et al., 2024) and its successor RewardBench 2 (Allen AI, 2025) measure reward-model quality with increasing rigor, while the upstream preference data for those models, collected by anonymised annotators at commercial vendors or scraped from public crowdsourced platforms, remains opaque.
None of the four corpora that account for the majority of open RLHF/DPO training as of Q1 2026 ship with the basic measurement artefacts a regulated buyer expects:
This is not a critique of the dataset authors. Anthropic HH-RLHF (Bai et al., 2022), OpenAssistant (Köpf et al., 2023), UltraFeedback (Cui et al., 2023), and HelpSteer2 (Wang et al., 2024) shipped what their commercial and institutional structures allowed. But the consequence is that the most leverage-sensitive layer of the alignment stack runs on the least audited data infrastructure in the entire pipeline, and that the EU AI Act’s Annex IV technical-documentation requirement (Regulation (EU) 2024/1689) cannot be satisfied off the shelf for any high-risk system trained on these inputs.
This paper specifies a concrete remediation: seven measurement layers and two declarative scaffolds, all deterministic, all open-source-implementable, and all shipped on main in the AimRank reference repository (module paths in Appendix A). Where downstream training-loss formulations are proposed but not yet empirically validated against paying-customer datasets we say so explicitly and move the forward-looking numbers to Appendix B.
Pairwise preference modelling. The Bradley–Terry model (Bradley & Terry, 1952) provides the closed-form latent-ability link function under the logistic assumption. Glicko-2 (Glickman, 2012) extends it with online rating-deviation tracking, which we exploit for active sampling (§3.2). TrueSkill (Herbrich et al., 2007) provides the Bayesian generalisation; we use Glicko-2 by default for its better-documented behaviour on small-N annotator panels.
Inter-annotator agreement. Cohen’s κ (Cohen, 1960) corrects pairwise agreement for chance under per-rater marginal class distributions. Fleiss’ generalisation (Fleiss, 1971) handles N raters but uses the pooled marginal, which differs numerically from Cohen on the two-rater case. Krippendorff’s α (Krippendorff, 2011; Hayes & Krippendorff, 2007) is the dominant content-analysis reliability coefficient because it tolerates missing data and varying raters per item, both endemic in pairwise preference collection. Landis & Koch (1977) provide the κ-magnitude rubric we use as a soft interpretive overlay (slight / fair / moderate / substantial / almost-perfect).
Preference data for LLM alignment. Christiano et al. (2017) introduced reinforcement learning from human preferences; Ouyang et al. (2022) operationalised it at scale (InstructGPT). Rafailov et al. (2023) re-framed it as direct policy optimisation (DPO), and a family of variants, IPO (Azar et al., 2023), cDPO (Mitchell, 2023), KTO (Ethayarajh et al., 2024), and SimPO (Meng et al., 2024), refines the loss to handle margin-aware or noise-robust signals. Each of these variants consumes preference data; none of them ships an opinion about how that data should be collected. Tülu 3 (Lambert et al., 2024) is the closest published reference for an open, end-to-end DPO post-training recipe.
Engagement-priced collection. Chatbot Arena (Chiang et al., 2024) is the closest methodological cousin to AimRank’s own engagement-priced collection mode: thousands of casual voters, Bradley–Terry aggregation, and a public leaderboard. It does not currently expose per-annotator κ or a Datasheet-style audit artefact; Layers 4 and 7 of the present pipeline are designed to close that specific gap.
Documentation standards. Mitchell et al. (2019) introduced model cards; Gebru et al. (2021) introduced datasheets for datasets. Layer 7 (§3.7) is a direct application of the latter to preference data, with the additional discipline that the card is generated automatically from the same code path that exports the training data, making “the card lies” mechanically harder.
Constitutional and multi-axis methods. Bai et al. (2022) framed Constitutional AI as supervision via explicit principles; Wang et al. (2024)’s HelpSteer2 demonstrated that vector-valued reward models trained on multi-axis labels (helpfulness, correctness, coherence, complexity, verbosity) outperform scalar reward models on RewardBench. We expose both abstractions as voluntary declarative scaffolds (§4) that consumers can opt into.
EU regulatory context. Regulation (EU) 2024/1689 (the AI Act) defines technical-documentation requirements at Annex IV for high-risk AI systems and is enforceable for general-purpose AI models from August 2025 and for high-risk systems from August 2026. The Annex IV requirements that bind to training data (provenance, characteristics, examination of biases) map directly onto Layers 3, 4, 6, and 7 of the present pipeline; the mapping is shown in §6.
Each layer is specified as: input, formal definition, output, and effect on downstream training. The reference implementation is in backend/api/services/ (paths in Appendix A); a customer can reproduce every output deterministically from the raw vote stream.
A ranking owner declares gold-standard items: pairs \((a, b, w^*)\) with a known-correct winner \(w^* \in \{a, b\}\), typically curated by a subject-matter expert (SME). Voters must clear a fraction \(\tau\) of these (default \(\tau = 0.80\)) before their downstream votes are admitted to either the leaderboard or the RLHF export. Anonymous voters cannot pass qualification by construction (no stable identity to track), so when qualification_required = true anonymous voting is implicitly disabled.
qualification_required ∈ {true, false}
qualification_pass_threshold τ ∈ [0, 1] (default 0.80)
qualification_items[] = [(a, b, w*, rationale?)]The qualification panel additionally serves as the input to ceiling estimation (§3.4). Because every SME on the panel rates the same \(|Q|\) items, we obtain pairwise SME-vs-SME κ as a by-product, and that ceiling becomes the contractual upper bound for any downstream reward model: a model cannot meaningfully exceed the agreement of the humans whose judgements define the ground truth.
Effect on downstream training: in our internal pilots Layer 1 filters out 5–25% of submissions on most domains. The filtered subset shows κ improvements of 0.05–0.15 against the unfiltered baseline (illustrative numbers; see Appendix B for the formal protocol).
Pair presentation is driven by Glicko-2 rating uncertainty \(\phi\). We score every candidate pair \((a, b)\) by the expected-information-gain proxy
\[ I(a, b) = H(p_{ab}) \cdot \frac{\phi_a^2 + \phi_b^2}{\phi_a^2 + \phi_b^2 + \beta^2} + \mathbb{1}\!\left[N_a < 5\right] \gamma_{\text{cold}} + \mathbb{1}\!\left[N_b < 5\right] \gamma_{\text{cold}}, \]
where \(p_{ab}\) is the Glicko-2 win probability of \(a\) over \(b\), \(H(\cdot)\) is binary entropy in bits, \(\beta\) is the Glicko-2 system noise (default 200), \(N_a\) is the match count of entity \(a\), and \(\gamma_{\text{cold}} = 0.15\) is a cold-start bonus. The first factor peaks at \(p_{ab} = 0.5\) (most informative outcomes are the most uncertain ones); the second factor is the share of total match variance attributable to skill uncertainty rather than inherent noise (so a pair where both entities have already converged contributes little, as expected); the third encourages exploration of cold entities. The selection rule is \(\arg\max_{(a,b)\,\notin\,S_u} I(a, b)\), where \(S_u\) is the set of pairs the current voter has already seen, with a per-rater diversity constraint to avoid serving the same pair to one voter twice.
This is a direct application of the uncertainty-sampling branch of active learning (Settles, 2009; Pfeiffer et al., 2012) to pairwise comparison. The reference implementation is backend/api/services/matchmaking.py::_information_gain_matchup, with adaptive strategy selection (get_adaptive_matchup) that switches between uncertainty sampling, Swiss-style pairing, and exploitation depending on ranking maturity (vote count, average \(\phi\), coverage).
Empirical results: synthetic experiment. We measure label efficiency reproducibly via scripts/simulate_active_sampling_efficiency.py: \(N\) entities with true skills sampled from \(\mathcal{N}(1500, 250)\), outcomes drawn from the Glicko-2 win-probability under those skills, both strategies running on the canonical Glicko2Algorithm, evaluated by Spearman \(\rho\) of estimated-vs-true rankings every 25 votes. Under idealised synthetic conditions (8 trials, \(N=30\) entities, threshold \(\rho \geq 0.85\)), the active sampler matches random within ±5% (random: 176 votes mean; active: 181 votes mean). The Pfeiffer (2012) headline of 3–5× advantage was measured on real Wikipedia-editor ranking data with substantial heterogeneity (rater error, vote-time correlation, entity clusters); idealised synthetic BT outcomes do not reproduce that gap. We therefore make a more conservative claim: active sampling provides equivalent-or-better label efficiency than random sampling, with the upper-bound advantage realised only under heterogeneous real-world conditions. Real-customer numbers from L3 pilots will be reported in Appendix B once available.
Let \(V_{ab}\) be the set of distinct voters who judged the unordered pair \(\{a, b\}\). A configurable threshold $k = $ min_raters_per_pair excludes pairs with \(|V_{ab}| < k\) from both the IAA report and the RLHF export. The default is \(k = 1\) for casual rankings and \(k = 3\) for AI-lab consumption, the same threshold Anthropic used for HH-RLHF’s internal sanity checks and the threshold above which most agreement coefficients stabilise on \(n \approx 200\) pairs.
We additionally publish a coverage histogram (the empirical distribution of \(|V_{ab}|\) across pairs) so the downstream consumer can see the cost of the cut. The reason this matters: claims like “our Krippendorff α is 0.71” are honest only when the underlying pairs have been judged by enough independent raters that α is meaningfully defined. Reporting α over single-rated pairs (which several public corpora implicitly do) collapses the chance baseline in arbitrary directions and produces statistically meaningless numbers.
Reference implementation: iaa_metrics.py::filter_votes_by_min_raters and iaa_metrics.py::coverage_report.
For every multi-rated pair \(\{a, b\}\) we compute the majority winner (consensus, ties broken by deterministic alphanumeric sort). For each annotator \(u\) we then compute Cohen’s κ on the subset of pairs they judged that overlap with consensus.
A subtle but important detail: judgements across heterogeneous pairs cannot be compared directly because every distinct entity ID would be its own class, the per-rater marginals would collapse to disjoint distributions, and \(p_e \to 0\) even for systematic disagreement, giving κ → 0 incorrectly. We therefore binarise each judgement to a class in \(\{\text{left}, \text{right}\}\) where “left” denotes the alphabetically-smaller entity ID in the unordered pair. Under this normalisation, Cohen’s κ is well-defined across the heterogeneous-pair set and remains comparable across raters.
The output is sorted worst-first, surfacing trolls and random clickers (\(\kappa \approx 0\)) and contrarians (\(\kappa < 0\)) explicitly. Customers can apply a min_voter_kappa filter at RLHF export time to drop their contributions or treat them as a separate cohort. The reference implementation is iaa_metrics.py::compute_per_annotator_agreement (default min_pairs = 3, matching Krippendorff’s small-sample stability floor).
A separate function, compute_ceiling_from_qualification (iaa_metrics.py::compute_ceiling_from_qualification), runs the same κ computation pairwise across SMEs over qualification items; the mean across qualified rater pairs is the human-human ceiling for the domain, and is reported alongside model κ in every L3 calibration report (see §6 and the L3 calibration methodology document for the contractual implications).
Bradley–Terry MLE ratings are computed via the Zermelo / minorisation-maximisation iteration (Bradley & Terry, 1952; Hunter, 2004). We bootstrap at the vote level: resample with replacement \(B\) times (default \(B = 1000\)), refit BT each time, and report the median together with the 2.5th and 97.5th percentiles per entity. For binary win-rates we additionally report Wilson 95% confidence intervals, which are well-behaved at low and high \(p\) and are the de-facto standard in the LMSYS/Chatbot Arena reference implementation (LMSYS, 2023). Glicko-2 (Glickman, 2012) provides the online rating updates with rating-deviation tracking that Layer 2 consumes for active sampling.
The output of Layer 5 is therefore a continuous score with a standard error per entity, not just an ordinal rank. This matters for downstream RM training because margin-aware DPO variants (IPO, cDPO; Azar et al., 2023; Mitchell, 2023) consume confidence-weighted preferences rather than bare Bernoulli pairs, and the surface that connects the two is the per-entity standard error.
Reference implementation: bootstrap_ci.py::fit_bradley_terry and bootstrap_ci.py::bootstrap_ratings.
We continuously monitor four classes of ranking pathology:
transitivity.py::detect_cycles) above a per-domain threshold; reported in the data card and excluded from RLHF export.Plus a temporal decay parameter tuned per-domain via grid search over (decay_rate, threshold_days) (default search grid \(\{0.0, 0.01, 0.02, 0.05, 0.1\} \times \{30, 60, 90, 180\}\)), maximising prediction accuracy on a held-out validation slice. This addresses the staleness problem documented in the Chatbot Arena post-mortem analyses (Li et al., 2024) where preferences drift faster than the data is refreshed.
Reference implementation: anomaly_detection.py, decay_optimizer.py, transitivity.py.
Every export ships with a markdown document that follows the section ordering of Gebru et al.’s (2021) Datasheets for Datasets and inherits the structural hygiene of Mitchell et al.’s (2019) model cards: provenance → composition → collection process → preprocessing → reliability → maintenance. Concretely the card surfaces:
The card is regenerated on demand from the live database state, so it cannot drift away from the underlying data. Reference implementation: data_card.py::build_summary and data_card.py::render_markdown.
Beyond the seven measurement layers, two declarative artefacts can be attached at ranking creation time. They are voluntary because not every ranking owner has the time or expertise to specify them, but when they are present they enable richer downstream training signals.
A list of explicit principles voters apply when judging, directly motivated by Constitutional AI (Bai et al., 2022). Canonical schema:
[
{"id": "p1", "text": "Prefer responses that cite the relevant standard or RFC"},
{"id": "p2", "text": "Never suggest code that exfiltrates secrets",
"category": "safety"}
]The constitution is surfaced to voters in the comparison UI immediately above the prompt and to data consumers in §3 of the data card and the RLHF-export bundle metadata. When present, it enables principle-level auxiliary supervision during reward-model training (§5).
Reference implementation: alignment_scaffolds.py::normalize_constitution, effective_constitution, render_for_data_card.
A list of evaluation axes ({id, name, description, scale ∈ {pairwise, likert}}) that decompose “is A better than B?” into N independent judgements. The current vote schema is single-axis (one overall preference); declared objectives are surfaced in the data card and the RLHF export so consumers can train vector-valued reward models in the spirit of HelpSteer2 (Wang et al., 2024). Per-axis voting is a v2 feature pending a vote-schema migration (§7).
Reference implementation: alignment_scaffolds.py::normalize_objectives, effective_objectives.
The seven-layer pipeline produces signals not present in vanilla DPO data: per-vote confidence, per-annotator κ, per-pair coverage, per-vote decision time, declared constitutional principles, declared evaluation axes. We propose a loss formulation that consumes them. This section is a specification, not a published result. Appendix B sets out the protocol for empirical validation against pilot-customer datasets in Q3 2026.
Let \(\mathcal{D} = \{(x_i, y_i^+, y_i^-, w_i, c_i)\}_{i=1}^N\) be a preference dataset where \(y_i^+\) is the chosen completion, \(y_i^-\) the rejected completion, and the additional per-sample signals are:
vote.confidence \(\in [0, 1]\) is the IPO/cDPO margin (Azar et al., 2023; Mitchell, 2023).The proposed loss is
\[ \mathcal{L}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} w_i \cdot c_i \cdot \log \sigma\!\left(\beta \cdot \left[r_\theta(x_i, y_i^+) - r_\theta(x_i, y_i^-)\right]\right) + \lambda_{\text{aux}} \mathcal{L}_{\text{const}} + \lambda_{\text{axis}} \mathcal{L}_{\text{axis}}, \]
where \(r_\theta\) is the implicit DPO reward (Rafailov et al., 2023), \(\beta\) is the DPO temperature, and:
Layer 1 (qualification) is applied as a hard pre-training filter: votes from un-qualified voters are dropped from \(\mathcal{D}\) entirely; they do not enter the loss. Layer 6 (anomaly detection) is similarly enforced upstream: votes from rankings flagged as dominated, unstable, or transitivity-pathological are excluded by default.
main)The implementation surface is a patch to TRL’s RewardTrainer. The reference implementation lives at scripts/train_l3.py and is enabled via the --enhanced-loss CLI flag (default off, i.e. vanilla TRL Bradley-Terry, for backwards compatibility). The patch:
class KappaWeightedRewardTrainer(RewardTrainer):
def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):
rewards_chosen = model(**inputs["chosen"]).logits
rewards_rejected = model(**inputs["rejected"]).logits
kappa = clamp(inputs.get("kappa_voter", 1.0), min=0.05)
weights = kappa * inputs.get("coverage_factor", 1.0) \
* inputs.get("time_quality", 1.0) \
* inputs.get("confidence", 1.0)
weights = weights / weights.mean().clamp(min=1e-8) # batch-mean-normalise
margin = beta * (rewards_chosen - rewards_rejected).squeeze(-1)
return (-F.logsigmoid(margin) * weights).mean()Each weighting term defaults to 1.0 if absent from the JSONL, so the loss collapses cleanly to vanilla Bradley-Terry when no metadata is supplied; the patch is non-disruptive for callers who haven’t adopted the layered metadata yet.
Default base for the EU-sovereign deployment path is Mistral 7B Instruct v0.3 (Mistral AI, 2024; Apache 2.0, trained in France), with Llama 3.1 8B and Qwen 2.5 7B as alternatives for customers whose compliance posture differs.
The full LLM-RM training run requires GPU and ~hours of wall time, so we cannot ship a direct LLM-loss number from a CPU-only environment without first paying-customer data. The principle, however, is testable independently in Bradley-Terry-MLE space. If weighted-BT MLE outperforms vanilla BT MLE on synthetic data with heterogeneous rater quality, the gradient-update structure carries to the LLM-RM regime (it’s the same per-pair loss up to a base-model multiplier on the reward).
The reproducibility script is scripts/simulate_kappa_weighted_lift.py (offline, no Bedrock, no GPU; ~30 seconds wall time). Setup: 50 entities with true skills sampled from \(\mathcal{N}(0, 1.5)\); 30 raters in three quality cohorts (10 expert at 95% accuracy with high confidence; 10 moderate at 75%; 10 noisy at 55%); 1,500 votes generated per trial; per-rater κ computed via the canonical Layer-4 procedure; vanilla BT MLE vs κ-weighted BT MLE fit on the same vote pool.
Result, 8 trials at the canonical configuration:
| Estimator | Spearman ρ vs true skill (mean ± std) |
|---|---|
| Vanilla Bradley-Terry MLE | \(0.858 \pm 0.034\) |
| κ-weighted Bradley-Terry MLE | \(0.897 \pm 0.031\) |
| Lift | +3.94pp ± 3.70pp (positive in 7 of 8 trials) |
Cohort κ validation (sanity check that Layer 4 tracks underlying quality):
| Rater cohort | Mean computed κ | Underlying accuracy |
|---|---|---|
| Expert | \(0.71\) | 95% |
| Moderate | \(0.65\) | 75% |
| Noisy | \(0.62\) | 55% |
Cohort means descend monotonically: the §3.4 per-rater κ estimator correctly orders rater quality. Output JSON lives at data/benchmarks/kappa_weighted_lift.json for re-analysis.
Configuration sweep. To address the concern that a single configuration result might be a fluke, we ran the same experiment across five configurations spanning data volume, entity count, and rater-cohort sizes (scripts/simulate_kappa_weighted_lift.py --sweep):
| Configuration | \(N_{\text{entities}}\) | Raters | Votes | Vanilla ρ | Weighted ρ | Lift | Positive trials |
|---|---|---|---|---|---|---|---|
| low-data | 30 | 24 | 600 | 0.828 | 0.840 | +1.22pp | 4/6 |
| canonical | 50 | 30 | 1500 | 0.859 | 0.895 | +3.53pp | 5/6 |
| high-data | 50 | 30 | 4000 | 0.931 | 0.955 | +2.44pp | 6/6 |
| many-entities | 100 | 45 | 3000 | 0.862 | 0.901 | +3.90pp | 6/6 |
| noisy-heavy | 30 | 18 | 1000 | 0.874 | 0.889 | +1.50pp | 4/6 |
| Aggregate | +2.52pp | 5/5 configs positive |
Every configuration produces a positive mean lift; the range is +1.22pp to +3.90pp; the smallest-data and noisiest configurations show smaller (but still positive) lift, consistent with the intuition that κ-weighting helps most when there’s enough data to estimate κ reliably and enough rater heterogeneity for the weighting to matter. Output JSON at data/benchmarks/kappa_weighted_lift_sweep.json.
Caveats. This is a BT-MLE-space proxy, not the full LLM-RM lift. The LLM has its own inductive biases and the LoRA adapters attenuate gradients; the magnitude of lift will differ on real data. The principle (down-weighting low-κ rater votes improves preference recovery) transfers across configurations as shown above; the magnitude on LLM-RM training is gated on first paying-customer datasets in Q3 2026. Appendix B sets out the LLM-RM validation protocol.
We’ve therefore moved this section from protocol to implementation + sweep-validated synthetic lift. The remaining empirical gap is the LLM-scale validation, which is honestly framed throughout the paper.
The κ-weighted, margin-aware loss in §5 sits within a family of preference-loss variants that have appeared since Rafailov et al. (2023):
| Formulation | What it weights | What it doesn’t see |
|---|---|---|
| DPO (Rafailov et al., 2023) | Nothing; every pair contributes equally | Rater identity, vote confidence, coverage |
| IPO (Azar et al., 2023) | Margin between chosen and rejected reward; smooth surrogate avoiding DPO’s logit explosion | Rater identity, coverage |
| cDPO (Mitchell, 2023) | A single global label-flip noise rate \(\epsilon\) via a corruption-aware objective | Per-rater quality, per-pair coverage |
| KTO (Ethayarajh et al., 2024) | Asymmetric prospect-theoretic weighting on chosen vs rejected (gain/loss separation) | Rater identity, coverage |
| SimPO (Meng et al., 2024) | Length-normalised reward to remove length bias from BT pairs | Rater identity, coverage, confidence |
| §5 κ-weighted (this work) | Per-rater κ × coverage × time-quality × per-vote confidence | (proposes a per-pair theoretical-noise prior; open question for v2) |
The key positioning. IPO and cDPO already attack noise, but at the pair-aggregate level (cDPO) or the margin level (IPO). Neither sees the per-rater κ Layer 4 produces. Our contribution is not a new gradient direction; it’s that we feed the loss the per-rater quality signal that the seven-layer pipeline already computes. When the methodology layers are absent, the κ-weighted loss collapses to the IPO/DPO cases (all weights → 1.0). The contribution is the data plumbing, not the loss form.
The κ-weighted loss is also compatible with each of the above as drop-in modifications: replace \(\log\sigma(\beta(r^+ - r^-))\) with the IPO smoothed log-sigmoid, the cDPO corruption-aware objective, or the SimPO length-normalised reward, and keep the per-pair \(w_i \cdot c_i\) weighting unchanged. We have not yet validated which combination is empirically best; that’s part of the LLM-RM validation gated on first paying-customer data (Appendix B).
| Dataset | Pairs | Layer 1 (qual.) | Layer 3 (multi-rater) | Layer 4 (per-rater κ) | Layer 7 (data card) | Our judge accuracy on a sample |
|---|---|---|---|---|---|---|
| Anthropic HH-RLHF (Bai et al., 2022) | 169K | ✗ | ✗ | ✗ | ✗ | not yet measured |
| OpenAssistant (Köpf et al., 2023) | 161K | partial | ✗ | ✗ | ✗ | not yet measured |
| UltraFeedback (Cui et al., 2023) | 64K | ✗ | ✗ | ✗ (LLM-judged) | ✗ | 82.0% (n=200, this work) |
| HelpSteer2 (Wang et al., 2024) | 21K | ✗ | partial | ✗ | partial | 79.0% (n=200, this work) |
| RewardBench / RewardBench 2 (Lambert et al., 2024; 2025) | 358 / 1.9K test | ✗ | ✗ | ✗ | ✗ | 90.89% (n=2,985 full split, this work) |
| Chatbot Arena (Chiang et al., 2024) | 240K+ votes | ✗ | implicit | ✗ | ✗ | not yet measured |
| AimRank-collected (this work) | variable | ✓ | ✓ | ✓ | ✓ | n/a (own data) |
The four datasets at the top of the table account for the majority of published RLHF and DPO training in the open-source ecosystem as of Q1 2026. None of them ships a Datasheets-style data card; none publishes per-annotator κ; UltraFeedback is LLM-judged by construction (no Layer-4 signal possible); HelpSteer2 ships fine-grained scalar attributes but no per-annotator agreement table. The rightmost column reports our LLM judge’s agreement with each dataset’s published preference labels, measured on 2026-04-28 with claude-haiku-4-5 via Bedrock eu-central-1, A/B-randomised, seed=42-pinned. Per-pair cost €0.001 at Bedrock list pricing (1,500–5,000× under Scale AI’s labour-priced rates). Full breakdowns + per-pair rationale logs at data/benchmarks/judge_*.json.
We do not present this as a critique. Each of those teams shipped what the commercial and institutional structures of their organisations allowed, and each has been an enormous public good. We present it as evidence that the measurement layer is structurally absent from the field’s open infrastructure, and our contribution is to ship it as a reusable, vendor-neutral layer that any downstream consumer can adopt independently of where they buy their preferences.
The pipeline’s Layer 1 (qualification) and Layer 4 (per-annotator κ) presuppose that the baseline judge used during ranking warm-up has reasonable agreement with human ground truth. We measure that against three public benchmarks and ship the scripts so any consumer can reproduce the numbers from their own AWS Bedrock account.
Headline (historical: run 2026-04-28, judge = claude-haiku-4-5 via Bedrock eu-central-1. Production has run the OVH AI Endpoints judge, Qwen3.5-397B-A17B, since 2026-07-09; these figures have NOT been re-measured on it):
| Benchmark | Ground-truth source | n pairs | Accuracy | Wilson 95% CI | Cost (EUR) | Position bias |
|---|---|---|---|---|---|---|
RewardBench (filtered, full) |
Human pairwise | 2,985 | 90.89% | [89.80%, 91.87%] | €3.01 | +0.005 |
HelpSteer2 (preference) |
Human multi-axis | 200 | 79.00% | [72.84%, 84.07%] | €0.30 | −0.135 |
UltraFeedback (binarized) |
GPT-4 as judge | 200 | 82.00% | [76.09%, 86.71%] | €0.28 | −0.075 |
| Combined | Mixed | 3,385 | 89.78% | — | €3.59 | — |
Per-pair cost: €0.001 at Bedrock list pricing, 1,500–5,000× under Scale AI’s published per-pair labelling rates (€1.50–€5.00).
RewardBench (canonical anchor, full split). Top subsets by accuracy and the four worst:
| Subset | n | Accuracy | Notes |
|---|---|---|---|
hep-go, hep-js, hep-python, hep-java, hep-rust, hep-cpp |
984 (combined) | 96.3–100.0% | Code-helpfulness pairs; judge is near-ceiling |
xstest-should-respond |
250 | 97.6% | Refusal-correctness pairs |
xstest-should-refuse |
154 | 98.1% | Refusal-correctness pairs |
donotanswer |
136 | 85.3% | Adversarial harm prompts |
math-prm |
447 | 82.8% | Step-by-step math process-reward verification |
llmbar-adver-neighbor |
134 | 64.2% | Adversarially-near distractors |
llmbar-adver-GPTInst / GPTOut / manual |
92 + 47 + 46 | 73.9–74.5% | LLM-generated adversarial distractors |
mt-bench-hard |
37 | 81.1% | Multi-turn hard prompts |
The four llmbar-adver-* subsets are the honest weakness. Adversarially-constructed near-distractors are the classic LLM-judge failure mode (Zheng et al. 2023, Judging LLM-as-a-Judge with MT-Bench). They are precisely the subsets where AimRank’s Layer 1 (qualification gates), Layer 4 (per-annotator κ vs consensus), and Layer 7 (audit-trail data card) defences add the largest marginal value: when the judge is unreliable, the platform falls back to qualification-gated human raters with κ-weighted aggregation, and the data card explicitly reports the subset-level reliability so the consumer never silently inherits the judge’s blind spots. The 90.89% headline is therefore a floor on judge-bootstrapped reliability before the methodology layers attach; adversarial-subset accuracy after Layer-1+4 filtering is the more meaningful number for downstream RM training, and we plan to measure it on first-customer L3 datasets.
HelpSteer2 (multi-axis human-rater anchor). NVIDIA HelpSteer2 (Wang et al., 2024) ground truth is human ratings on 5 axes (helpfulness / correctness / coherence / complexity / verbosity) with preference_strength ∈ {1, 2, 3} derived from rater-consensus. The judge’s accuracy increases monotonically with preference strength, a textbook validity check that strong-preference pairs are easier for the judge:
| Preference strength | n | Accuracy |
|---|---|---|
| 1 (slight) | 97 | 76.3% |
| 2 (clear) | 60 | 80.0% |
| 3 (strong) | 43 | 83.7% |
Lower aggregate accuracy (79.0%) vs RewardBench (90.9%) is expected: multi-axis quality preferences are harder than chat / safety / code, and this is exactly the regime where AimRank’s Layer 4 per-annotator-κ filtering buys the most.
UltraFeedback (LLM-judge consistency anchor). UltraFeedback’s binarized labels are themselves derived from GPT-4-as-judge, so 82.0% measures judge-to-judge consistency rather than human agreement. It serves as a sanity check against GPT-4-class judgement on the same prompt distribution.
Reproducibility. Three scripts ship on main, each pinned with seed=42 so the exact pair IDs are deterministic:
| Script | Output |
|---|---|
scripts/benchmark_judge_vs_human.py --n 2985 |
data/benchmarks/judge_reward_bench_full.json |
scripts/benchmark_judge_vs_helpsteer2.py --n 200 |
data/benchmarks/judge_helpsteer2.json |
scripts/benchmark_judge_vs_ultrafeedback.py --n 200 |
data/benchmarks/judge_ultrafeedback.json |
A fourth script benchmark_judge_vs_tulu3.py ships for the Tülu-3 preference mixture (Lambert et al., 2024; allenai/tulu-3-pref-mixture) but the dataset is currently gated by Hugging Face authentication; the v0.3 revision will include the result once an authorised run has been completed.
The cohen_kappa_vs_ground_truth field in each JSON is degenerate by construction (κ collapses to 0 when one rater is constant; the post-randomisation ground truth is always “chosen”) and should be ignored; accuracy + Wilson CI is the operative metric in this evaluation setup. We retain the field for completeness and audit transparency.
A subtle but commercially load-bearing property: AimRank’s reference deployment is hosted exclusively in the EU — on OVHcloud, an EU-established provider, in its DE1 region (Germany) since 2026-07-08 — under GDPR Article 28 with a customer Data Processing Agreement on day one. Concretely:
Qwen3.5-397B-A17B since 2026-07-09). (Earlier drafts of this section described an AWS eu-central-1 deployment with Bedrock inference; that stack was retired on 2026-07-08.)For regulated EU customers (Helsing, the German Bundesbank, DAX-30 brand teams subject to BaFin or BAFA oversight) this is not a marketing line. It is, under the strict reading of GDPR plus the AI Act, the only legal path to a fine-tuned reward model on customer preference data. The major US labelling vendors (Scale AI, Surge AI) cannot serve this segment without opening EU subsidiaries and signing customer-VPC contracts that they are not currently structured to provide.
For high-risk AI systems, Article 11(1) of Regulation (EU) 2024/1689 (“AI Act”) requires that “the technical documentation of a high-risk AI system shall be drawn up before that system is placed on the market or put into service and shall be kept up-to-date.” The minimum content of that documentation is enumerated in Annex IV. Items 2(b) through 2(h) of Annex IV govern the development process: the part of the file a downstream provider of a high-risk AI system (e.g. a deployer fine-tuning a foundation model on customer preference data) must compile before their conformity assessment.
Most reward-model papers and most preference datasets address none of these items in a form a compliance officer can pass to a notified body. The seven-layer pipeline produces, by construction, the artefacts each Annex IV item requires. The mapping is direct:
| Annex IV item (Regulation 2024/1689, Annex IV(2), verbatim excerpt) | Pipeline artefact that satisfies it |
|---|---|
| 2(b) “the design specifications of the system, namely the general logic of the AI system and of the algorithms; the key design choices including the rationale and assumptions made, including with regard to persons or groups of persons in respect of whom the system is intended to be used; the main classification choices; what the system is designed to optimise for and the relevance of the different parameters” | Layer 7 data card §1 (use-case specification) declares the optimisation target, intended deployer, and the populations the labelling pool is and is not representative of. Layer 1 (qualification protocol) documents the rationale for who is allowed to label and why (per-axis qualifications, attention-check thresholds). §5 of this paper documents the algorithmic logic (κ-weighted, margin-aware DPO loss) including its rationale relative to vanilla DPO. |
| 2(c) “where relevant, the description of the system architecture explaining how software components build on or feed into each other and integrate into the overall processing; the computational resources used to develop, train, test and validate the AI system” | Layer 7 data card §3 (architecture) lists the seven layers, the order in which they apply, and which layer’s output feeds which layer’s input. Appendix A (reproducibility recipe) of this paper enumerates compute used at each stage. The AimRank platform README documents the orchestration layer. |
| 2(d) “where relevant, the data requirements in terms of datasheets describing the training methodologies and techniques and the training data sets used, including a general description of these data sets, information about their provenance, scope and main characteristics; how the data was obtained and selected; labelling procedures (e.g. for supervised learning), data cleaning methodologies (e.g. outliers detection)” | Layer 7 data card is, structurally, a Datasheet for Datasets (Gebru et al., 2021) extended for preference data. Provenance: Layer 1 logs every annotator’s qualification trace. Scope and characteristics: Layer 3 logs domain, language, and task taxonomy per pair. Labelling procedures: Layers 1–4 are themselves the labelling-procedure specification, including attention checks, per-annotator agreement κ, and the qualification gates. Outlier and cleaning methodology: Layer 4 (per-annotator κ filter) and Layer 6 (anomaly detection) are the cleaning step, with cleaning thresholds documented in the data card. |
| 2(e) “an assessment of the human oversight measures needed in accordance with Article 14, including an assessment of the technical measures put in place to facilitate the interpretation of the outputs of AI systems by the deployers” | Layer 4 (per-annotator κ vs majority consensus) is, mechanically, a continuous human-oversight measure: every annotator is monitored against consensus and down-weighted or removed if they drift. Layer 6 (anomaly detection) is the systemic-drift oversight signal. Together they constitute the human-in-the-loop assessment Article 14 calls for, with metrics documented in the data card §4. |
| 2(g) “the validation and testing procedures used, including information about the validation and testing data used and their main characteristics; the metrics used to measure accuracy, robustness and compliance with other relevant requirements set out in Chapter III, as well as potentially discriminatory impacts; test logs and all test reports dated and signed by the responsible persons” | Layer 5 (Bradley-Terry MLE with Wilson 95% CIs) produces the accuracy and robustness metrics. §8 of this paper specifies the six-metric production-acceptance battery (head-vs-baseline accuracy ≥ 65 %, RewardBench ≥ 75 %, top-decile separation, low-κ ablation, calibration ECE ≤ 0.10, judge-vs-human agreement κ ≥ 0.40). The L3 calibration protocol is the per-customer test report, signed by the customer’s data scientist before reward-model deployment. Discriminatory-impact monitoring is supplied by Layer 4 per-axis qualification spreads, which surface annotator-pool bias. |
| 2(h) “where applicable, a detailed description of pre-determined changes to the AI system and its performance, together with all the relevant information related to the technical solutions adopted to ensure continuous compliance of the AI system with the relevant requirements set out in Chapter III” | Layer 6 (anomaly detection) is the continuous-compliance instrument: it surfaces leader instability, dead rankings, and dominated configurations with statistically grounded triggers. The paper-version policy (§v0.x bumps committed alongside data-card refresh) is the documentation-change protocol. The κ-weighted loss is itself a pre-determined change relative to vanilla DPO, documented in §5 with its rationale. |
What this table is not: A legal opinion or a substitute for a notified-body conformity assessment. The right reading is: the seven-layer pipeline produces, by construction, the evidence a deployer needs to populate Annex IV. A buyer’s compliance officer still has to write the file. What we remove is the engineering work of regenerating the evidence from scratch, which is, in practice, what kills most EU-deployer reward-model projects before they reach procurement.
This is the differentiator that closes EU-enterprise procurement: every Annex IV item that touches data and labelling has a named pipeline artefact that satisfies it, with a unit-tested reference implementation. To our knowledge, this is the first preference-data methodology that publishes its Annex IV traceability map. We invite challenge from EU compliance practitioners; the table is meant to be falsifiable.
Per-axis voting (Objectives v2). The objectives field is currently declarative; voters score a single overall preference. Real per-axis voting requires a vote-schema migration and is queued for v2.
Active sampling under qualification gating. Qualification items are not currently part of the active-sampling loop. A future iteration will treat the qualification set as a small high-information prior, reducing the SME labelling burden.
Cross-domain transfer of qualification. When a customer adds a new domain, can we re-use κ scores from prior domains as a quality prior on returning voters? Pilot scheduled Q3 2026.
Disagreement-as-signal. Instead of dropping low-κ raters, we can treat them as a separate cohort and report cohort-conditioned reward models, useful for value-pluralistic alignment in the spirit of Sorensen et al. (2024).
Empirical validation of the §5 loss. The κ-weighted, margin-aware DPO formulation is a specification. Appendix B sets out the protocol; the validating run is gated on first paying-customer datasets in Q3 2026.
L3 reward-model calibration. A trained reward model is shipped to a paying customer only after passing all six of the production-threshold metrics below. Customer-side reproduction with ±1pp tolerance against a hashed held-out test set is part of every L3 contract; missing the threshold by >5pp triggers a refund clause. The full protocol is in docs/raise/08_l3_calibration_methodology.md; the operative table is:
| Dimension | Metric | Production threshold |
|---|---|---|
| Agreement | Cohen κ vs human ground truth | ≥ 0.6 (or ≥ 85% of the human-human ceiling) |
| Accuracy | Pairwise on the customer’s hashed test set | ≥ 85% |
| Calibration error | Expected Calibration Error (ECE) | ≤ 0.07 |
| Reliability | Per-bucket calibration curve | Monotonic, no major dips |
| OOD detection | Abstention rate on adversarial / out-of-distribution inputs | ≥ 5% |
| Drift sensitivity | Time to alarm on distribution shift | < 7 days |
Calibration is relative to the human-human ceiling on the same domain (cf. §3.1 ceiling estimation): a model at \(\kappa = 0.45\) on a domain where humans agree at \(\kappa = 0.50\) is excellent (90% of ceiling); the same \(\kappa = 0.45\) on a domain where humans agree at \(\kappa = 0.80\) is mediocre. We report the ratio \(\kappa_{\text{model}} / \kappa_{\text{human}}\) on every customer benchmark, not just the absolute κ. The customer chooses the held-out test set, locks it via SHA-256 before training begins, and re-runs the κ calculation themselves; AimRank’s training script and evaluation script (scripts/{train_l3,eval_l3}.py) are the same code paths used to produce the published number, so the customer’s reproduction is bit-for-bit equivalent to ours.
Open preference-data infrastructure has progressed in scale (hundreds of thousands of pairs), in collection mode (LLM judges, expert vendors, engagement-priced platforms), and in downstream training algorithms (DPO, IPO, cDPO, KTO, SimPO). The measurement layer between raw votes and downstream training has not progressed correspondingly: most published preference corpora ship without per-annotator κ, without coverage histograms, without qualification gating, and without datasheet-style audit artefacts. This is the gap the present pipeline closes. Each of the seven layers is a small, deterministic, open-source-implementable piece of code; together they make the difference between a preference dataset that is plausibly clean and one that is demonstrably clean; under the EU AI Act’s Annex IV that demonstrability is now a regulatory floor, not a courtesy.
We invite the community to fork the reference implementation, contradict our defaults, and improve them. The methodology layer should be a public good, not a moat.
The author is the founder of AimRank UAB (Vilnius, Lithuania), the commercial entity whose reference implementation is described throughout this paper. AimRank UAB sells (1) a hosted SaaS labelling tool, (2) a domain-specific preference-data marketplace, (3) custom fine-tuned reward models, (4) a curated rater marketplace, and (5) an inference-time MCP preference API built on the methodology layers described herein. The author has a direct financial interest in the commercial adoption of this methodology.
To partially mitigate this conflict, the methodology code referenced at every “Reference implementation” line in §3–§7 is MIT-licensed and reproducible against any AWS account at https://github.com/AimRank1/aimrank-platform, and the empirical results in §3.2, §5.2, and §6 ship with deterministic seeds (seed=42) so any reader can re-run them without access to the author’s infrastructure. Reviewers are invited to challenge the falsifiable claims (judge accuracy, κ-weighted lift, methodology-layer outputs) by re-execution; the author commits to correcting the paper if the reported numbers do not reproduce. The author has not received any commercial revenue from the methodology described in this paper as of the v0.5 freeze date (2026-05-23).
Voter data and consent. Every vote in the AimRank reference deployment is captured under GDPR Article 6(1)(a) (consent for anonymous votes) or Article 6(1)(b) (contract performance for signed-in voters), with a per-vote provenance record. Voluntary voter-profile fields (industry, country, languages, expertise tags) require active opt-in and can be revoked via the per-user export/delete endpoints documented in §10 of the platform’s privacy policy. All data is hosted in the EU (OVHcloud DE1, Germany) under GDPR Article 28; no data is exported to non-EU jurisdictions.
Demographic axes and downstream training. The Layer 7 data card aggregates voluntary voter-profile fields at the cohort level only — no per-voter PII enters the RLHF export, and no demographic field is used as a weighting term in the per-vote loss. The §3.4 per-annotator κ scoring operates on a stable annotator identity (hashed in the export) but does NOT take demographic attributes as inputs. Discriminatory-impact monitoring is a downstream-consumer responsibility under EU AI Act Article 10(2)(f); the data card provides the cohort-level statistics a deployer needs to satisfy that obligation, but does not itself assert that any specific cohort distribution is representative of any specific protected population.
LLM-as-judge transparency. When the AimRank reference deployment uses an LLM judge as an additional vote source (in production, Qwen3.5-397B-A17B on OVH AI Endpoints) (§5.1, §6.1), every such vote is recorded with source="judge:<model_id>" and disclosed in §2.1 of the auto-generated data card. AI-cast votes are excluded by default from the RLHF export (include_llm_votes=false) so a downstream RM training pipeline cannot accidentally train on synthetic preferences mislabelled as human. This satisfies EU AI Act Article 50 (transparency to natural persons interacting with AI systems) and Article 13 (transparency to deployers) for the AimRank platform itself.
Reproducibility seeds and bias-sensitivity. All deterministic experiments in this paper pin seed=42. The §3.2 synthetic experiment uses Glicko-2 outcomes drawn from a fixed true-skill distribution; the §5.2 κ-weighted-lift experiment uses a fixed three-cohort rater quality structure. Both are documented in the reproducibility recipe (Appendix A) so a reader can vary the seed to assess bias-sensitivity directly.
This paper is released under Creative Commons Attribution 4.0 International (CC-BY 4.0). You may share, adapt, and redistribute the text and figures in any medium or format, including for commercial purposes, provided appropriate credit is given (canonical citation in Appendix C). The accompanying methodology code is released under the MIT License — see LICENSE at the reference implementation repository.
Allen AI. (2025). RewardBench 2: Advancing Reward Model Evaluation. arXiv:2506.01937. URL: https://arxiv.org/abs/2506.01937.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., & Munos, R. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences (introducing IPO, Identity Preference Optimisation). arXiv:2310.12036.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345.
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M. I., Gonzalez, J. E., & Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems 30. arXiv:1706.03741.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., & Sun, M. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. arXiv:2310.01377.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. arXiv:2402.01306.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92.
Glickman, M. E. (2012). Example of the Glicko-2 system. Boston University. URL: https://glicko.net/glicko/glicko2.pdf.
Hayes, A. F., & Krippendorff, K. (2007). Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures, 1(1), 77–89.
Herbrich, R., Minka, T., & Graepel, T. (2007). TrueSkill™: A Bayesian Skill Rating System. In Advances in Neural Information Processing Systems 19, pp. 569–576.
Hunter, D. R. (2004). MM algorithms for generalized Bradley–Terry models. Annals of Statistics, 32(1), 384–406.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z., Stevens, K., et al. (2023). OpenAssistant Conversations — Democratizing Large Language Model Alignment. arXiv:2304.07327.
Krippendorff, K. (2011). Computing Krippendorff’s Alpha-Reliability. Departmental Papers (ASC), University of Pennsylvania, paper 43. URL: https://repository.upenn.edu/asc_papers/43/.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., & Hajishirzi, H. (2024). RewardBench: Evaluating Reward Models for Language Modeling. arXiv:2403.13787. (Also: Findings of NAACL 2025, pp. 1755–1797.)
Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
Li, T., Chiang, W.-L., et al. (2024). From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv:2406.11939.
LMSYS. (2023). From live online Elo rankings to offline Bradley-Terry. LMSYS Blog, 7 December 2023. URL: https://lmsys.org/blog/2023-12-07-leaderboard/.
Meng, Y., Xia, M., & Chen, D. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2405.14734.
Mistral AI. (2024). Mistral-7B-Instruct-v0.3. Apache 2.0 release, 22 May 2024. URL: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3.
Mitchell, E. (2023). A note on DPO with noisy preferences and relationship to IPO (introducing cDPO, conservative DPO). URL: https://ericmitchell.ai/cdpo.pdf.
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19), 220–229. arXiv:1810.03993.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback (InstructGPT). In Advances in Neural Information Processing Systems 35. arXiv:2203.02155.
Pfeiffer, T., Gao, X., Mao, A., Chen, Y., Rand, D., & Chen, Y. (2012). Adaptive Polling for Information Aggregation. In Proceedings of the 26th AAAI Conference on Artificial Intelligence.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36. arXiv:2305.18290.
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (the AI Act). Official Journal of the European Union L 1689, 12 July 2024.
Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–325.
Settles, B. (2009). Active Learning Literature Survey. University of Wisconsin–Madison Computer Sciences Technical Report 1648.
Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C. M., et al. (2024). A Roadmap to Pluralistic Alignment. arXiv:2402.05070.
Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., & Kuchaiev, O. (2024). HelpSteer2: Open-source dataset for training top-performing reward models. arXiv:2406.08673.
All seven layers are open-source-implementable; the reference implementation lives at https://github.com/AimRank1/aimrank-platform on main. The code paths, with line counts at v0.2 freeze (2026-04-28), are:
| Layer | Code path | Key entry points |
|---|---|---|
| L1: Qualification | backend/api/services/iaa_metrics.py |
compute_ceiling_from_qualification |
| L2: Active sampling | backend/api/services/matchmaking.py |
MatchmakingService._information_gain_matchup, get_adaptive_matchup |
| L3: Coverage filter | backend/api/services/iaa_metrics.py |
filter_votes_by_min_raters, coverage_report |
| L4: Per-annotator κ | backend/api/services/iaa_metrics.py |
compute_per_annotator_agreement |
| L5: BT + bootstrap | backend/api/services/bootstrap_ci.py |
fit_bradley_terry, bootstrap_ratings |
| L6: Anomaly detection | backend/api/services/anomaly_detection.py |
AnomalyDetectionService.detect_all |
| L6: Decay tuning | backend/api/services/decay_optimizer.py |
DecayOptimizer.optimize |
| L6: Cycle detection | backend/api/services/transitivity.py |
detect_cycles |
| L7: Data card | backend/api/services/data_card.py |
build_summary, render_markdown |
| Scaffolds (§4) | backend/api/services/alignment_scaffolds.py |
normalize_constitution, effective_objectives, render_for_data_card |
Ten-minute reproduction. A reader with Docker installed can reproduce the seven-layer outputs against the seeded demo dataset with:
git clone https://github.com/AimRank1/aimrank-platform.git
cd aimrank-platform
make dev-up # boots backend, frontend, Postgres, Redis
make seed-demo # 7 demo rankings (text/image/audio); 50 synthetic
# votes each from a 12-persona pool; idempotent
curl -s "localhost:8000/api/v1/rankings/<id>/data-card"The endpoint returns the Layer-7 card as text/markdown by default; append ?format=json for the structured summary (same code path, machine-readable). The IAA numbers in §4 of that card are the live outputs of iaa_metrics.compute_agreement on the seeded dataset; the coverage histogram in §3 is the live output of iaa_metrics.coverage_report. The 25K-pair brand-voice corpus referenced in Appendix B is a separate, customer-provided dataset, not part of the demo seed.
L3 reward-model training. The reference training and evaluation scripts are at scripts/train_l3.py and scripts/eval_l3.py. Both have a --dry-run flag for CPU-only validation. The full L3 calibration protocol (six metrics, customer-side reproduction with ±1pp tolerance, refund-clause linkage) is specified in docs/raise/08_l3_calibration_methodology.md.
The following empirical claims are forward-looking. They are placed here, separated from the deterministic measurement layers in §3, so that no reader confuses the two. The validating run is scheduled for Q3 2026 against the first three paying L3 customers’ datasets, and the numbers will replace the placeholders in this appendix when the run completes.
B.1 Hypothesis. Each of the four data-quality signals produced by Layers 1, 3, 4, and the per-vote confidence field contributes a non-overlapping improvement in pairwise reward-model accuracy on a held-out test slice, relative to a vanilla DPO baseline trained on the same raw vote stream. The aggregate improvement closes a measurable fraction of the gap from vanilla DPO to the human-human κ ceiling on the same domain.
B.2 Dataset. A 25K-pair brand-voice preference dataset collected via AimRank from a single SME-led annotation team (\(n = 18\) raters, of whom an estimated 12–14 will clear qualification at \(\tau = 0.80\)). The dataset is partitioned into a 20K training slice and a 5K held-out test slice; the test slice is hashed (SHA-256) and the hash is committed to the repository before training begins, in the protocol described in 08_l3_calibration_methodology.md §29.6.
B.3 Configurations. Each row below is a strict superset of the row above; differences isolate the contribution of a single layer.
| # | Configuration |
|---|---|
| 0 | Zero-shot judge: vanilla Claude Haiku 4.5 prompted with the constitution. |
| 1 | Vanilla DPO on raw 25K pairs (no Layer-1 / 3 / 4 filtering, \(w_i = 1\), \(c_i = 1\)). |
| 2 | + Layer 1 (qualification filter): votes from un-qualified voters dropped. |
| 3 | + Layer 4 (per-annotator κ weighting in \(w_i\)). |
| 4 | + IPO/cDPO confidence margin ($c_i = $ vote.confidence). |
| 5 | + Constitutional auxiliary heads (3 declared principles). |
B.4 Metric. Pairwise accuracy on the held-out 5K slice with 95% bootstrap CI (\(B = 1{,}000\)). Aggregate gain vs configuration 1 reported with a paired-bootstrap significance test (\(\alpha = 0.05\)); per-row CIs reported non-overlapping where applicable.
B.5 Ceiling baseline. Pairwise human-human κ on the same brand-voice domain, computed per iaa_metrics.compute_ceiling_from_qualification over the SME panel, reported alongside the model accuracy. The “% of human ceiling closed” figure is what we expect investors and procurement officers to quote.
B.6 Pre-registration. The hypothesis, dataset partition, configurations, and metric above will be committed to git at a tagged commit before the validating run starts. Deviations from pre-registration will be reported in the published numbers in plain language. This protocol mirrors the L3 calibration protocol in 08_l3_calibration_methodology.md and is designed to make any post-hoc claim auditable.
B.7 Reporting venue. Numbers will be released as v0.3 of this paper plus a reference Colab notebook (docs/examples/researcher_quickstart.ipynb) and a public dataset card on Hugging Face (CC-BY-4.0) for any non-confidential portion of the dataset.
| Symbol | Meaning |
|---|---|
| \(\tau\) | Qualification pass threshold (default 0.80) |
| \(\phi_a\) | Glicko-2 rating deviation of entity \(a\) |
| \(\mu_a\) | Glicko-2 rating of entity \(a\) |
| \(\beta\) | Glicko-2 system noise (default 200) |
| \(p_{ab}\) | Win probability of \(a\) over \(b\) under Glicko-2 |
| \(H(p)\) | Binary entropy in bits |
| \(I(a, b)\) | Expected information gain of comparing \(a\) and \(b\) |
| \(V_{ab}\) | Set of distinct voters who judged pair \(\{a, b\}\) |
| \(k\) | Multi-rater coverage threshold (min_raters_per_pair) |
| \(\kappa_u\) | Cohen’s κ of voter \(u\) vs majority consensus |
| \(w_i\) | Per-sample weight in the κ-aware DPO loss |
| \(c_i\) | Per-vote confidence (\(\in [0, 1]\)); IPO/cDPO margin |
| \(B\) | Bootstrap iteration count (default 1{,}000) |
| \(T_d, \rho, C\) | Anomaly thresholds: dead-days, dominance fraction, instability count |
@misc{sadaunykas2026datasheets,
author = {Sadaunykas, Gediminas},
title = {Datasheets for Reward Models:
A Seven-Layer Pipeline for Auditable Preference Data},
year = {2026},
month = may,
howpublished = {arXiv preprint},
note = {Working draft v0.5, 2026-05-23. Reference implementation:
\url{https://github.com/AimRank1/aimrank-platform}.
Paper licensed CC-BY 4.0; code MIT.},
url = {https://aimrank.io/paper},
}Plain-text citation:
Sadaunykas, G. (2026). Datasheets for Reward Models: A Seven-Layer Pipeline for Auditable Preference Data. Working draft v0.5, 2026-05-23. AimRank UAB, Vilnius. CC-BY 4.0.
Comments, corrections, and pull requests are welcome. The reference implementation lives at https://github.com/AimRank1/aimrank-platform . Bug reports for the methodology code should be filed against backend/api/services/iaa_metrics.py, bootstrap_ci.py, data_card.py, alignment_scaffolds.py, matchmaking.py, anomaly_detection.py, decay_optimizer.py, or transitivity.py and tagged area:methodology.