Skip to main content
External-benchmark calibration · Run 2026-04-28 · Not the current production judge

April 2026 judge calibration: 90.89% on RewardBench full

Three independent public benchmarks. 3,385 pairs total, €3.59 cost. Wilson 95% CI tightened to ±1pp at full sample. Position bias 0.005. Three reproduction scripts, pinned with seed=42; the package is available on request.

Judge model, as measured on 2026-04-28: claude-haiku-4-5 via AWS Bedrock (eu-central-1). Production has run Qwen3.5-397B-A17B on OVH AI Endpoints since 2026-07-09; these figures have not been re-measured on it. A/B-randomised on every call. Per-pair cost €0.001; vs Scale AI list pricing of €1.50–€5.00/pair, that's 1,500–5,000× under market.

Read the methodology paperRequest the reproduction package

Headline results

One judge, three benchmarks, three different ground-truth shapes. Each run is seed-pinned and keeps full per-pair rationale logs.

N=2,985
90.9%
CI95: [89.8%, 91.9%]
RewardBench (`filtered`, full)
Ground truth: Human pairwise · Lambert et al., 2024
Industry-standard RM evaluation; tight CI at full sample.
€3.01 total€0.000978/pair
Script + result JSON: on request Dataset on HuggingFace
N=200
79.0%
CI95: [72.8%, 84.1%]
HelpSteer2 `preference`
Ground truth: Human multi-axis (5 axes) · Wang et al., 2024
Accuracy rises monotonically with preference strength (76% → 84%), a calibration validity check.
€0.30 total€0.0015/pair
Script + result JSON: on request Dataset on HuggingFace
N=200
82.0%
CI95: [76.1%, 86.7%]
UltraFeedback `binarized`
Ground truth: GPT-4 as judge · Cui et al., 2023
Judge-to-judge consistency vs GPT-4: sanity check, not human-agreement.
€0.28 total€0.0014/pair
Script + result JSON: on request Dataset on HuggingFace
Scale AI · per-pair list price
€1.50 – €5.00
Labour-priced; published list rates. EU residency: no.
April 2026 run · per-pair cost
€0.001
Bedrock list pricing for claude-haiku-4-5 in eu-central-1. Engagement-priced labour cost regime, not paid-per-task. 1,500–5,000× under Scale. The production judge (Qwen3.5-397B-A17B on OVH AI Endpoints) is priced differently and is not costed here.

RewardBench breakdown: strengths and the honest weakness

Top-quartile across code-helpfulness, refusal, and safety subsets. Adversarial-distractor subsets (llmbar-adver-*) are the known LLM-as-judge failure mode, exactly where AimRank's Layer-1 qualification gates and Layer-4 per-annotator κ filtering carry the most marginal value.

Strong subsets

  • hep-* (code helpfulness, 6 langs)n=98497.0%
  • xstest-should-respondn=25097.6%
  • xstest-should-refusen=15498.1%
  • donotanswern=13685.3%
  • math-prmn=44782.8%
  • mt-bench-hardn=3781.1%

Adversarial weakness (mitigated by Layers 1 + 4)

  • llmbar-adver-neighborn=13464.2%
  • llmbar-adver-GPTInstn=9273.9%
  • llmbar-adver-manualn=4673.9%
  • llmbar-adver-GPTOutn=4774.5%
Why we publish this: most LLM-as-judge papers omit the llmbar-adver-* subsets because they embarrass the judge. Owning the weakness publicly is the precondition for the methodology layer's commercial argument: where the judge is unreliable you need qualified human votes, and AimRank measures judge–human agreement (κ) so you can see where that is. Routing low-confidence judge calls to humans automatically is not built yet.

Calibration validity check

On HelpSteer2 the dataset publishes a preference_strength from 1 (slight) to 3 (strong). If our judge is calibrated, accuracy should increase monotonically with strength: easier pairs are easier. It does:

Strength 1 (slight)
76.3%
n=97
Strength 2 (clear)
80.0%
n=60
Strength 3 (strong)
83.7%
n=43
A flat curve would mean the judge is guessing. A monotonic increase proves the judge is tracking the same signal humans track, just less reliably on close pairs.

Position bias + latency

Position bias (RewardBench full)
+0.005
(A_picks − B_picks) / N, after randomisation
Below the 0.20 threshold typically flagged in LLM-as-judge literature. The judge has no detectable A-side preference once we A/B-swap each pair.
Decision time (RewardBench full)
2,078 ms
median · p95: 7,801 ms
Sub-3-second median is suitable for batch judging; the L5 MCP inference API uses smaller prompts to hit sub-second p95 for agent-loop integration.

Reproduce these numbers

Three scripts, all pinned with seed=42, run from an AWS account with Bedrock access in eu-central-1; total cost to replicate the 3,385-pair table above: €3.59. They are not public yet: the reproduction package is available on request.

# What the package runs (~30 min wall, ~€3.60 cost):
$ python scripts/benchmark_judge_vs_human.py --n 2985
$ python scripts/benchmark_judge_vs_helpsteer2.py --n 200
$ python scripts/benchmark_judge_vs_ultrafeedback.py --n 200

Result JSONs land in data/benchmarks/judge_*.json with full per-pair rationale logs for audit.

Want preference data with κ + audit trail on every export?

An LLM judge plus human votes, in your domain. Free tier covers 1,000 votes / month. No credit card.

Start a ranking