$ applied research · AI evaluation & reliability

Measuring whether AI measurements can be trusted

AI models are graded by other AI models, by small panels of people and, during training, by reward code. I study when those scores hold up and when they don't, and I build the checks that fix them: judge bias, rater reliability, reward hacking and leaderboard statistics. Every result is open and reproducible.

Read the studies → Audit services Get in touch

PhDsignal processing & ML

10+ yrsresearch & industry

230+Scholar citations

Fig. 1 judge swap test6 models · 200 items

rank of the true #1 model#1

Same answers, different grader. The answers of six models were held fixed; only the grader changed. The model that is best by the answer key drops as far as 5th under three LLM judges from three labs. Data: leaderboard-audit.

01findings

Consistent is not the same as correct

A judge that gives the same answer every time looks reliable. These results show why that check is not enough.

−38%

change in a published benchmark's mean score when one stated voting rule moves from 3-of-4 to unanimous. 19 of 35 models change rank.

judge-ceiling · ASR benchmark, 39/39 scores reproduced

6.0 pts

bias against one system from a judge that agreed with itself 98.3% of the time. Repeat grading could not reveal it; only a second grader did.

leaderboard-audit · Holm-corrected p = .005

r = 0.35

best automatic judge's agreement with human listeners on voice identity, the TTS dimension where judges are weakest.

judge-ceiling · public TTS judge leaderboard, 8 × 7

Fig. 2 per-model judge biaspoints vs answer key

One judge, two directions. Same judge, same temperature, 98.3% self-consistent across three gradings.

Fig. 3 rater-panel calculatorSpearman–Brown

ρ_k = kρ₁ / (1 + (k − 1)ρ₁)

single-rater reliability ρ₁ 0.40 raters per item k 4

panel reliability ρ_k0.73

raters needed for ρ_k ≥ 0.806

How many listeners do you need? Panel reliability rises with raters, but with diminishing returns. Sizing the panel from measured single-rater reliability avoids paying for raters you don't need.

02talk

Reliable AI, in one talk

A short overview of the studies on this page and the question behind them: when can an AI score be trusted?

Video reliable AI overviewnarrated by my digital twin

▶ play · opens in place

Reliable AI. If the player does not load, watch it on Google Drive.

$ what the talk covers

Why a clean number can still be wrong

The same pattern shows up across LLM judges, human rating panels, speech benchmarks and long-running agents: a system that looks fine at one point and degrades along an axis a single score cannot show.

judge bias that repeat grading cannot see
human yardsticks with unmeasured reliability
memory that slowly misleads an agent

03research

Open studies

Each project regenerates its published numbers in CI from shipped data, states its limits, and reports negative results next to positive ones.

real dataRL environments
60 public environments scanned
21 inspected · ~$1
13 toy graders validated

RL-Env-Audit

Does an RL environment's grader ever pay for a wrong answer?

Method: Call each environment's own scoring code with ten replies that should never be paid (empty, the prompt echoed, grader injection, six two-guess forms); a cross-task rule separates real hedging holes from graders that read one answer.
Finding: 19 certified (each check paid on <4.9% of tasks, 95% Clopper–Pearson), 2 holes. Fixes cut them from 466/500 and 11/299 tasks to 0, with 4,997/5,000 honest answers paid as before.

2 of 21

graders paid wrong replies, fixed to 0

code & paper →

real dataRL rewards
Qwen2.5-0.5B · GRPO + LoRA
GSM8K
proof of unbiasedness

Reward-Audit · Audited Rewards

Can a judge the policy learns to exploit still give an honest reward?

Method: Pay the judge's calibrated score plus a correction on 1 in 8 randomly audited outputs; prove the reward unbiased with its exact variance cost; watch the judge with an anytime-valid monitor.
Finding: The judge alone read 0.66 against a true 0.26; the audited reward read 0.22 against 0.23. A reference-free judge paid 43% of wrong answers; the monitor flagged it after 5 audited outputs and never fired on the exact verifier.

0.66 → 0.22

judge vs audited, true 0.23

code & paper →

real datahuman raters
DICES-350
123 real raters
expert safety labels

Rater-Audit

Is the human gold label right?

Method: Audit every stage of an expert-rating pipeline: majority against expert, rater selection on a small gold set, hidden canary items as a per-batch quality gate.
Finding: A 3-rater majority matched the expert label 63% of the time; all 123 raters reached only 64–65%. Five raters picked on a gold set matched 69% at the same cost. Canaries caught a model-copying rater 87% of the time with 0.8% false alarms.

63%

majority agrees with the expert

code & data →

preprintLLM judges
300 TruthfulQA answers
human labels
2 judge models

Score-Audit

How often does a judge call a correct answer wrong?

Method: Require the judge to quote the false sentence, then check the quote against the answer.
Finding: False "incorrect" verdicts fell from 19 to 7 (GPT-4o-mini) and from 42 to 5 (Gemini-2.5-Flash-Lite). The recall cost is reported: 91→78% and 96→53%.

42 → 5

false "incorrect" verdicts

code & preprint →

paperspeech
Scribe v2 · 6 languages
FLEURS · 200 sentences each
anytime-valid test

ASR-Tier-Audit

Does a vendor's per-language accuracy claim survive the scoring choices it doesn't state?

Method: Rescore with every reasonable normalizer and tokenization; certify or refute each claimed tier with an anytime-valid test.
Finding: The verdict depends on unstated scoring choices in 5 of 6 languages; the normalizer alone moves Finnish between 3.2% and 5.4% WER. Whisper's normalizer deletes bracketed spoken words and splits combining marks.

5 of 6

languages where scoring flips the verdict

code & paper →

real dataLLM judges
6 models · 200 GSM8K items
3 judges from 3 labs
124 tests · $0.49 API

Leaderboard Audit

Does an LLM grader treat every model the same?

Method: Hold every model's answers fixed, vary only the grader; compare against a deterministic answer key with exact nonparametric tests.
Finding: The answer-key #1 ranked 2nd, 4th and 5th under the judges. With 200 items the minimum detectable effect is 5.1–7.6 points: two tiers, not six ranks.

#1 → #5

true winner under judge swap

code & data →

real dataspeechhuman panels
arXiv:2608.19936 reproduced
TTS judge board 8×7
109 tests

Judge-Ceiling

Is the human yardstick behind a leaderboard reliable enough to rank on?

Method: Reproduce published scores from released data; derive the reliability bound each judge correlation implies (ρ₁ ≥ r²); test rater screens on simulations with a known answer.
Finding: One stated threshold cut the mean by 38% and reranked 19 of 35 models. A two-sigma outlier gate overstated reliability by 44%. One result contradicted my own hypothesis and is reported.

−38%

mean score, one rule change

code & data →

preprintagentsedge
Qwen2.5-3B Q4_K_M
~1,000 episodes / arm
4 CPU · 8 GB

LongHaul-Bench

Does an agent's accumulated memory improve it, or slowly rot it?

Method: Seed-reproducible long-horizon protocol with corruption dose-response and an inert control arm; Holm-corrected tests; public CI rerun under resource caps.
Finding: Correct recall +9.4 points, misleading recall −19.8. Plausible corruption hurts; crude corruption does not. The same memory rule gave Llama-3.2 +24.6 and Qwen +0.8. Limit: an LLM-free heuristic beats every agent arm.

−19.8

points from misleading memory

code & preprint →

PyPIagents
747 tests
~100k events in ~3 s

Runopsy

Where did an agent run start going wrong, not just where it stopped?

Method: Record the run, localize the likely failure onset with evidence, test the cause by counterfactual replay in a disposable sandbox.
Finding: 94.4% top-1, 100% top-3 on a labelled synthetic suite (baselines 22.2%, 50%). 0% top-1 on expert-labelled TRAIL / Who&When traces, published as a limit.

94.4%

top-1 onset localization

code →

simulationvoice agents
40-turn sessions
Llama-3.1-8B judge

VoiceHaul

What happens to voice agents across long conversations?

Finding: The agent the judge panel rated most empathetic failed every 40-turn conversation; the calibrated one failed none. Estimates how many human ratings one judge rating is worth.

all failed

long sessions of the agent judges rated "most empathetic"

code →
live demo →

04methods

How I check a number

Three questions, asked of every reported score.

Q1 · BIAS

Does the judge treat every system the same?

Per-model bias against an independent second grader, not just repeat grading of the same judge.

bias_m = score_judge(m) − score_ref(m)

Q2 · RESOLUTION

Is the gap bigger than the test can see?

A minimum detectable effect for every reported difference; tiers instead of ranks when models cannot be separated.

report a gap only if |Δ| > MDE_{0.8, α}

Q3 · RELIABILITY

Is the human yardstick itself reliable?

Single-rater reliability, panel size and the effect of screening rules, measured rather than assumed.

ρ₁ ≥ r² (judge–human correlation bound)

05audits

Evaluation audits for teams

The same methods, applied to your benchmark or internal eval, under NDA. Useful before you publish a claim, launch a release or size a human-rating study.

Process2-week pilot → monthly

day 1
Scoping call
Pick the one number your decisions rely on most.
week 1
Reproduce & test
Judge bias, rater reliability, minimum detectable effect.
week 2
Verdict
Plain-language report, reproducible code, review call.
monthly
Release checks
Before each release goes public. Cancel any month.

Outputone verdict per claim

"New model is better than v2"✓ supported

"Ranked #1 for voice quality"! needs more data

"Beats competitor X on naturalness"✗ not supported

Illustrative example. Each verdict comes with its reason and the fix. One-page overview (PDF) → · 40 problem types and fixed-scope packages →

06about

Vahit Feryad, PhD

Senior applied research engineer. PhD in electrical and electronics engineering (signal processing and machine learning), 10+ years across research and industry, from embedded and edge systems to LLM agents and speech models.

Vahit Feryad, PhDIstanbul · AI evaluation & reliability

LLM-as-judgebenchmark statisticsrater reliabilityspeech / TTS / ASR evallong-horizon agentsedge & quantized inferenceInspect AIPyTorchTensorRT

Two Guesses for the Price of One: Acceptance Tests for the Graders of Reinforcement-Learning Environments. Paper, 2026.
Noise Is Not Noise When the Policy Chooses It: Unbiased Reinforcement Learning from an Exploitable Judge with Random Audits. Paper, 2026.
The Normalizer Decides the Tier: Measuring and Certifying Per-Language Speech Recognition Accuracy Claims. Paper, 2026.
Show Me the False Claim: Evidence-Backed Grading Cuts False Error Verdicts from LLM Judges. Preprint, 2026.
LongHaul-Bench: A Reproducible Benchmark for Long-Horizon Reliability of Self-Improving LLM Agents Under Industrial Edge Constraints. Preprint, 2026.
Efficient Design of Energy Disaggregation Model with BERT-NILM Trained by AdaX Optimization Method for Smart Grid. Energies, 2021.
New Design of a Supervised Energy Disaggregation Model Based on the Deep Neural Network for a Smart Grid. Energies, 2019.
Multi Objective Clustering for Wireless Sensor Networks. Expert Systems with Applications, 2016.

full record on google scholar →

2026–now
Independent Applied Research Engineer
AI evaluation & reliability · open-source audits and papers
2024–now
Data Analytics Manager
Turkuvaz Media Digital, Istanbul
2022–2023
Computer Vision Engineer (contractor)
SMT SportsMEDIA Technology, Durham NC · remote
2017–2022
AI Researcher
ASIS Automation and Fueling Systems
2016–2017
R&D Engineer
Surteks Automotive
2015–2016
Electronics & Communication Engineer
Delphisonic
2014–2021
PhD, Electrical & Electronics Eng.
Karadeniz Technical University

$ contact

Have a number you need to trust?

Research collaborations, evaluation audits and applied evaluation work. Based in Istanbul, available remotely for teams in the US and Europe.

[email protected] email github.com/vahit19 code google scholar papers linkedin profile video overview talk