Audit
How wrong is one judge, grader, label set or claim, and what fixes it.
- Report with error rates and intervals
- Fixes or patches where possible
- Repository to rerun everything
- Review call with your team
$ evaluation audits · for AI teams, vendors and investors
Your AI's scores decide what ships. Are they right?
Benchmarks, LLM judges, human labels and training rewards all feed release decisions. I measure how wrong they are, correct them with small random checks, and hand over code that keeps checking after I leave.
PhDsignal processing & ML
10+ yrsresearch & industry
230+Scholar citations
5sole-author papers, 2026
Fig. 1 one training run, three numbersQwen2.5-0.5B · GSM8K
AI judge says
0.66
True accuracy
0.26
Audited, 1 in 8 checked
0.22
0.25.50.751
judge overstated accuracy by 40 points
01problems
If one of these sounds familiar, I can help
Forty problem types, grouped by where they show up. Rows marked evidence link to my public work on that exact problem; the rest use the same methods.
Training a model
RL · fine-tuning- Reward goes up, real skill doesn't. The model games its reward.evidence
- The environment's grader pays wrong answers.evidence
- Answers get longer and more flattering as the reward model is over-optimized.
- Preference labels disagree in RLHF or DPO data.
- Synthetic training data carries errors nobody measured.
- Benchmark questions leaked into training data.
- A fine-tune breaks skills the model already had.
Measuring quality
judges · benchmarks- The AI judge is biased by order, length or its own model family.evidence
- The judge marks correct answers wrong.evidence
- Nobody knows how often the judge agrees with people.evidence
- "2% better" on a sample too small to show it.evidence
- The test set doesn't look like real user prompts.
- Nightly regression alerts are mostly noise.
- Good offline, worse with real users.
Products and agents
RAG · agents · production- The hallucination rate is a guess with no error bar.
- RAG retrieves the wrong documents or answers beyond them.
- An agent fails and nobody can say where it started.evidence
- Agents fall apart on long tasks.evidence
- Wrong tool calls and wrong parameters.
- A model or prompt update quietly makes the product worse.
- The agent writes its own test labels, so its blind spots never show.
- Quality drifts in production and nobody is alerted.
Human data
raters · labelsBuying and investing
vendors · due diligenceSafety
guardrails · red teaming- Guardrail classifiers with unknown miss and false-alarm rates.
- Too many refusals, or too few.
- Prompt injection steers an agent through outside text.
- "No issue in 100 red-team tries" with no bound on the real risk.
Speech, vision, edge
ASR · TTS · CV- ASR and TTS quality measured inconsistently.evidence
- Generated product images drift in colour, logo or shape.
- Accuracy lost when a model is quantized for edge devices.
- Speed or position estimates from video with no error budget.
Compliance
EU AI Act · assurance- EU AI Act testing evidence that must be reproducible.
- An outside assessor or insurer needs to verify reported numbers from a small sample.
02evidence
What these audits have already found
All public, all reproducible from the linked code and data.
2 of 21
reinforcement learning graders paid replies with no correct answer. After the fixes those replies scored zero, while 4,997 of 5,000 honest answers scored as before.
rl-env-audit · 60 public environments scanned
1st → 5th
rank of the model that is best by the answer key, once only the grader changes. Same answers, three LLM judges from three labs.
leaderboard-audit · 6 models, 200 items, $0.49
63%
of the time a three-rater majority matched the expert safety label. Even all 123 raters only reached 64 to 65%.
rater-audit · DICES-350, real raters
42 → 5
false "incorrect" verdicts once the judge had to quote the false sentence and the quote was checked. The recall cost is reported too.
score-audit · 300 human-labelled answers
−38%
change in a published benchmark's mean when one stated voting rule moves to unanimous. 19 of 35 models change rank.
judge-ceiling · 39/39 scores reproduced
5 of 6
languages where a speech vendor's accuracy tier flipped with scoring choices the claim never stated.
asr-tier-audit · FLEURS, 200 sentences each
03how we work
Fixed scope, fixed price, clear deliverables
Start with one number your decisions rely on. Each engagement ends with something your team can run again.
Build
An evaluation pipeline your team keeps using after I leave.
- Audited scores with error bars
- Drift and regression alarms
- Tests in CI
- Documentation and handover
Report or paper
Your results written up so outsiders can check them.
- arXiv-ready applied paper or report
- Every number traced to code and data
- Statistics a reviewer will accept
milestone paymentscontract · B2B · EORNDA or open sourceweekly written updatesUS & EU time zones
04about
Vahit Feryad, PhD
Applied research engineer, AI evaluation and reliability. Based in Istanbul, working remotely with teams in the US and Europe.
PhD in electrical and electronics engineering (signal processing and machine learning). Working in applied AI since 2014, across research and industry, with 230+ citations on Google Scholar. In 2022 and 2023 I built AI-based sports analytics as a remote contractor for SportsMEDIA Technology, the US company behind live TV sports graphics.
- Two Guesses for the Price of One
acceptance tests for the graders of RL environments - Noise Is Not Noise When the Policy Chooses It
unbiased RL from an exploitable judge with random audits - The Normalizer Decides the Tier
certifying per-language speech recognition claims - Show Me the False Claim
evidence-backed grading for LLM judges - LongHaul-Bench
long-horizon reliability of self-improving LLM agents
$ contact
Have a number you need to trust?
Send one paragraph on the score, judge or claim you're unsure about. I'll reply with whether I can help, a scope and a fixed price.