How to Lie to Yourself With an Eval Judge
Three ways our eval harness produced wrong accuracy numbers: a judge that couldn't see correct answers, a heuristic that accepted almost anything, and weeks of measuring code we don't ship.
Our benchmark accuracy jumped from 75.5% to 91.7% overnight. We didn't touch the model, the prompts, or the pipeline. All 16.2 points came from fixing the measurement layer: the stored gold answers and the judge that scored against them. It is the largest single correction in the benchmark's history.
A sixteen-point swing with no system change means the number was never measuring the system in the first place.
Quick context so the numbers mean something. Onta is a system that auto-builds a knowledge graph from your data and answers natural-language questions by generating SPARQL over it. To know whether that works, we run an internal holdout benchmark (currently 302 questions over 26 knowledge graphs the system has never seen) with execution-grounded gold answers and an automated judge deciding right or wrong. Everything in this post is from that internal holdout. We haven't calibrated against external benchmarks like Spider 2.0 or BIRD yet, so nothing here is a claim about the field.
Over the life of that harness, it has lied to us three times that we know of: under-reporting twice, over-reporting once. Every time, the aggregate number looked plausible.
Plausible is the problem. An obviously broken number gets fixed the same day.
Bug one: the judge that couldn't see correct answers
Our gold answers come from a generation pipeline: an LLM writes a question and its reference SPARQL, we execute the SPARQL against the graph store, and the result gets saved as ground truth.
Except it didn't save the result. It saved the first ten items.
A formatting helper in our gold-generation pipeline truncated the stored answer to 10 items and appended a "... (N total)" suffix. That formatting was written for display. It ended up in the ground-truth store, and the judge treated those ten visible items as the complete gold answer.
A question like "list the complaints about prepaid cards" has, say, 400 correct rows. The model generates correct SPARQL and returns a 15-item slice of those 400. The judge compares it against the first ten items the formatter kept. The intersection is frequently near zero. Marked wrong. The SPARQL was correct; the judge couldn't see it.
When we audited a 5× multirun, 56 of that audit's 68 "wrong" answers were this exact false negative. Not model failures. Judge blindness.
The fix had to touch both layers. The gold store is now rebuilt by re-executing every reference query against the live graph (Neptune, in our case) and storing the full result set, up to 2,000 items per question, alongside the display version. Yes, 2,000 is still a truncation. The honest description of the fix is that we moved the cliff far enough out that it stops biting, not that we removed it.
The judge was rewritten as a subset-membership check: parse the model's items, normalize them, verify that at least 70% are provable members of the full gold set, and require a minimum of five items so one lucky match can't pass. The normalization step matters more than it sounds. Gold might store 12345 while the model returns .../entities/ConsumerComplaint/12345; the judge extracts URI path tails before comparing, or half your true positives evaporate on formatting alone. Small gold sets (under 20 items: counts, single-value lookups) still go through exact matching.
Before you object: a membership check is precision-only, and I know it. A model that answers every list question with five safe items passes. Worse, an over-constrained query ("prepaid-card complaints in California") returns a valid strict subset of the gold set and gets marked correct even though the SPARQL is wrong. Today's judge cannot see that failure class. And the 70% and five-item thresholds were chosen by judgment, not derived from anything. More on why this worries me at the end.
Result: 75.5% to 91.7% on the identical holdout, with zero changes to the system under test. It had been that good for weeks. We couldn't see it.
The rule I took from it: truncation for display is fine; truncation in the ground-truth store is never fine.
Bug two: the heuristic that accepted almost anything
If the first bug tempts you to conclude that judges err strict, this same harness had already erred the other way.
Earlier in the benchmark's life, someone (me) added an auto-accept heuristic for long list answers: if the model's answer contained multiple items, accept it as close enough. That single heuristic inflated the overall score from 79.8% to 94.6%. We caught it only through an independent audit of the judge's decisions, and the heuristic was deleted outright.
The bug-one fix superficially resembles the bug-two sin. We made the judge more permissive and the score jumped 16 points. It isn't the same thing: the old heuristic accepted any multi-item answer regardless of content, while the membership check verifies every returned item against the gold set. But the resemblance is exactly why the recall hole I flagged above stays on my mind.
The same harness ran too loose in one era and too strict in the next, and neither number looked wrong at the time. That's the part that should scare you. Audit the accepts as hard as the rejects.
Bug three: weeks of measuring code we don't ship
Our production question-answering path retrieves few-shot examples from an example bank before generating SPARQL. For several weeks, the eval harness's client-side path silently skipped that retrieval for every multi-table knowledge graph, 24 of the 26 in the holdout. So 92% of the KGs were being measured on a degraded configuration no user would ever touch, and we were under-reporting our own system while believing we were being rigorous.
"At least it erred in the honest direction" is false comfort. A harness that silently drops a production feature in one direction can silently drop an anti-cheat gate in the other. And it wasn't free: we dispatched roughly five full eval runs before catching it, each costing $50 to $200 in API budget plus one to three hours of waiting and analysis.
Two structural fixes came out of it. Parity between the eval harness and the production endpoint became a tested invariant rather than an intention; if the harness invokes a code path production doesn't, it fails loudly instead of producing a plausible number. And every eval dispatch now goes through a mandatory pre-eval checklist covering pipeline parity, the anti-cheat stack, gold-file integrity hashes, and apples-to-apples config across every system being compared: same model ID, same gold, same judge commit, same seeds. No checklist, no run. My favorite part of that document is a table titled "Things we have CAUGHT after the fact (and never want to repeat)," currently fourteen entries long. This bug is entry eleven.
What a benchmark number actually claims
The common shape of all three: your benchmark number is a claim about your judge and your gold data at least as much as it is a claim about your model. Across all three, the system under test never changed. Only the measurement did.
For the record, the corrections cut both ways. Deleting the auto-accept heuristic cost us 14.8 points. And when we later got suspicious of our own headline, we attacked it: the v1 holdout's 91.7% included seven clinical-trials knowledge graphs whose schemas were near-identical to training variants. We estimated the risk at roughly five points of schema-memorization inflation. We had flagged that risk ourselves, and then shipped a headline number on it anyway.
So we rebuilt the holdout. The seven offenders got deleted, a schema-adjacency gate went in (a hard type-and-relationship overlap rule plus a Jaccard similarity cap; the closest surviving KG scores 0.115 against training), and the domain mix expanded into legal and scientific data.
The rebuilt number came back 91.4% (Wilson 95% CI: 87.7–94.1%, holdout v2.0). Essentially flat. One honest complication: the rebuild coincided with a model upgrade, Gemini 2.5 Flash to Gemini 3 Flash Preview, so a stronger model could in principle be masking some memorization drop. With that caveat on the table, the flat line is still the result I lean on most, because a real collapse would have meant the old headline was memorization.
The part we still don't trust
The anti-cheat stack gets judge-level rigor now: holdout questions are blocklisted from example retrieval at ask time, near-duplicate examples are similarity-capped, and the example bank refuses to ingest holdout KGs at all. One near-miss still makes me sweat. A fine-tuning data builder defaulted to pulling from every ground-truth file and nearly shipped 222 holdout examples inside a training set; every builder since runs a contamination check that hard-exits if any holdout KG shows up in a training prompt.
Plenty is still unfixed, and I'd rather say so here than have you find it later. The gold pairs are LLM-generated, and our 20% human-review target for them hasn't been met. We haven't calibrated against external benchmarks, so 91.4% is a claim about our holdout, not about the field. Post-majority-vote run-to-run variance is unmeasured; we know Flash-class models swing five to ten points per seed at temperature zero, which is why nothing gets reported single-seed, but the residual noise floor is an open item. And by base rates alone, a fourth self-deception is sitting in the harness right now. My leading candidate is the recall hole in bug one's fix. History says I should assume it flatters us, which is why the accepts get audited hardest.
If you run LLM evals: what's the worst false-accept your judge ever produced? Email hi@getonta.com or leave it in the comments, and I'll trade you our sanitized pre-eval checklist and the other thirteen entries in the caught-after-the-fact table. I read and reply to everything. If you'd rather talk it through, I keep office hours: 30 minutes, no pitch, bring your ugliest eval.