AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. A panel of five models flagged 236 of the reference answers as broken, 31.1 percent of the set. With those removed, Claude Opus 5.5 went from 74.9 to 87.8 percent, and Gemini 4 Argon went from 64.3 to 78.6 percent.
The 35-second reel: 236 of 759 reference answers flagged, and both featured scores moved.
759 questions, 236 flagged
BIRD-SQL is a text-to-SQL benchmark, and its answer key is the ground truth every model is scored against. AIMultiple ran five models as judges over 759 of its questions. Three of five had to agree before a reference was flagged. That bar caught 236 references, nearly a third of the audited set. The other 523 stayed unflagged.
74.9 to 87.8
The flagged answers were not noise. Excluding them moved Claude Opus 5.5 from 74.9 to 87.8 percent, a 12.9 point jump. Gemini 4 Argon moved from 64.3 to 78.6 percent, a 14.3 point jump. Both scores moved in the same direction by double digits, which is what you expect when the answer key itself is carrying errors.
The takeaway
Check the answer key first. A benchmark score is only as honest as its ground truth, and a broken reference punishes every model that gets the question right. Before you trust a leaderboard gap, ask whether the key has been audited, and by whom.
Caveats
Five models did the auditing, and three of five had to agree before a reference was flagged. Independent human annotators have not repeated the review, so the 236 count is a model judgment, not a confirmed error count. The two models also ran under different setups, Claude Code versus OpenRouter, so their score gap is not a clean model-to-model comparison. Both featured scores moved once the 236 flagged answers were excluded.
Sources
- AIMultiple, AIM-Text-to-SQL Benchmark, updated Nov 10, 2026: https://aimultiple.com/text-to-sql
This post started as an Instagram post →
- https://www.instagram.com/reel/Df2SL39CeHg/
- https://www.threads.com/@dataeng.ai/post/Df2SkWGpBot
- https://www.youtube.com/shorts/fcJ3i2K_0PI
- https://aimultiple.com/text-to-sql
Numbers above trace to these sources. If one moved, tell us and we fix it.


Talk it through
Argue with us on Instagram.