RESEARCH TO PRODUCTION

31% of an Audited SQL Answer Key Was Flagged as Broken (BIRD-SQL)

AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. Removing the 236 flagged references moved Claude Opus 5.5 from 74.9 to 87.8 percent.

2026-11-17

AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. A panel of five models flagged 236 of the reference answers as broken, 31.1 percent of the set. With those removed, Claude Opus 5.5 went from 74.9 to 87.8 percent, and Gemini 4 Argon went from 64.3 to 78.6 percent.

The 35-second reel: 236 of 759 reference answers flagged, and both featured scores moved.

759 questions, 236 flagged

BIRD-SQL is a text-to-SQL benchmark, and its answer key is the ground truth every model is scored against. AIMultiple ran five models as judges over 759 of its questions. Three of five had to agree before a reference was flagged. That bar caught 236 references, nearly a third of the audited set. The other 523 stayed unflagged.

74.9 to 87.8

The flagged answers were not noise. Excluding them moved Claude Opus 5.5 from 74.9 to 87.8 percent, a 12.9 point jump. Gemini 4 Argon moved from 64.3 to 78.6 percent, a 14.3 point jump. Both scores moved in the same direction by double digits, which is what you expect when the answer key itself is carrying errors.

The takeaway

Check the answer key first. A benchmark score is only as honest as its ground truth, and a broken reference punishes every model that gets the question right. Before you trust a leaderboard gap, ask whether the key has been audited, and by whom.

Caveats

Five models did the auditing, and three of five had to agree before a reference was flagged. Independent human annotators have not repeated the review, so the 236 count is a model judgment, not a confirmed error count. The two models also ran under different setups, Claude Code versus OpenRouter, so their score gap is not a clean model-to-model comparison. Both featured scores moved once the 236 flagged answers were excluded.

Sources

This post started as an Instagram post →

Discussion

Talk it through

Argue with us on Instagram.