Your AI is not flaky. Your question is. We asked three current Claude models, Haiku 5.5, Sonnet 5.5 and Opus 5.5, one revenue question sixty times: 10 runs per model per wording, every query executed against synthetic store data. The answer never changed run to run. Only the wording moved the number.
The 29-second reel: same question, same models, 60 runs, one number each way.
$242,737.15 vs $241,060.74
With a vague prompt, all 30 runs summed order totals, shipping and discounts included, and returned $242,737.15. With a one-line definition of revenue, sum of quantity times unit price from order items, on completed orders, excluding test customers, for October 2026, all 30 returned $241,060.74. The gap is $1,676.41, about 1,700 dollars, or 0.70 percent of defined revenue.
The vague answer was not wrong
The vague answer is a reasonable reading of revenue, just not the one we defined. Both readings are defensible. Only the defined one matched our spec. All three models behaved identically, so there is no ranking here. Claude models only.
The takeaway
Before you blame the model, write down what your metric means. A definition is one line. Without it, every run is a coin flip over which reading the model picks, and the model will pick the same one every time, which looks like determinism and hides the ambiguity. Enforce the definition in the prompt, in a metrics layer like dbt or Cube, or in a review step before the SQL runs, but enforce it somewhere.
Caveats
Our own test, run Nov 16, 2026. Synthetic data, 781 completed non-test October orders. No sampling controls were set; the runs went through the Claude Code harness, so this is determinism as shipped, not a temperature-zero guarantee. The vague wording and the defined wording were the only two wordings tested.
Sources
- Instagram reel: https://www.instagram.com/reel/Df5qqDED6jh/
- Threads post: https://www.threads.com/@dataeng.ai/post/Df5sKw6z4ZU
- YouTube Short: https://www.youtube.com/shorts/eVkQdwsHHuM
This post started as an Instagram post →
- https://www.instagram.com/reel/Df5qqDED6jh/
- https://www.threads.com/@dataeng.ai/post/Df5sKw6z4ZU
- https://www.youtube.com/shorts/eVkQdwsHHuM
Numbers above trace to these sources. If one moved, tell us and we fix it.


Talk it through
Argue with us on Instagram.