The worst bug is the one that doesn’t crash. We gave three current Claude models (Haiku 5.5, Sonnet 5.5, and Opus 5.5) the same messy 202-line customer export, three tries each. Eight of nine scripts cleaned all 184 customers exactly right. The ninth exited 0 and wrote one row.
The 29-second reel: nine scripts, one silent failure, 183 customers gone.
The export
The export had 184 customers plus 10 exact duplicates, 6 near duplicates, a repeated header, mixed date formats, dollar amounts like $1,204.50 and (45.00), quoted names like “García, Élodie”, and accented names. The file holds 200 customer rows (184 unique plus 16 duplicates), 1 top header, and 1 repeated mid-file header, 202 lines total. It is the kind of file a real pipeline sees every week.
The failure
The failing script stored a helper function per row that read the loop variable after the loop ended, so every row became a copy of the last one. Python late binding, not a crash. The script exited 0, reported success, and delivered one row out of 184. Nothing in the run looked wrong. The output file existed, the exit code was clean, and 183 customers were gone.
The takeaway
Keep a few hand-checked rows and diff every run against them. In production: assert row counts before and after dedupe, and never trust exit 0 without a contract check. A script that fails loudly is a gift. The one that exits 0 and writes one row is the one that costs you a customer file.
Caveats
Our own benchmark run Nov 16, 2026 across Claude models only (three runs per model, small sample, not a ranking). A row passed only if all six cells matched the hand-checked file. The ninth script was one Opus 5.5 run out of three; its other two runs passed 184 of 184. This is a failure mode we observed, not a model ranking.
Sources
- Instagram reel: https://www.instagram.com/reel/Df_DrokCUiz/
- Threads post: https://www.threads.com/@dataeng.ai/post/Df_DcXSpQc7
- YouTube Short: https://www.youtube.com/shorts/RDOE48-pYYw
This post started as an Instagram post →
- https://www.instagram.com/reel/Df_DrokCUiz/
- https://www.threads.com/@dataeng.ai/post/Df_DcXSpQc7
- https://www.youtube.com/shorts/RDOE48-pYYw
Numbers above trace to these sources. If one moved, tell us and we fix it.


Talk it through
Argue with us on Instagram.