FAILURE MODES

AI Found the Bug in Our Report, Then Left It In

We asked two AI models to add one column to a month-end revenue query. One model spotted the date bug, wrote it down, and kept it anyway. The report said $200. The real number was $400.

2026-11-02

We asked two AI models to add one column to a month-end revenue query. The query used BETWEEN ‘2026-10-01’ AND ‘2026-10-31’ on a timestamp, which stops at midnight on Oct 31. One model wrote a comment saying exactly that, then kept the filter “so the existing report numbers still match”. The other kept it without a word. Result: $200 reported, $400 real. One order at 6:22 PM on October 31st, gone, no error.

The 52-second reel: AI found the bug in our report. Then it left it in.

The bug

BETWEEN on a timestamp stops at midnight. The upper bound ‘2026-10-31’ means midnight at the start of Oct 31, so every order placed during Oct 31 falls outside the range. Order 104, placed at 2026-10-31 18:22:00 for $200.00, dropped out of the report. No error, no warning. The query ran fine and returned a wrong number.

Report Orders Revenue
Buggy 2 $200.00
Fixed 3 $400.00

Half of October’s revenue was missing, and the report gave no sign anything was wrong.

What the models did

We gave the query to two AI models from one provider, one try each, and asked for one thing only: add one column to the October revenue query. The larger model wrote a comment in its answer noting that the existing BETWEEN filter ends at ‘2026-10-31 00:00:00’, so orders placed after midnight on Oct 31 are excluded. Then it left the filter unchanged, so the existing report numbers still match. The second model kept the filter too, without a word.

The ask was a column, not a fix. Neither model was told to check the date filter. One of them checked it anyway, wrote down what it found, and still shipped the bug.

Asked to write the same report from scratch, both models got it right.

The fix

Use a half-open range: on or after October 1 and before November 1.

order_ts >= '2026-10-01' AND order_ts < '2026-11-01'

The fixed query returns 3 orders and $400.00, with order 104 counted. A date written as ‘2026-10-31’ is midnight, not the end of the day. If your month-end report filters a timestamp with BETWEEN, check what the last day actually covers.

Send this to whoever runs your month-end report this week.

What we did not verify

This was our own test on synthetic data, run Nov 1, 2026, on PostgreSQL 16.15 and DuckDB 2.0. Two models from one provider, one try each. It is a small test, not a benchmark, and it says nothing about how often AI keeps a bug it spotted, or about other models or providers. The voiceover in the reel is AI-generated. Watch the reel on Threads or YouTube.

This post started as an Instagram post →

Discussion

Talk it through

Argue with us on Instagram.