FAILURE MODES

AI Fixed Our Bug Without Telling Us

We asked two AI models to add one column to a win-back list query. Both fixed a bug nobody mentioned, and neither said a word about it. The list went from 3 customers to 4.

2026-11-03

We asked two AI models to add one column to a win-back list query: each customer’s total October spend. Both models did the ask. Both also fixed a bug nobody mentioned, and neither said a word about it. The list went from 3 customers to 4.

The 45-second reel: AI fixed our bug without telling us.

The bug

Our win-back list was meant to list every customer with their October order count, including zero. The query filtered October in the WHERE clause. On a LEFT JOIN, that turns the join into an inner join: the customer with no October orders drops out of the result entirely. That customer was Dee, and Dee is the exact person a win-back email is for.

The old query returned 3 rows:

Customer October orders October spend
Ava 2 $180.00
Ben 2 $120.00
Cal 1 $200.00

Dee is missing. No error, no warning. The query ran fine and returned a wrong list.

What the models did

We gave the query to two AI models from one provider, one try each, and asked for one thing only: add each customer’s total October spend as a third column. Both models moved the date filter from the WHERE clause into the ON clause of the join. That is the right fix. The list came back with 4 customers, Dee included at zero orders and $0.00. Neither model left a comment or a note about the change.

The prompt stated the list’s intent, to include zero, and asked for a column only. Nobody asked for a fix.

The fixed query returned 4 rows:

Customer October orders October spend
Ava 2 $180.00
Ben 2 $120.00
Cal 1 $200.00
Dee 0 $0.00

The fix

A silent fix is still a change. The output changed shape, one row appeared, and nothing in the reply said so. If that query feeds a churn email, the extra row is the win. If it feeds a report, the extra row is a number that moves with no explanation. Either way, the check is the same: diff the output every time AI edits a query. Run the old query and the new one side by side and look at what changed, not just whether it ran.

What we did not verify

This was our own test on synthetic data, run Nov 1, 2026, on PostgreSQL 16.15 and DuckDB 2.0. Two models from one provider, one try each. It is a small test, not a benchmark, and it says nothing about how often AI fixes bugs it was not asked to fix. The voiceover in the reel is AI-generated. Watch the reel on Threads.

This post started as an Instagram post →

Discussion

Talk it through

Argue with us on Instagram.