FAILURE MODES

One AI Fixed Our Conversion Rate. The Other Kept It 20% Low.

We asked two AI models to add a last-7-days column to our conversion rate KPI. One fixed the integer division bug in both columns. The other copied it into the new one, so the tile reads 3% instead of 3.75%.

2026-11-04

We asked two AI models to add one column to our conversion rate KPI: the same rate for just the last 7 days, so the tile can show the trend. The original query, SUM(conversions) * 100 / SUM(visits), is integer division on Postgres. With 45 conversions over 1,200 visits, the real rate is 3.75%. The tile reads 3%.

The 47-second reel: one AI fixed our conversion rate, the other kept it 20% low.

The bug

In Postgres, whole number divided by whole number drops the decimals before you ever see them. SUM(conversions) is a bigint and SUM(visits) is a bigint, so bigint divided by bigint is integer division. The .75 never survives. The tile showed 3% for a rate that is really 3.75%, twenty percent low, and nothing errored. The query ran fine and returned a wrong number.

Both windows in this synthetic data have the same rate, so the two columns match. What differs is the model, not the window:

All-time Last 7 days
Larger model 3.75% 3.75%
Smaller model 3% 3%

What the models did

We gave the query to two AI models from one provider, one try each, and asked for one thing only: add the same rate for just the last 7 days. The larger model fixed both columns and left a comment explaining the truncation. The smaller one copied the bug into the new column. Same request, opposite answers. Nobody asked for a fix, and the smaller model was not wrong about the ask. It copied the existing formula exactly, including the bug. The only difference is that the larger model noticed the formula itself was broken.

The two 7-day columns, verbatim from the test log:

-- Larger model, 7-day column (fixed):
-- Note: the original SUM(...) * 100 / SUM(...) is integer division and truncates
ROUND(SUM(conversions) FILTER (WHERE day > CURRENT_DATE - 7 AND day <= CURRENT_DATE)::numeric * 100
  / NULLIF(SUM(visits) FILTER (WHERE day > CURRENT_DATE - 7 AND day <= CURRENT_DATE), 0), 2)

-- Smaller model, 7-day column (copied the bug):
SUM(conversions) FILTER (WHERE day >= CURRENT_DATE - 6) * 100
  / NULLIF(SUM(visits) FILTER (WHERE day >= CURRENT_DATE - 6), 0)

The two models also wrote the 7-day window differently, today plus the 6 days before it in both cases, just spelled two ways. The window is not the bug. The missing cast is.

The fix

Cast to numeric before you divide. SUM(conversions)::numeric * 100 / NULLIF(SUM(visits), 0). The cast turns the division into numeric division, the decimals survive, and the NULLIF keeps a zero-visit window from erroring out. The larger model’s own answer had the guard, so the fix keeps it. The check that catches this before it ships is a test that compares the query output against a known rate, or a linter rule that flags integer division in a rate calculation. Eyeballing the SQL did not catch it. The original query passed review.

What we did not verify

This was our own test on synthetic data, run Nov 1, 2026, on PostgreSQL 16.15 and DuckDB 2.0. Two models from one provider, one try each. It is a small test, not a benchmark, and it says nothing about how often AI copies a bug it was not asked to copy. The voiceover in the reel is AI-generated. Watch the reel on Instagram and Threads.

This post started as an Instagram post →

Discussion

Talk it through

Argue with us on Instagram.