Research to Production
Research results translated into production defaults. The papers, the benchmarks, and what to actually change on Monday.

The Bug That Does Not Crash (183 Customers Gone)
We gave three current Claude models the same messy 202-line customer export, three tries each. Eight of nine scripts cleaned all 184 customers exactly right. The ninth exited 0 and wrote one row.

Two Warehouses, One Idea in Eight Days
In eight days, Snowflake and Databricks each announced their own fix: a dictionary of what your business terms mean. Snowflake Business Ontology, announced Nov 3, preview soon. Databricks Genie Ontology, generally available Nov 11. Vendor announcements, no measured results yet.

Your AI Is Not Flaky. Your Question Is. (Revenue Definition Test)
We asked three Claude models one revenue question sixty times. Vague wording returned $242,737.15 every time. One line defining revenue returned $241,060.74 every time. The answer never changed run to run. Only the wording moved the number.

31% of an Audited SQL Answer Key Was Flagged as Broken (BIRD-SQL)
AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. Removing the 236 flagged references moved Claude Opus 5.5 from 74.9 to 87.8 percent.

Your Agent Passes 70%. It Finishes 26% (Co-Design Bench)
The best coding agent in the Co-Design Bench study passed 69.9 percent of criteria on average and fully finished only 26.4 percent of 273 real tasks. The fix is not a better average. It is a gate on full-pass.

One Agent Per User Lost to One Coordinator (MAMUBench)
Anthropic gave every user their own agent and one shared API budget. One coordinator beat the whole team, 64 percent to 30. The fix is not a smarter model. It is a coordinator, or a gate that acts like one.

What Happens When AI Meets Data Engineering?
We spent a week running AI against real data work. It fixed a bug nobody mentioned, got a channel pick wrong ten times, and read the same wrong number twelve times. Here is what each run actually showed.

Temperature zero is not a promise: one setup repeats, another does not
We asked a small model the same question 100 times at temperature zero. One request at a time: 1 unique answer. Four at a time: 16 different answers. A second run on a hosted model held at zero. Two setups, not a general rule.

Rerunning an LLM step changes the output. Land it once, key it, and stop paying twice.
Temperature 0 was never deterministic. Treat every LLM call like a vendor API pull: land each response once, keyed on sha256 of input, model, and prompt version, and let backfills read the stored rows.

Your pipeline samples 2%. Judge all of them.
MotherDuck ran the numbers on judging all 100,000 rows: 89% accuracy, 40 seconds, fifty cents. The blocker was never accuracy. It was price.

RAG is just a data pipeline
Ingest, chunk, embed, index, retrieve, generate. The model is the last box in the diagram, and everything before it is your world.
