Pillar

Research to Production

Research results translated into production defaults. The papers, the benchmarks, and what to actually change on Monday.

11 POSTS
A dark server room at night, a large wall screen showing spreadsheet rows funneling down into a single row, a small terminal glowing green with exit code 0
RESEARCH TO PRODUCTION

The Bug That Does Not Crash (183 Customers Gone)

We gave three current Claude models the same messy 202-line customer export, three tries each. Eight of nine scripts cleaned all 184 customers exactly right. The ninth exited 0 and wrote one row.

2026-11-20READ
A dark server room at night, a large wall screen showing a business glossary with revenue and active customer defined in neat rows, a calendar strip marking Nov 3 and Nov 11 with an 8 days bracket between them
RESEARCH TO PRODUCTION

Two Warehouses, One Idea in Eight Days

In eight days, Snowflake and Databricks each announced their own fix: a dictionary of what your business terms mean. Snowflake Business Ontology, announced Nov 3, preview soon. Databricks Genie Ontology, generally available Nov 11. Vendor announcements, no measured results yet.

2026-11-19READ
A dark server room at night, a large wall screen showing a revenue dashboard with two totals side by side, one highlighted in green
RESEARCH TO PRODUCTION

Your AI Is Not Flaky. Your Question Is. (Revenue Definition Test)

We asked three Claude models one revenue question sixty times. Vague wording returned $242,737.15 every time. One line defining revenue returned $241,060.74 every time. The answer never changed run to run. Only the wording moved the number.

2026-11-18READ
A dark server room at night, a large screen showing a grid of test squares with 236 lit red, a magnifying glass over one flagged row
RESEARCH TO PRODUCTION

31% of an Audited SQL Answer Key Was Flagged as Broken (BIRD-SQL)

AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. Removing the 236 flagged references moved Claude Opus 5.5 from 74.9 to 87.8 percent.

2026-11-17READ
A dark data center server aisle at night, a warm red panel glowing on the left and a cool blue panel on the right, a thin luminous scoreboard bar floating between them
RESEARCH TO PRODUCTION

Your Agent Passes 70%. It Finishes 26% (Co-Design Bench)

The best coding agent in the Co-Design Bench study passed 69.9 percent of criteria on average and fully finished only 26.4 percent of 273 real tasks. The fix is not a better average. It is a gate on full-pass.

2026-11-14READ
A dark data center corridor at night, a single glowing blue coordinator node at the center with many small scattered agent lights tangled around it
RESEARCH TO PRODUCTION

One Agent Per User Lost to One Coordinator (MAMUBench)

Anthropic gave every user their own agent and one shared API budget. One coordinator beat the whole team, 64 percent to 30. The fix is not a smarter model. It is a coordinator, or a gate that acts like one.

2026-11-13READ
Server aisle under the headline What happens when AI meets data engineering, with three claim lines: pipelines become more intelligent, insights arrive faster, possibilities expand
RESEARCH TO PRODUCTION

What Happens When AI Meets Data Engineering?

We spent a week running AI against real data work. It fixed a bug nobody mentioned, got a channel pick wrong ten times, and read the same wrong number twelve times. Here is what each run actually showed.

2026-11-10READ
A single row of glowing data points splitting into divergent paths on a dark background
COUNTED RUN

Temperature zero is not a promise: one setup repeats, another does not

We asked a small model the same question 100 times at temperature zero. One request at a time: 1 unique answer. Four at a time: 16 different answers. A second run on a hosted model held at zero. Two setups, not a general rule.

2026-10-14READ
A dark vault corridor with glowing ledger books on steel shelves, one open page showing a hash key stamped in amber light
RESEARCH TO PRODUCTION

Rerunning an LLM step changes the output. Land it once, key it, and stop paying twice.

Temperature 0 was never deterministic. Treat every LLM call like a vendor API pull: land each response once, keyed on sha256 of input, model, and prompt version, and let backfills read the stored rows.

2026-09-29READ
A vast dark warehouse of glowing data cubes in perfect rows with a giant magnifying lens illuminating every single one
RESEARCH TO PRODUCTION

Your pipeline samples 2%. Judge all of them.

MotherDuck ran the numbers on judging all 100,000 rows: 89% accuracy, 40 seconds, fifty cents. The blocker was never accuracy. It was price.

2026-09-24READ
Industrial refinery pipeline at night feeding paper documents in and emitting glowing vector streams
THE ACCOUNT THESIS

RAG is just a data pipeline

Ingest, chunk, embed, index, retrieve, generate. The model is the last box in the diagram, and everything before it is your world.

2026-09-20READ