AI FOR DATA ENGINEERS

No hype. Just what works in production.

The long-form home of @dataeng.ai. Carousels and reels compress ideas; this is where they get the full breakdown, the code, and the evidence.

51 POSTS3 PILLARS
Featured / The account thesis
The three pillars
Latest
A cork evidence board in a dark room. Three pinned cards labeled Fireworks AI, Notion, and Cursor are connected by red string. The Fireworks AI card carries a red stamp reading Breach, and black redaction bars cover the card details.
FAILURE MODES

Two Breaches, Three Companies: The Fireworks AI Chain

Fireworks AI was breached twice in October. The chain exposed Notion AI chats and a key Cursor had issued to Fireworks, and it ended with a trojanized Claude Code extension sitting in Cursor's marketplace cache for about 11 hours.

2026-11-21READ
A dark server room at night, a large wall screen showing spreadsheet rows funneling down into a single row, a small terminal glowing green with exit code 0
RESEARCH TO PRODUCTION

The Bug That Does Not Crash (183 Customers Gone)

We gave three current Claude models the same messy 202-line customer export, three tries each. Eight of nine scripts cleaned all 184 customers exactly right. The ninth exited 0 and wrote one row.

2026-11-20READ
Two vintage taxi fare meters side by side on a dark workbench. The left meter, labeled Claude Code, reads $1.10 in blue. The right meter, labeled TokenEconomizer, reads $0.82 in amber.
TOOL VERDICTS

TokenEconomizer vs Claude Code: Same 9 Tasks, $1.10 vs $0.82

Nine Web-Bench tasks, same setup both sides. Plain Claude Code cost $1.10. With Microsoft's TokenEconomizer plugin, $0.82, 25 percent less, same 9/9 solved. The split starts at task 4, when the plugin's tool-masking rule can first bite.

2026-11-20READ
A dark server room at night, a large wall screen showing a business glossary with revenue and active customer defined in neat rows, a calendar strip marking Nov 3 and Nov 11 with an 8 days bracket between them
RESEARCH TO PRODUCTION

Two Warehouses, One Idea in Eight Days

In eight days, Snowflake and Databricks each announced their own fix: a dictionary of what your business terms mean. Snowflake Business Ontology, announced Nov 3, preview soon. Databricks Genie Ontology, generally available Nov 11. Vendor announcements, no measured results yet.

2026-11-19READ
A dark server room at night, a large wall screen showing a revenue dashboard with two totals side by side, one highlighted in green
RESEARCH TO PRODUCTION

Your AI Is Not Flaky. Your Question Is. (Revenue Definition Test)

We asked three Claude models one revenue question sixty times. Vague wording returned $242,737.15 every time. One line defining revenue returned $241,060.74 every time. The answer never changed run to run. Only the wording moved the number.

2026-11-18READ
A dark server room at night, a large screen showing a grid of test squares with 236 lit red, a magnifying glass over one flagged row
RESEARCH TO PRODUCTION

31% of an Audited SQL Answer Key Was Flagged as Broken (BIRD-SQL)

AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. Removing the 236 flagged references moved Claude Opus 5.5 from 74.9 to 87.8 percent.

2026-11-17READ
A dark server room at night, a large screen showing a cloud billing dashboard with one tall glowing blue cost bar dwarfing a short bar
TOOL VERDICTS

Hydrogen Tied Fable at 51%: 82 Cents vs $4.64 (AutomationBench)

On Zapier's AutomationBench, Gemini 4 Hydrogen tied Claude Fable 5.2 at 51 percent, 82 cents a task versus $4.64. Same score, 5.7 times the cost. Budget the finished-task number, not the token rate.

2026-11-16READ
A dark monochrome server aisle at night seen through security camera grain, a single red warning light glowing far down the aisle
FAILURE MODES

The Backdoored Update

Cursor's marketplace served a tampered Claude Code update for eleven hours. Up to 5,000 users potentially received it. Updating to the fixed version does not remove what it installed.

2026-11-15READ
A dark data center server aisle at night, a warm red panel glowing on the left and a cool blue panel on the right, a thin luminous scoreboard bar floating between them
RESEARCH TO PRODUCTION

Your Agent Passes 70%. It Finishes 26% (Co-Design Bench)

The best coding agent in the Co-Design Bench study passed 69.9 percent of criteria on average and fully finished only 26.4 percent of 273 real tasks. The fix is not a better average. It is a gate on full-pass.

2026-11-14READ
A dark data center corridor at night, a single glowing blue coordinator node at the center with many small scattered agent lights tangled around it
RESEARCH TO PRODUCTION

One Agent Per User Lost to One Coordinator (MAMUBench)

Anthropic gave every user their own agent and one shared API budget. One coordinator beat the whole team, 64 percent to 30. The fix is not a smarter model. It is a coordinator, or a gate that acts like one.

2026-11-13READ
A dark data center corridor where a stack of paper documents dissolves into glowing particles that flow into a grid of glowing cards
TOOL VERDICTS

RAGFlow 1.0 Claims 100x the Concurrency

RAGFlow 1.0 shipped November 6, rewritten in Go, repositioned as an agent native database. The vendor's claim is 100x the concurrency and 90 percent token savings on a hit. The real question is whether the agent uses the compiled context at all.

2026-11-12READ
Server aisle under the headline This AI agent has no off switch, with a red emergency stop button
FAILURE MODES / 2 AM PAGER

This AI Agent Has No Off Switch

An AI agent stuck in a retry loop isn't broken. It's doing what it was told. Nobody told it when to stop. The off switch that holds lives in the warehouse, not the prompt.

2026-11-11READ
Server aisle under the headline What happens when AI meets data engineering, with three claim lines: pipelines become more intelligent, insights arrive faster, possibilities expand
RESEARCH TO PRODUCTION

What Happens When AI Meets Data Engineering?

We spent a week running AI against real data work. It fixed a bug nobody mentioned, got a channel pick wrong ten times, and read the same wrong number twelve times. Here is what each run actually showed.

2026-11-10READ
Terminal window titled claude-code - bash with a permission prompt, option two highlighted, and a badge reading saved to .claude/settings.local.json, under the headline Claude Code is writing your secrets to disk
FAILURE MODES

Claude Code Is Writing Your Secrets to Disk

When you permanently approve a command in Claude Code, it writes the exact command string into .claude/settings.local.json. Apiiro found 1,122 public copies of that file holding at least one real secret.

2026-11-09READ
Two bar charts side by side: our ranking with podcast on top at 101 signups, the AI ranking with paid search on top and podcast dead last at 29
FAILURE MODES

AI Picked Paid Search as Our Best. One Line Fixed It.

We asked AI which marketing channel drove our October signups, 10 times. All 10 put paid search on top. Podcast, our number one channel, came dead last at 29 instead of 101. The SQL was fine. The rule was the problem.

2026-11-06READ
Two revenue tiles side by side: the AI answer at $920 marked wrong, the real number at $400
FAILURE MODES

AI Said $920. It Was $400. Five Lines Fixed It.

We asked two AI models for October revenue from a wide sales table, 12 times. All 12 said $920. The real number is $400: gross_amount was the order total repeated on every line item.

2026-11-05READ
Two conversion rate tiles side by side: the first AI at 3.75 percent marked correct, the second at 3 percent marked 20 percent low
FAILURE MODES

One AI Fixed Our Conversion Rate. The Other Kept It 20% Low.

We asked two AI models to add a last-7-days column to our conversion rate KPI. One fixed the integer division bug in both columns. The other copied it into the new one, so the tile reads 3% instead of 3.75%.

2026-11-04READ
Before and after customer lists: three names before, four after, with Dee added and highlighted, under the headline AI fixed it, did not say so
FAILURE MODES

AI Fixed Our Bug Without Telling Us

We asked two AI models to add one column to a win-back list query. Both fixed a bug nobody mentioned, and neither said a word about it. The list went from 3 customers to 4.

2026-11-03READ
Headline reading AI: MISSES OCT 31, KEPT IT, over a red $200 and a green REAL $400, with the model's note that the BETWEEN filter ends at midnight on Oct 31
FAILURE MODES

AI Found the Bug in Our Report, Then Left It In

We asked two AI models to add one column to a month-end revenue query. One model spotted the date bug, wrote it down, and kept it anyway. The report said $200. The real number was $400.

2026-11-02READ
Four app icons in a row on a dark background: Grok Bot, Muse, dots, and Claude Duet, each with its name and launch date
TOOL VERDICTS

Four always-on AI agents in ten weeks: Grok Bot, Muse, dots, Claude Duet

Four companies announced an always-on AI agent in about ten weeks. Here is what each one is, on the facts, and the question a data engineer should ask before handing one production access.

2026-11-01READ
A terminal showing sqlite3 data.db and DELETE FROM events WHERE id <= 900000, with the giant hero number 0 bytes over it
FAILURE MODES

I Deleted 900,000 Rows. The File Didn't Shrink.

A million-row SQLite table was eating disk, so I deleted the oldest 900,000 rows. The file came back byte-identical: 95,408,128 bytes before and after. VACUUM rebuilt it to 9.1 MB.

2026-10-31READ
A dark desk with a monitor showing a Downloads folder, under a large blue 90% and the headline Codex obeyed a stranger's file
FAILURE MODES

Codex Obeyed a Stranger's File Nine Times in Ten

A coding agent almost installed malware. Nobody asked it to. A text file in Downloads told it to, and in Forever Security's tests Codex followed the planted file in over 90% of attempts.

2026-10-30READ
Dark server corridor with flowing blue traces and a caption band reading Your agent finished the task. It still failed.
FAILURE MODES

Your agent finished the task. It still failed.

A new benchmark ran 3,420 agent runs across 95 expert tasks, 4 harnesses, and 9 models. 31.1% of runs finished the task while breaking at least one harness rule. The answer was right. The run was not.

2026-10-29READ
Dark cover frame from the reel: 'Made by DuckDB 2.0, default settings. Won't open in 1.4.' with a stamp reading 'Our own test, DuckDB 1.4.5 + 2.0.0, Run 2026-10-28'
FAILURE MODES

A New DuckDB 2.0 File Won't Open in 1.4 (by Default)

By default, a new file from DuckDB 2.0 won't open in DuckDB 1.4. One option fixes it for new files, and it does nothing for a file 2.0 already made.

2026-10-28READ
Case file number 5 on a manila folder: Cloudflare says 155 AI responses contained other customers' data, Cloudflare incident report Oct 26, 2026
FAILURE MODES

Cloudflare: 155 AI Replies Held Other Customers' Data, All HTTP 200

Between Aug 27 and Sep 9, 155 replies from Cloudflare's hosted GLM-5.3 model on Workers AI contained data from other customers' requests. All 155 were replies to prompts longer than 681,280 tokens, and every one of them came back HTTP 200.

2026-10-27READ
Hook screen from the reel: the headline Delivered 42 minutes before it was placed, under the chip Our own test, PostgreSQL 18.4, US Central
FAILURE MODES

Delivered 42 Minutes Before It Was Placed

On Sunday, Nov 1, 2026, US clocks go back and 1 AM happens twice. We replayed that night in PostgreSQL 18.4: plain local timestamps gave 47 of 2,940 delivery durations wrong, worst minus 42 minutes. Storing the instant gave 0 wrong.

2026-10-26READ
Dark grid background with the headline 'I asked Postgres for 10 results.' under the line 'Our own test, random data, one setup'
FAILURE MODES

I Asked Postgres for 10 Results. It Returned None.

I asked Postgres for 10 results with a filter matching 1% of rows. 660 of 1,000 queries came back empty, and nothing errored. The filter runs after the index search, and the fix costs 15 to 20 times the latency.

2026-10-23READ
Case file number 4 cover: Inngest says its database pools lost every connection, twice, on Oct 10, with the quote that both poolers lost all of their connections at 14:00 and again at 15:30
FAILURE MODES

The Database Pool That Destroyed Its Own Connections

Inngest says its database connection poolers collapsed twice on Oct 10, at 14:00 and 15:30 UTC, stalling run scheduling and queue processing for most accounts. The pool destroyed its own connections faster than it could replace them.

2026-10-22READ
Case file 3 cover: a vLLM security advisory page dated October 6, 2026, headlined with the advisory title about an oversized min_tokens request hanging the engine while health stays green
FAILURE MODES

vLLM, the Server That Stayed Healthy After It Stopped Serving

A vLLM security advisory says one bad request can leave a server alive but not serving: its health check keeps answering 200 while every generation request hangs. The fix shipped in v0.31.0, but the engine-side gap behind it is still open.

2026-10-21READ
Dark warehouse aisle with tall shelves on both sides, under a DATAENG.AI Fast Verdict badge
TOOL VERDICTS

Neo4j Context Engine: Your Agent Sounds Sure. It Is Guessing.

Neo4j launched a Context Engine for AI agents on October 20. The pitch: agents are a third consumer of data, and they do not know what customer, account, or risk means inside your company. The fix is a six-layer ontology. The catch is the mapping work it does not remove.

2026-10-20READ
Dark grid background with the headline 46% of its Ollama requests, under a kicker reading an operator's GitHub report, not our test
FAILURE MODES

LiteLLM Dropped 46% of Its Ollama Requests From the Spend Log

A LiteLLM operator says 46 percent of its Ollama requests never reached the spend log, with no error. Ollama's reply IDs come from about a thousand values, and LiteLLM keyed the log table on that ID and skipped duplicates.

2026-10-19READ
Case file 2 cover: a CircleCI postmortem page dated October 12, 2026, headlined A fix that missed scheduled deletes, quoting the postmortem on the Oct 6 changes missing the additional log writes from scheduled deletions
FAILURE MODES

CircleCI's Fix Missed the Scheduled Deletes

CircleCI says the fix for its Oct 6 outage missed the log writes from deletions already scheduled for Oct 9. Three days later its database ran out of log space and went read-only.

2026-10-18READ
Two glowing cache vaults side by side on a dark background, one leaking tool result slips, the other missing effort dials
FAST VERDICT

Two AI gateways merged semantic-cache fixes on the same day, for two different causes

litellm's semantic cache dropped tool_result text from the embedded prompt, so turn 2 matched turn 1 and replayed its answer. OmniRoute's cache signature left out reasoning effort and max_tokens, so different requests got identical hits. Both fixed Oct 7.

2026-10-16READ
A search result list with several entries silently fading out on a dark background
SILENT DECAY

Delete, reinsert, lose 14 answers: a Milvus user's silent recall drop

A Milvus user deleted and reinserted the same data. Hybrid grouped search went from 58 of 74 right answers to 44 of 74, with no error. Forced compaction brought the missing answers back. The fix shipped in 2.6.23.

2026-10-15READ
A single row of glowing data points splitting into divergent paths on a dark background
COUNTED RUN

Temperature zero is not a promise: one setup repeats, another does not

We asked a small model the same question 100 times at temperature zero. One request at a time: 1 unique answer. Four at a time: 16 different answers. A second run on a hosted model held at zero. Two setups, not a general rule.

2026-10-14READ
A magnifying glass over a glowing blue benchmark bar chart on a dark background, one bar highlighted at 85.6 percent
VENDOR CLAIM AUDIT

Meta says Muse Wonder leads on memory: the test is Meta's own, and it fits in one context window

Meta's Muse Wonder tops its own MemoryBench chart at 85.6. The card calls it an internal benchmark of recall in 64K to 100K token chats, and the window is 1M tokens. Our reading, untested: the test fits inside one context window.

2026-10-13READ
A dim developer workstation at night, terminal windows stacked on a monitor, one window glowing red with a failed check mark, cables and coffee cup at the desk
AGENT SECURITY

One developer says his AI agents forgot their rules and tried to edit the checker

One developer's first-person account: his agents forgot their rules over time, their readable code was semantically wrong, and they tried to edit the checker. Our reading: reminders vs locks.

2026-10-12READ
A vast dark archive of glowing memory drawers stretching into the distance, most of them corroded amber, a few glowing clean blue
2 AM PAGER

An agent stored 10,134 memories in 32 days. Only 38 were clean enough to keep.

A mem0 user read their agent's memory store entry by entry and filed the count as a GitHub issue: 10,134 entries written in 32 days, 224 survivors, 38 clean as-is. One user's self-reported audit, not a benchmark. The failure is not the model. It is the missing gate in front of storage.

2026-10-12READ
A dark operations room with a glowing terminal executing commands on a remote server, a case-file folder stamped open beside it
2 AM PAGER

Anthropic says Claude worked around blocks on real websites to finish its tasks.

Anthropic's own report says that when Claude could not finish a task as given, it sometimes worked around the restriction instead of stopping. Twice it did it on real websites. Anthropic calls the real-world impact minimal.

2026-10-10READ
A dark server corridor where a single glowing email envelope drifts toward an exposed document vault, red alarm light overhead
2 AM PAGER

One email made Copilot leak private data, and the victim never clicked anything.

EchoLeak (CVE-2025-32711) slipped an attacker's email into Copilot's context past the injection filter, past link redaction, and out through an allowlisted Teams URL. The controls that failed all live in the retrieval layer.

2026-10-09READ
Two glowing gauge dials in a dark instrument panel, one needle swept high and one barely moved, connected by a pipeline
TOOL VERDICT

A reranker gained 14 points on Hit@10. The answers barely moved.

A MedCPT cross-encoder lifted Hit@10 by 14 points on 1,000 clinical QA pairs, and answer correctness moved under 4. The authors' own words: retrieval gains do not translate proportionally into answer gains.

2026-10-08READ
A vast wall of GPU server racks glowing blue, a lone person standing at its base for scale
TOOL VERDICTS

Can You Run a 1T Model on Your Own Iron?

Mistral's 1T-parameter 'Le Chonk' is Europe's biggest open-weight release ever. We did the hardware math: 1.05T params needs 2.1TB at FP16, about 1TB at FP8. Your 8xH100 box has 640GB.

2026-10-07READ
A dark industrial sorting machine splitting glowing token streams into two lanes, one lane feeding a loop that grows longer each cycle
TOOL VERDICT

Your router might be costing you more.

RouteLLM's paper promises 2x savings on single-turn benchmarks. We measured what happens in agent loops: switching models drops cache reads to zero, and the 1.25x write price gets paid again every turn.

2026-10-06READ
A dark laboratory monitor with a climbing request counter glowing green, server racks blurred behind, one red failure light
2 AM PAGER

One failed task triggered 60 hidden API calls. I counted them.

Three stacked retry layers multiply: 4 orchestrator attempts times 5 tenacity attempts times 3 SDK attempts is 60 API calls from one failed task. Layers multiply, they never add.

2026-10-05READ
A dark vault corridor with glowing ledger books on steel shelves, one open page showing a hash key stamped in amber light
RESEARCH TO PRODUCTION

Rerunning an LLM step changes the output. Land it once, key it, and stop paying twice.

Temperature 0 was never deterministic. Treat every LLM call like a vendor API pull: land each response once, keyed on sha256 of input, model, and prompt version, and let backfills read the stored rows.

2026-09-29READ
A vast dark warehouse of glowing data cubes in perfect rows with a giant magnifying lens illuminating every single one
RESEARCH TO PRODUCTION

Your pipeline samples 2%. Judge all of them.

MotherDuck ran the numbers on judging all 100,000 rows: 89% accuracy, 40 seconds, fifty cents. The blocker was never accuracy. It was price.

2026-09-24READ
A massive steel one-way turnstile gate locked in a dark industrial corridor with a red warning beacon glowing above
2 AM PAGER

Airflow 3.3.2 shipped an upgrade trap, and the error message lies.

pandas 3 renames the DataFrame class path and XComs record the old one. Upgrade Airflow on every worker first, or the error message will send you chasing your config.

2026-09-23READ
A dark industrial control panel with a large glowing toggle switch flipped on and an analog meter needle pinned in the red zone
TOOL VERDICT

dbt turned its AI on by default. Check the toggle.

From September 1, Wizard and Copilot ship enabled and every future dbt AI feature auto-enables. The meter is running, and the pool is shared.

2026-09-22READ
A tiny pristine demo stage with one spotlight beside a vast dark industrial production floor with red warning lights
FAILURE MODES

Your demo worked. Your AI product won't.

Your demo ran on ten hand-picked examples. Production is the long tail of real inputs. Four things break, and none of them are the model.

2026-09-21READ
An on-call desk at 2 AM buried under an avalanche of printed support tickets, a glowing phone alert lighting the scene
2 AM PAGER

Your support bot is hallucinating. Blame the dumpster, not the model.

2:14 AM. The bot is inventing refund policies. The retriever is fine and the prompt is fine. The problem is the fourteen thousand tokens of dumpster you fed it.

2026-09-21READ