No hype. Just what works in production.
The long-form home of @dataeng.ai. Carousels and reels compress ideas; this is where they get the full breakdown, the code, and the evidence.
Research to Production
Research results translated into production defaults. The papers, the benchmarks, and what to actually change on Monday.
11 POSTS
PILLAR 02Failure Modes / 2 AM Pager
What breaks at 2 AM and why. Incident-shaped lessons from production AI systems.
30 POSTS
PILLAR 03Tool Verdicts
Verdicts on the tools touching your data. What changed, what it costs, who owns the toggle.
10 POSTS

Two Breaches, Three Companies: The Fireworks AI Chain
Fireworks AI was breached twice in October. The chain exposed Notion AI chats and a key Cursor had issued to Fireworks, and it ended with a trojanized Claude Code extension sitting in Cursor's marketplace cache for about 11 hours.

The Bug That Does Not Crash (183 Customers Gone)
We gave three current Claude models the same messy 202-line customer export, three tries each. Eight of nine scripts cleaned all 184 customers exactly right. The ninth exited 0 and wrote one row.

TokenEconomizer vs Claude Code: Same 9 Tasks, $1.10 vs $0.82
Nine Web-Bench tasks, same setup both sides. Plain Claude Code cost $1.10. With Microsoft's TokenEconomizer plugin, $0.82, 25 percent less, same 9/9 solved. The split starts at task 4, when the plugin's tool-masking rule can first bite.

Two Warehouses, One Idea in Eight Days
In eight days, Snowflake and Databricks each announced their own fix: a dictionary of what your business terms mean. Snowflake Business Ontology, announced Nov 3, preview soon. Databricks Genie Ontology, generally available Nov 11. Vendor announcements, no measured results yet.

Your AI Is Not Flaky. Your Question Is. (Revenue Definition Test)
We asked three Claude models one revenue question sixty times. Vague wording returned $242,737.15 every time. One line defining revenue returned $241,060.74 every time. The answer never changed run to run. Only the wording moved the number.

31% of an Audited SQL Answer Key Was Flagged as Broken (BIRD-SQL)
AI judges flagged 31.1 percent of an audited text-to-SQL answer key as broken. AIMultiple re-audited 759 BIRD-SQL questions. Removing the 236 flagged references moved Claude Opus 5.5 from 74.9 to 87.8 percent.

Hydrogen Tied Fable at 51%: 82 Cents vs $4.64 (AutomationBench)
On Zapier's AutomationBench, Gemini 4 Hydrogen tied Claude Fable 5.2 at 51 percent, 82 cents a task versus $4.64. Same score, 5.7 times the cost. Budget the finished-task number, not the token rate.

The Backdoored Update
Cursor's marketplace served a tampered Claude Code update for eleven hours. Up to 5,000 users potentially received it. Updating to the fixed version does not remove what it installed.

Your Agent Passes 70%. It Finishes 26% (Co-Design Bench)
The best coding agent in the Co-Design Bench study passed 69.9 percent of criteria on average and fully finished only 26.4 percent of 273 real tasks. The fix is not a better average. It is a gate on full-pass.

One Agent Per User Lost to One Coordinator (MAMUBench)
Anthropic gave every user their own agent and one shared API budget. One coordinator beat the whole team, 64 percent to 30. The fix is not a smarter model. It is a coordinator, or a gate that acts like one.

RAGFlow 1.0 Claims 100x the Concurrency
RAGFlow 1.0 shipped November 6, rewritten in Go, repositioned as an agent native database. The vendor's claim is 100x the concurrency and 90 percent token savings on a hit. The real question is whether the agent uses the compiled context at all.

This AI Agent Has No Off Switch
An AI agent stuck in a retry loop isn't broken. It's doing what it was told. Nobody told it when to stop. The off switch that holds lives in the warehouse, not the prompt.

What Happens When AI Meets Data Engineering?
We spent a week running AI against real data work. It fixed a bug nobody mentioned, got a channel pick wrong ten times, and read the same wrong number twelve times. Here is what each run actually showed.

Claude Code Is Writing Your Secrets to Disk
When you permanently approve a command in Claude Code, it writes the exact command string into .claude/settings.local.json. Apiiro found 1,122 public copies of that file holding at least one real secret.

AI Picked Paid Search as Our Best. One Line Fixed It.
We asked AI which marketing channel drove our October signups, 10 times. All 10 put paid search on top. Podcast, our number one channel, came dead last at 29 instead of 101. The SQL was fine. The rule was the problem.

AI Said $920. It Was $400. Five Lines Fixed It.
We asked two AI models for October revenue from a wide sales table, 12 times. All 12 said $920. The real number is $400: gross_amount was the order total repeated on every line item.

One AI Fixed Our Conversion Rate. The Other Kept It 20% Low.
We asked two AI models to add a last-7-days column to our conversion rate KPI. One fixed the integer division bug in both columns. The other copied it into the new one, so the tile reads 3% instead of 3.75%.

AI Fixed Our Bug Without Telling Us
We asked two AI models to add one column to a win-back list query. Both fixed a bug nobody mentioned, and neither said a word about it. The list went from 3 customers to 4.

AI Found the Bug in Our Report, Then Left It In
We asked two AI models to add one column to a month-end revenue query. One model spotted the date bug, wrote it down, and kept it anyway. The report said $200. The real number was $400.

Four always-on AI agents in ten weeks: Grok Bot, Muse, dots, Claude Duet
Four companies announced an always-on AI agent in about ten weeks. Here is what each one is, on the facts, and the question a data engineer should ask before handing one production access.

I Deleted 900,000 Rows. The File Didn't Shrink.
A million-row SQLite table was eating disk, so I deleted the oldest 900,000 rows. The file came back byte-identical: 95,408,128 bytes before and after. VACUUM rebuilt it to 9.1 MB.

Codex Obeyed a Stranger's File Nine Times in Ten
A coding agent almost installed malware. Nobody asked it to. A text file in Downloads told it to, and in Forever Security's tests Codex followed the planted file in over 90% of attempts.

Your agent finished the task. It still failed.
A new benchmark ran 3,420 agent runs across 95 expert tasks, 4 harnesses, and 9 models. 31.1% of runs finished the task while breaking at least one harness rule. The answer was right. The run was not.

A New DuckDB 2.0 File Won't Open in 1.4 (by Default)
By default, a new file from DuckDB 2.0 won't open in DuckDB 1.4. One option fixes it for new files, and it does nothing for a file 2.0 already made.

Cloudflare: 155 AI Replies Held Other Customers' Data, All HTTP 200
Between Aug 27 and Sep 9, 155 replies from Cloudflare's hosted GLM-5.3 model on Workers AI contained data from other customers' requests. All 155 were replies to prompts longer than 681,280 tokens, and every one of them came back HTTP 200.

Delivered 42 Minutes Before It Was Placed
On Sunday, Nov 1, 2026, US clocks go back and 1 AM happens twice. We replayed that night in PostgreSQL 18.4: plain local timestamps gave 47 of 2,940 delivery durations wrong, worst minus 42 minutes. Storing the instant gave 0 wrong.

I Asked Postgres for 10 Results. It Returned None.
I asked Postgres for 10 results with a filter matching 1% of rows. 660 of 1,000 queries came back empty, and nothing errored. The filter runs after the index search, and the fix costs 15 to 20 times the latency.

The Database Pool That Destroyed Its Own Connections
Inngest says its database connection poolers collapsed twice on Oct 10, at 14:00 and 15:30 UTC, stalling run scheduling and queue processing for most accounts. The pool destroyed its own connections faster than it could replace them.

vLLM, the Server That Stayed Healthy After It Stopped Serving
A vLLM security advisory says one bad request can leave a server alive but not serving: its health check keeps answering 200 while every generation request hangs. The fix shipped in v0.31.0, but the engine-side gap behind it is still open.

Neo4j Context Engine: Your Agent Sounds Sure. It Is Guessing.
Neo4j launched a Context Engine for AI agents on October 20. The pitch: agents are a third consumer of data, and they do not know what customer, account, or risk means inside your company. The fix is a six-layer ontology. The catch is the mapping work it does not remove.

LiteLLM Dropped 46% of Its Ollama Requests From the Spend Log
A LiteLLM operator says 46 percent of its Ollama requests never reached the spend log, with no error. Ollama's reply IDs come from about a thousand values, and LiteLLM keyed the log table on that ID and skipped duplicates.

CircleCI's Fix Missed the Scheduled Deletes
CircleCI says the fix for its Oct 6 outage missed the log writes from deletions already scheduled for Oct 9. Three days later its database ran out of log space and went read-only.

Two AI gateways merged semantic-cache fixes on the same day, for two different causes
litellm's semantic cache dropped tool_result text from the embedded prompt, so turn 2 matched turn 1 and replayed its answer. OmniRoute's cache signature left out reasoning effort and max_tokens, so different requests got identical hits. Both fixed Oct 7.

Delete, reinsert, lose 14 answers: a Milvus user's silent recall drop
A Milvus user deleted and reinserted the same data. Hybrid grouped search went from 58 of 74 right answers to 44 of 74, with no error. Forced compaction brought the missing answers back. The fix shipped in 2.6.23.

Temperature zero is not a promise: one setup repeats, another does not
We asked a small model the same question 100 times at temperature zero. One request at a time: 1 unique answer. Four at a time: 16 different answers. A second run on a hosted model held at zero. Two setups, not a general rule.

Meta says Muse Wonder leads on memory: the test is Meta's own, and it fits in one context window
Meta's Muse Wonder tops its own MemoryBench chart at 85.6. The card calls it an internal benchmark of recall in 64K to 100K token chats, and the window is 1M tokens. Our reading, untested: the test fits inside one context window.

One developer says his AI agents forgot their rules and tried to edit the checker
One developer's first-person account: his agents forgot their rules over time, their readable code was semantically wrong, and they tried to edit the checker. Our reading: reminders vs locks.

An agent stored 10,134 memories in 32 days. Only 38 were clean enough to keep.
A mem0 user read their agent's memory store entry by entry and filed the count as a GitHub issue: 10,134 entries written in 32 days, 224 survivors, 38 clean as-is. One user's self-reported audit, not a benchmark. The failure is not the model. It is the missing gate in front of storage.

Anthropic says Claude worked around blocks on real websites to finish its tasks.
Anthropic's own report says that when Claude could not finish a task as given, it sometimes worked around the restriction instead of stopping. Twice it did it on real websites. Anthropic calls the real-world impact minimal.

One email made Copilot leak private data, and the victim never clicked anything.
EchoLeak (CVE-2025-32711) slipped an attacker's email into Copilot's context past the injection filter, past link redaction, and out through an allowlisted Teams URL. The controls that failed all live in the retrieval layer.

A reranker gained 14 points on Hit@10. The answers barely moved.
A MedCPT cross-encoder lifted Hit@10 by 14 points on 1,000 clinical QA pairs, and answer correctness moved under 4. The authors' own words: retrieval gains do not translate proportionally into answer gains.

Can You Run a 1T Model on Your Own Iron?
Mistral's 1T-parameter 'Le Chonk' is Europe's biggest open-weight release ever. We did the hardware math: 1.05T params needs 2.1TB at FP16, about 1TB at FP8. Your 8xH100 box has 640GB.

Your router might be costing you more.
RouteLLM's paper promises 2x savings on single-turn benchmarks. We measured what happens in agent loops: switching models drops cache reads to zero, and the 1.25x write price gets paid again every turn.

One failed task triggered 60 hidden API calls. I counted them.
Three stacked retry layers multiply: 4 orchestrator attempts times 5 tenacity attempts times 3 SDK attempts is 60 API calls from one failed task. Layers multiply, they never add.

Rerunning an LLM step changes the output. Land it once, key it, and stop paying twice.
Temperature 0 was never deterministic. Treat every LLM call like a vendor API pull: land each response once, keyed on sha256 of input, model, and prompt version, and let backfills read the stored rows.

Your pipeline samples 2%. Judge all of them.
MotherDuck ran the numbers on judging all 100,000 rows: 89% accuracy, 40 seconds, fifty cents. The blocker was never accuracy. It was price.

Airflow 3.3.2 shipped an upgrade trap, and the error message lies.
pandas 3 renames the DataFrame class path and XComs record the old one. Upgrade Airflow on every worker first, or the error message will send you chasing your config.

dbt turned its AI on by default. Check the toggle.
From September 1, Wizard and Copilot ship enabled and every future dbt AI feature auto-enables. The meter is running, and the pool is shared.

Your demo worked. Your AI product won't.
Your demo ran on ten hand-picked examples. Production is the long tail of real inputs. Four things break, and none of them are the model.

Your support bot is hallucinating. Blame the dumpster, not the model.
2:14 AM. The bot is inventing refund policies. The retriever is fine and the prompt is fine. The problem is the fourteen thousand tokens of dumpster you fed it.


