2 AM PAGER

One failed task triggered 60 hidden API calls. I counted them.

Three stacked retry layers multiply: 4 orchestrator attempts times 5 tenacity attempts times 3 SDK attempts is 60 API calls from one failed task. Layers multiply, they never add.

2026-10-05

Friday I estimated it. This weekend I counted it. One failing task, three stacked retry layers, and a stub server that answers 500 every time: exactly 60 HTTP requests. Twice.

This is the counted run that the Friday estimate promised. Same shape, real numbers.

The multiplier

60
HTTP requests from one failed task. 4 orchestrator attempts times 5 tenacity attempts times 3 SDK attempts. Counted twice on 2026-10-04 against a stub that always fails.

The math is the whole story: 4 x 5 x 3 = 60. Airflow-style retries=3 means 4 attempts. Tenacity configured with stop_after_attempt(5) means 5. The SDK default max_retries=2 means 3. Layers multiply. They never add.

That sentence is worth sitting with, because every pipeline you own has this shape somewhere. Nobody chose 60 retries. Three different people, at three different layers, each chose a reasonable number. The multiplication happened in the space between their decisions.

The setup, honestly labeled

Two of the three layers are the real libraries. Tenacity ran with stop_after_attempt(5), explicit config, because its default retries forever, which is its own horror story. The OpenAI SDK ran its true default, max_retries=2, on version 3.24.0, which is three attempts per call.

The first layer is modeled, not run. Full Airflow was not on the sandbox, so the orchestrator layer is a retry-semantics harness: retries=3 means 4 attempts, retry on any exception, 1s delay. I am labeling that plainly because the number is only as honest as its weakest layer, and you should be able to tell which layer that is.

The fix: kill the layer you didn’t pick

The SDK default retries are the layer nobody chose. Nobody sat down and decided the OpenAI client should retry twice; it just does. That makes it the first thing to kill. The orchestrator is the layer you can see, alert on, and reason about, so it stays.

The audit card

5
call page line. A task past 15 minutes or more than 5 API calls pages. Four attempts total stays under it. Sixty does not.

Run this on your own pipelines:

  1. List every retry layer: orchestrator, tenacity, SDK. Each one multiplies.
  2. Multiply attempts, not retries: retries plus one per layer.
  3. Set max_retries=0 on the SDK and stop_after_attempt(1) on tenacity. One retry layer only.
  4. Set timeout=30s per call so failures fail fast.
  5. Page past 15 minutes or more than 5 calls per task.

The question from the carousel is the right one to ask your team: how many retry layers are stacked in your pipeline? One, two, three, or no idea? “No idea” is the most common answer, and it is also the most expensive one.

This post started as an Instagram post →

Sources

Numbers above trace to these sources. If one moved, tell us and we fix it.

Discussion

Talk it through

Argue with us on Instagram.