Friday I estimated it. This weekend I counted it. One failing task, three stacked retry layers, and a stub server that answers 500 every time: exactly 60 HTTP requests. Twice.
This is the counted run that the Friday estimate promised. Same shape, real numbers.
The multiplier
The math is the whole story: 4 x 5 x 3 = 60. Airflow-style retries=3 means 4 attempts. Tenacity configured with stop_after_attempt(5) means 5. The SDK default max_retries=2 means 3. Layers multiply. They never add.
That sentence is worth sitting with, because every pipeline you own has this shape somewhere. Nobody chose 60 retries. Three different people, at three different layers, each chose a reasonable number. The multiplication happened in the space between their decisions.
The setup, honestly labeled
Two of the three layers are the real libraries. Tenacity ran with stop_after_attempt(5), explicit config, because its default retries forever, which is its own horror story. The OpenAI SDK ran its true default, max_retries=2, on version 3.24.0, which is three attempts per call.
The first layer is modeled, not run. Full Airflow was not on the sandbox, so the orchestrator layer is a retry-semantics harness: retries=3 means 4 attempts, retry on any exception, 1s delay. I am labeling that plainly because the number is only as honest as its weakest layer, and you should be able to tell which layer that is.
The fix: kill the layer you didn’t pick
The SDK default retries are the layer nobody chose. Nobody sat down and decided the OpenAI client should retry twice; it just does. That makes it the first thing to kill. The orchestrator is the layer you can see, alert on, and reason about, so it stays.
The audit card
Run this on your own pipelines:
- List every retry layer: orchestrator, tenacity, SDK. Each one multiplies.
- Multiply attempts, not retries: retries plus one per layer.
- Set max_retries=0 on the SDK and stop_after_attempt(1) on tenacity. One retry layer only.
- Set timeout=30s per call so failures fail fast.
- Page past 15 minutes or more than 5 calls per task.
The question from the carousel is the right one to ask your team: how many retry layers are stacked in your pipeline? One, two, three, or no idea? “No idea” is the most common answer, and it is also the most expensive one.
This post started as an Instagram post →
Numbers above trace to these sources. If one moved, tell us and we fix it.


Talk it through
Argue with us on Instagram.