A 429 means you exceeded a limit. What is less obvious is that there are several limits and you can hit any of them.
Three kinds of limit
Requests per minute. How often you may call. Usually hit by people parallelising small tasks.
Input tokens per minute. How much text you send. Hit by anyone pushing large context: agents, repository work.
Output tokens per minute. How much the model generates. Hit by long answers or many concurrent generations.
The first is rarely the binding one. On agentic workloads you hit the second long before it.
Backing off correctly
A naive immediate retry makes things worse: you add load exactly when there is already too much. You want growing delay plus a random component.
import random, time
def call_with_retry(fn, attempts=5, base=1.0):
for i in range(attempts):
try:
return fn()
except RateLimitError:
if i == attempts - 1:
raise
delay = base * (2 ** i) + random.uniform(0, 0.5)
time.sleep(delay)
The random part stops ten of your workers retrying in lockstep. Without it you get waves of failures instead of steady throughput.
Read the response headers
The API tells you how long until reset. That beats guessing: if the server says wait twenty seconds, wait twenty, not the thirty-two your formula produced.
Not hitting limits at all
Do not send a whole file when you need a method. The input limit is spent on context, and context compounds on agentic runs.
Split load across keys. Background processing and live human work should not share a limit, or a nightly run will ruin someone's morning.
Queue instead of parallelising. Ten concurrent requests hit the limit; ten sequential ones do not. If the work is not urgent, a queue is cheaper and steadier.
Cache repeated context. If every request drags the same large instruction along, cache it provider-side.
On separate keys
Splitting load is easiest with keys: one for production, one for background, one for experiments. Then you can see who consumed the limit and bound each independently.
Here every key carries its own limits and usage stats, and sub-organisations separate teams onto their own budgets.
In short
There are three limits and on agents you hit the token one, not the request one. Back off with growing delay and jitter, read the headers, and keep background work on a different key from live work.