Rate Limits and Quotas for Cloud AI APIs

Two meters run at once

Ordinary web APIs count requests. AI APIs count requests and tokens, and the second meter is usually the one that stops you.

A request limit is intuitive: so many calls in a window. A token limit is stranger, because the cost of a call depends on its content. One short question and one long document summary are both a single request, yet they consume wildly different amounts of the token budget.

Both meters typically cover input and output together. A conversation that carries a long history resends that history on every turn, so a chat application consumes its token allowance far faster than the request count suggests. That asymmetry is the root of most capacity surprises.

Underneath both sits the reason any of this exists: finite accelerator capacity shared among many customers. The same logic that produces compute ceilings, described in vCPU limits and service quotas, produces these.

How each platform expresses the limit

The vocabulary differs enough to cause confusion in a mixed environment.

Note the Azure row carefully. Elsewhere a quota is a ceiling you consume against. There it is an allowance you actively divide between deployments, so an over generous test deployment takes capacity away from production. The mechanics are covered in the Azure OpenAI Service overview.

The shared detail across all three is regional scoping. A quota approved in one region does nothing in another, and model availability varies by region as well, so capacity planning and region selection are the same exercise.

Reading what the service tells you

When you exceed an allowance the service responds with a throttling status rather than a server error. This distinction matters: the request was understood and refused for capacity reasons, so retrying the identical request later is expected behavior rather than a bug.

Many services include a header suggesting how long to wait. Honor it when present, because it reflects the service's own view of when capacity returns, and a client that ignores it and retries immediately makes congestion worse for everyone including itself.

Some services also publish remaining allowance in response headers, which lets a client see pressure building before it hits the wall. Where those headers exist, exporting them as metrics turns throttling from a surprise into a trend you can watch, in the spirit of monitoring cloud servers.

Client behavior that survives a limit

A well behaved client turns throttling into slightly slower service instead of a visible failure. Four practices do almost all of the work.

Exponential backoff with jitter. Wait longer after each successive refusal, and randomize the wait. Without randomization, every client that failed at the same moment retries at the same moment, producing a synchronized wave that guarantees another round of refusals.

A retry ceiling. Give up after a bounded number of attempts and surface a clear message. Unbounded retrying converts a short capacity dip into a queue that never drains.

Idempotency awareness. Know which operations are safe to repeat. Generation calls usually are; anything that writes a record as a side effect needs a deduplication key.

Client side pacing. Limit concurrency and rate in your own code rather than discovering the ceiling by hitting it. A token bucket in front of the SDK is a modest amount of code that prevents a whole class of incident.

Most official SDKs implement retry with backoff already. Check what your library does by default before writing your own, and check that it is not silently retrying in a way that multiplies your token consumption.

Buying predictability instead of hoping

Client discipline handles variability. It does not create capacity. When a workload genuinely needs more than shared throughput provides, there are three legitimate routes.

Request a quota increase. Every platform has a documented process through its support or quota console. Name the region, the model, the target figure and the workload. This is ordinary account administration and it takes time, so submit before a launch rather than during one.

Reserve throughput. Bedrock sells Provisioned Throughput, Azure sells provisioned throughput units, and Vertex AI offers Provisioned Throughput. You commit to a capacity level for a term and receive service that does not compete with shared demand. It costs more when idle and is the only way to make latency predictable under sustained load.

Use the batch path. All three platforms accept asynchronous jobs against submitted files, processed under separate and more generous allowances. Anything without an interactive deadline belongs here, and moving nightly work off the interactive path frees a surprising amount of headroom.

Designing an application that degrades gracefully

Architecture choices reduce demand more effectively than any retry policy.

Put a queue between users and the model for anything that can tolerate delay, with an explicit admission policy for what happens when it fills. Separate interactive traffic from background traffic so a bulk job cannot starve a live user, ideally through different credentials so the separation is enforced rather than assumed.

Cache aggressively. Identical requests recur more than teams expect, and a cache keyed on the full request removes that load entirely. Trimming context is the other half: sending a summarized conversation history instead of a verbatim one cuts token consumption directly, which helps both the quota and the bill discussed in the cost of AI workloads in the cloud.

Finally, define what the product does when capacity is unavailable. A queued response with an honest wait estimate, a smaller model as a fallback, or a plainly worded notice all beat a spinner that never resolves. Decide this deliberately, because the default is whatever the code happens to do.

Questions that come up during an incident