LLM Rate Limits & Concurrency: Handling 429 Errors and Token Quotas
LLM rate limits are traffic throttles imposed by AI API providers to protect GPU clusters from exhaustion, measured in Requests Per Minute (RPM), Requests Per Day (RPD), and Tokens Per Minute (TPM).
Overview #
LLM rate limits are traffic throttles imposed by AI API providers to protect GPU clusters from exhaustion, measured in Requests Per Minute (RPM), Requests Per Day (RPD), and Tokens Per Minute (TPM).
The Mechanics of HTTP 429 (Too Many Requests) #
When an application exceeds RPM or TPM quotas, the inference server returns HTTP status 429. Without resilient gateway handling, downstream users experience application crashes. Best practices require exponential backoff with jitter and multi-provider failover routing.
Token Bucket Concurrency Algorithms #
Providers utilize token bucket rate limiters. Tokens are replenished continuously. Sending a massive 128,000-token prompt can immediately drain a 200,000 TPM bucket, causing subsequent requests to be rejected for the remainder of the 60-second window.
Frequently Asked Questions #
Q: What is TPM vs RPM?
RPM (Requests Per Minute) limits the number of HTTP requests you can send in 60 seconds. TPM (Tokens Per Minute) limits the total token volume (input + output) processed in that window.
Build with API100
Access 100+ AI models through one lightning-fast OpenAI-compatible API with sub-50ms routing overhead and zero markup on cached tokens.
