Token EconomicsIntermediate
LLM Rate Limits & Concurrency: Handling 429 Errors and Token Quotas
Direct Answer & Overview
LLM rate limits are traffic throttles imposed by AI API providers to protect GPU clusters from exhaustion, measured in Requests Per Minute (RPM), Requests Per Day (RPD), and Tokens Per Minute (TPM).
1.The Mechanics of HTTP 429 (Too Many Requests)
When an application exceeds RPM or TPM quotas, the inference server returns HTTP status 429. Without resilient gateway handling, downstream users experience application crashes. Best practices require exponential backoff with jitter and multi-provider failover routing.
2.Token Bucket Concurrency Algorithms
Providers utilize token bucket rate limiters. Tokens are replenished continuously. Sending a massive 128,000-token prompt can immediately drain a 200,000 TPM bucket, causing subsequent requests to be rejected for the remainder of the 60-second window.
Frequently Asked Questions
What is TPM vs RPM?
RPM (Requests Per Minute) limits the number of HTTP requests you can send in 60 seconds. TPM (Tokens Per Minute) limits the total token volume (input + output) processed in that window.
A100
API100 Engineering Team
Infrastructure & Latency Research