ArchitectureIntermediate
AI Inference Latency Explained: TTFT, Decoding Speed, and Network Overhead
Direct Answer & Overview
AI inference latency is the total round-trip time required to receive responses from a machine learning model, composed of network propagation, gateway proxy overhead, Time to First Token (prefill), and decoding token throughput.
1.The Four Stages of LLM Request Latency
Understanding where milliseconds are spent is critical for optimization:
1. Network Round-Trip Time (RTT): Time for packets to travel from client to gateway edge node (typically 10-35ms).
2. Gateway Authorization & Validation: Checking keys, rate limits, and credit wallets (API100: < 8ms).
3. Time to First Token (TTFT / Prefill): The GPU computes embeddings and self-attention for all input prompt tokens. This depends on prompt length and GPU memory bandwidth (190ms - 340ms).
4. Token Generation Velocity (Decoding): Autoregressive decoding rate (e.g. 70-140 tokens per second).
2.TTFT vs. Tokens Per Second (TPS)
A model with high tokens-per-second (e.g. 100 t/s) can still feel sluggish to users if its TTFT is 2.5 seconds. For human-facing chatbots and inline code autocomplete, minimizing TTFT is the single most important latency metric.
Frequently Asked Questions
What is TTFT in LLM benchmarking?
TTFT (Time to First Token) measures the duration from when a client completes sending an HTTP request until the first generated token arrives back at the client.
A100
API100 Engineering Team
Infrastructure & Latency Research