AI Inference Latency Explained: TTFT, Decoding Speed, and Network Overhead
AI inference latency is the total round-trip time required to receive responses from a machine learning model, composed of network propagation, gateway proxy overhead, Time to First Token (prefill), and decoding token throughput.
Overview #
AI inference latency is the total round-trip time required to receive responses from a machine learning model, composed of network propagation, gateway proxy overhead, Time to First Token (prefill), and decoding token throughput.
The Four Stages of LLM Request Latency #
Understanding where milliseconds are spent is critical for optimization:
- Network Round-Trip Time (RTT): Time for packets to travel from client to gateway edge node (typically 10-35ms).
- Gateway Authorization & Validation: Checking keys, rate limits, and credit wallets (API100: < 8ms).
- Time to First Token (TTFT / Prefill): The GPU computes embeddings and self-attention for all input prompt tokens. This depends on prompt length and GPU memory bandwidth (190ms - 340ms).
- Token Generation Velocity (Decoding): Autoregressive decoding rate (e.g. 70-140 tokens per second).
TTFT vs. Tokens Per Second (TPS) #
A model with high tokens-per-second (e.g. 100 t/s) can still feel sluggish to users if its TTFT is 2.5 seconds. For human-facing chatbots and inline code autocomplete, minimizing TTFT is the single most important latency metric.
Frequently Asked Questions #
Q: What is TTFT in LLM benchmarking?
TTFT (Time to First Token) measures the duration from when a client completes sending an HTTP request until the first generated token arrives back at the client.
Build with API100
Access 100+ AI models through one lightning-fast OpenAI-compatible API with sub-50ms routing overhead and zero markup on cached tokens.
