Empirical Benchmark Methodology
How API100 measures Time to First Token (TTFT), streaming decoding throughput, and gateway proxy overhead with statistical rigor and zero payload distortion.
Testing Framework Parameters
Core Metrics & Definitions
1. Time to First Token (TTFT)
Defined as the precise interval (in milliseconds) between the moment the HTTP client dispatches the final byte of the JSON request payload and the moment the client receives the first Server-Sent Event (SSE) byte containing a non-empty token delta from the upstream inference engine.
2. Generation Throughput (Tokens per Second)
The decoding velocity of the model once generation commences. Calculated by dividing the total count of completion tokens by the duration between the first token delta and the final [DONE] stream termination event.
3. Gateway Routing Overhead
The additional latency introduced by the API100 proxy layer for token authorization, rate limiting, and credit validation. Measured by comparing a direct TLS handshake with the upstream provider against the API100 edge routing path.
Statistical Normalization
To eliminate outliers caused by upstream cold starts, 10% warm-up requests are executed before initiating test suites. Prompts of varying lengths (100 tokens, 1,000 tokens, and 8,000 tokens) are evaluated to assess how prompt token length impacts prompt evaluation prefill latency.