ArchitectureIntermediate
Streaming AI Responses with Server-Sent Events (SSE) & HTTP/2
Direct Answer & Overview
Streaming AI responses is an architecture where an LLM server pushes completion tokens to the client in real time via Server-Sent Events (SSE), eliminating the need to wait for the entire generation to finish.
1.How Server-Sent Events Relieve Latency Bottlenecks
Without streaming, generating a 1,000-token completion takes 10 to 15 seconds, during which the user stares at a loading spinner. With Server-Sent Events (SSE) over an HTTP/2 connection, the gateway immediately flushes the first token within 250ms, delivering a smooth typewriter effect that matches human reading speed.
2.The OpenAI SSE Stream Protocol
Chunks arrive formatted as text/event-stream:
`data: {"choices": [{"delta": {"content": "Hello"}}]}
data: {"choices": [{"delta": {"content": " world"}}]}
data: [DONE]`
API100 implements zero-copy stream relay, ensuring that chunks are forwarded to client sockets without buffer accumulation.
Streaming Tokens in Node.js / TypeScripttypescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.apihundred.com/v1",
apiKey: process.env.API100_API_KEY,
});
const stream = await client.chat.completions.create({
model: "claude-3-5-sonnet",
messages: [{ role: "user", content: "Explain streaming architecture." }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}Frequently Asked Questions
Why use Server-Sent Events instead of WebSockets for AI streaming?
SSE is built on standard HTTP/2, requires no special protocol handshake, natively handles proxy reconnection, and works seamlessly with edge caching and CDN firewalls.
A100
API100 Engineering Team
Infrastructure & Latency Research