Streaming AI Responses with Server-Sent Events (SSE) & HTTP/2

Overview #

Streaming AI responses is an architecture where an LLM server pushes completion tokens to the client in real time via Server-Sent Events (SSE), eliminating the need to wait for the entire generation to finish.

How Server-Sent Events Relieve Latency Bottlenecks #

Without streaming, generating a 1,000-token completion takes 10 to 15 seconds, during which the user stares at a loading spinner. With Server-Sent Events (SSE) over an HTTP/2 connection, the gateway immediately flushes the first token within 250ms, delivering a smooth typewriter effect that matches human reading speed.

The OpenAI SSE Stream Protocol #

Chunks arrive formatted as text/event-stream:
`data: {"choices": [{"delta": {"content": "Hello"}}]}

data: {"choices": [{"delta": {"content": " world"}}]}

data: [DONE]`

API100 implements zero-copy stream relay, ensuring that chunks are forwarded to client sockets without buffer accumulation.

Code Example: Streaming Tokens in Node.js / TypeScript #

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.apihundred.com/v1",
  apiKey: process.env.API100_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "claude-3-5-sonnet",
  messages: [{ role: "user", content: "Explain streaming architecture." }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}

Frequently Asked Questions #

Q: Why use Server-Sent Events instead of WebSockets for AI streaming?

SSE is built on standard HTTP/2, requires no special protocol handshake, natively handles proxy reconnection, and works seamlessly with edge caching and CDN firewalls.