Speech-to-Speech & WebRTC Real-Time Voice Agents: Building Voice AI in 2026

Overview #

Speech-to-Speech APIs bypass text conversion bottlenecks by processing audio tokens directly through native multimodal transformers, achieving natural sub-300ms conversational turn-taking, barge-in interruption handling, and human vocal realism.

The Cascaded Pipeline vs. Native S2S Architecture #

In a cascaded pipeline (VAD -> ASR -> LLM -> TTS), latency accumulates at every stage, typically totaling 1,200ms to 2,500ms—far too sluggish for natural human dialogue. In native Speech-to-Speech (such as OpenAI Realtime API and Gemini Live), audio waveforms are tokenized directly into continuous neural audio tokens. Total round-trip latency drops to 240-350ms, enabling real-time conversational rapport.

WebRTC vs. WebSocket Full-Duplex Audio #

Real-time voice APIs utilize WebRTC data channels and media streams for peer-to-peer audio transport. WebRTC's UDP-based transport prevents head-of-line blocking common in TCP, ensuring smooth audio even over lossy mobile cellular connections.

Interruption Handling & Acoustic Turn-Taking #

A crucial feature of conversational AI is barge-in: when the human user begins speaking while the AI is talking, the server immediately truncates the outgoing audio stream and discards downstream tokens, allowing seamless real-time interruption.

Code Example: Initializing a WebRTC Realtime Session with Voice Agent #

// Browser client creating WebRTC peer connection to Realtime Voice Gateway
async function initVoiceAgent(ephemeralToken: string) {
  const pc = new RTCPeerConnection();

  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  stream.getTracks().forEach((track) => pc.addTrack(track, stream));

  const audioEl = document.createElement("audio");
  audioEl.autoplay = true;
  pc.ontrack = (event) => { audioEl.srcObject = event.streams[0]; };

  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);

  const response = await fetch("https://api.apihundred.com/v1/realtime?model=gpt-live-1", {
    method: "POST",
    body: offer.sdp,
    headers: {
      Authorization: `Bearer ${ephemeralToken}`,
      "Content-Type": "application/sdp",
    },
  });

  const answerSdp = await response.text();
  await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });
  console.log("Realtime voice channel established.");
}

Frequently Asked Questions #

Q: What is barge-in in voice AI agents?

Barge-in allows human users to speak and interrupt the AI assistant mid-sentence, causing the AI to instantly cease vocal generation and listen.

Q: Why is WebRTC preferred over WebSockets for voice?

WebRTC uses UDP which avoids packet retransmission delays, ensuring minimal latency and preventing buffer stutters in live voice conversations.

Q: Can voice agents invoke API tools mid-conversation?

Yes, real-time voice sessions support function calling, allowing the agent to query a database or initiate an order while conversing naturally.