Multimodal & AudioAdvanced
Speech-to-Speech & WebRTC Real-Time Voice Agents: Building Voice AI in 2026
Direct Answer & Overview
Speech-to-Speech APIs bypass text conversion bottlenecks by processing audio tokens directly through native multimodal transformers, achieving natural sub-300ms conversational turn-taking, barge-in interruption handling, and human vocal realism.
1.The Cascaded Pipeline vs. Native S2S Architecture
In a cascaded pipeline (VAD -> ASR -> LLM -> TTS), latency accumulates at every stage, typically totaling 1,200ms to 2,500ms—far too sluggish for natural human dialogue. In native Speech-to-Speech (such as OpenAI Realtime API and Gemini Live), audio waveforms are tokenized directly into continuous neural audio tokens. Total round-trip latency drops to 240-350ms, enabling real-time conversational rapport.
2.WebRTC vs. WebSocket Full-Duplex Audio
Real-time voice APIs utilize WebRTC data channels and media streams for peer-to-peer audio transport. WebRTC's UDP-based transport prevents head-of-line blocking common in TCP, ensuring smooth audio even over lossy mobile cellular connections.
3.Interruption Handling & Acoustic Turn-Taking
A crucial feature of conversational AI is barge-in: when the human user begins speaking while the AI is talking, the server immediately truncates the outgoing audio stream and discards downstream tokens, allowing seamless real-time interruption.
Initializing a WebRTC Realtime Session with Voice Agenttypescript
// Browser client creating WebRTC peer connection to Realtime Voice Gateway
async function initVoiceAgent(ephemeralToken: string) {
const pc = new RTCPeerConnection();
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
stream.getTracks().forEach((track) => pc.addTrack(track, stream));
const audioEl = document.createElement("audio");
audioEl.autoplay = true;
pc.ontrack = (event) => { audioEl.srcObject = event.streams[0]; };
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
const response = await fetch("https://api.apihundred.com/v1/realtime?model=gpt-live-1", {
method: "POST",
body: offer.sdp,
headers: {
Authorization: `Bearer ${ephemeralToken}`,
"Content-Type": "application/sdp",
},
});
const answerSdp = await response.text();
await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });
console.log("Realtime voice channel established.");
}Frequently Asked Questions
What is barge-in in voice AI agents?
Barge-in allows human users to speak and interrupt the AI assistant mid-sentence, causing the AI to instantly cease vocal generation and listen.
Why is WebRTC preferred over WebSockets for voice?
WebRTC uses UDP which avoids packet retransmission delays, ensuring minimal latency and preventing buffer stutters in live voice conversations.
Can voice agents invoke API tools mid-conversation?
Yes, real-time voice sessions support function calling, allowing the agent to query a database or initiate an order while conversing naturally.
A100
API100 Engineering Team
Infrastructure & Latency Research