How OpenAI delivers low-latency voice AI at scale

OpenAI has rebuilt its WebRTC stack to support real-time Voice AI with low latency and global scale, aiming to eliminate awkward pauses and delayed barge-ins during conversations. According to OpenAI, voice AI requires fast connection setups and low, stable media round-trip times so that turn-taking feels crisp. To achieve this, the company rearchitected its infrastructure to address scaling challenges involving media termination, stateful ICE and DTLS sessions, and global routing. While WebRTC is traditionally used for peer-to-peer calling, OpenAI utilizes it as a foundation for client-to-server real-time systems, leveraging its standardization of NAT traversal, encrypted transport, and codec negotiation. This allows continuous audio streaming, enabling spoken agents to transcribe, reason, or generate speech while the user is still talking. OpenAI evaluated selective forwarding units, which are common for multiparty calls, but ultimately chose a transceiver model tailored for 1:1 interactions. In this architecture, a WebRTC edge service terminates the client connection and converts media and events into simpler internal protocols for model inference. The initial implementation relied on a Go service built on Pion running on Kubernetes. However, the conventional one-port-per-session WebRTC model proved incompatible with Kubernetes because it requires managing large public UDP port ranges, which complicates load balancing, security, and autoscaling. To resolve this, OpenAI designed a split relay plus transceiver architecture that exposes a small, fixed UDP surface to the public internet while accurately routing packets to the transceiver owning each specific WebRTC session.