How OpenAI Built GPT-Live

Voice systems consistently operate with audio inputs from the user and generate audio responses played back to the user. Although the input and output modalities remain the same, the intermediate processing differs according to the system design. Across their development, voice systems have evolved through three distinct architectural generations. These generations are categorized as cascaded design, turn-based end-to-end, and full-duplex architecture.

The cascaded design for voice systems chains three distinct models sequentially: an automatic speech recognition (ASR) model, a large language model (LLM), and a text-to-speech (TTS) model. While this pipeline allows each specialized component to handle what it does best, it introduces significant drawbacks including the loss of vocal cues such as tone and emotion during transcription. Furthermore, operating three systems in series leads to high latency and substantial operational complexity in serving and scaling. These challenges prompted the development of second-generation turn-based speech-to-speech architectures.

The second generation of voice AI introduces end-to-end speech-to-speech models that directly process and generate audio, capturing vocal nuances lost in cascaded systems. Despite this improvement, interactions remain turn-based because they still depend on turn-detection and interruption mechanisms to manage speaking flow. These detectors struggle with trade-offs between unnatural delays and premature cutoffs or false triggers. Furthermore, speech-to-speech systems are costly to keep current because integrating updated base LLM checkpoints requires full retraining runs.

Full-duplex voice models eliminate traditional turn detectors by continuously listening and generating audio tokens simultaneously, decoding silence as normal tokens. Reference models like Moshi demonstrate this by operating on a fixed clock of roughly 80 milliseconds per frame, learning natural conversational turn-taking directly from training data. While this approach resolves conversation unnaturalness and interruption handling, it introduces high serving costs due to constant inference and demands compact model capacity for millisecond latency. Systems like OpenAI's GPT-Live-1 address these constraints by keeping the primary voice model small and fast while delegating complex reasoning tasks.

OpenAI developed a voice assistant centered around a full-duplex architecture. The implementation addresses key technical ideas and techniques necessary to operate effectively at scale in production. This approach focuses on making real-time, bidirectional voice interactions viable for large-scale production deployment.

Standard frontier language models introduce latency when searching and reasoning, which creates awkward silence in voice-based systems. To solve this, OpenAI decouples conversational interaction from deep reasoning tasks. A voice model handles the real-time dialogue, while a separate, more capable model like GPT-5.5 performs necessary reasoning and tool calls in the background. This architecture reduces the traditional trade-off between response latency and answer quality. Furthermore, the modular design simplifies future upgrades by allowing newer reasoning models to be swapped in with minimal engineering effort.

Serving two models concurrently requires decoupling fast and slow operations because audio frames require millisecond latency while delegation can take seconds. To prevent latency spikes, GPT-Live splits traffic into two separate execution pathways. The live path strictly manages rapid audio transfer between the client and voice model. Meanwhile, the async path handles slower, non-audio tasks like tool calls without interfering with live audio streaming.

Full-duplex voice systems like GPT-Live require real-time audio transmission on a strict fixed clock to prevent audible artifacts from processing bottlenecks. To minimize initial setup latency, OpenAI created the WARP protocol to compress standard multi-step WebRTC handshakes into a single round trip. Continuous audio inference costs are managed by keeping conversation context pinned in GPU memory on dedicated model instances, alongside batching and speculative decoding. Additionally, OpenAI employs a managed handoff mechanism to seamlessly switch live conversations to replacement instances during context compaction, scaling, or instance failures.

The async path is responsible for managing non-audio tasks, including delegations and tool calling, between models in a real-time system. To make the interaction feel seamless, latency must be minimized, particularly the time spent on the prefill phase of prompt processing. OpenAI mitigates this delay by initiating an inference session with the frontier model at the start of a conversation and pre-loading context before a delegation is triggered.

Evaluating full-duplex voice systems requires a different approach than traditional turn-based speech models. Turn-based models can be scored per discrete request-and-response turn, whereas full-duplex architectures operate on one continuous audio stream without distinct turns. Consequently, evaluation in full-duplex architectures focuses on three main components: conversational behavior, stream health, and testing under real production traffic.

Evaluating conversational behavior in models primarily hinges on timing decisions such as when to speak, remain silent, or interrupt. The two core decisions are endpointing, which determines turn completion, and barge-in detection, which distinguishes genuine interruptions from background noise. These behaviors are evaluated like prediction tasks using annotated natural conversation datasets.

Evaluating stream health in serving systems traditionally relies on percentile targets like p95 latency, which suffices when requests are intermittent. In full-duplex architectures running continuous inference, a p95 event occurs every 20 inferences, resulting in multiple slow events per minute. Because even p99 delays appear regularly, systems must be engineered around p999 targets. In practice, full-duplex designs must emphasize fast recovery since occasional slow frames are inevitable in every session.

A silent launch serves as the final layer of evaluation to safely verify system reliability under real production traffic. During this process, users continue interacting with the existing system while a small fraction of voice sessions is routed to the new system. This approach uncovers difficult-to-detect issues, such as unexpected infrastructure bottlenecks. For example, during the silent launch of GPT-Live, the engineering team discovered that a CPU-side service exhausted capacity before the GPUs did.

Realtime serving architectures, such as OpenAI's GPT-Live, present distinct challenges compared to traditional request-response systems. In voice applications, engineering must prioritize tail latency over average latency, and capacity planning must track concurrent sessions rather than individual requests. Furthermore, moving complex tasks like turn detection directly into the model increases conversational naturalness while reducing overall system complexity. Keeping components outside the model minimal and focused strictly on realtime delivery ensures the broader system remains maintainable.

OpenAI views voice-driven computer interaction as the next major paradigm shift in computing, as demonstrated through features in the ChatGPT desktop app like screen perception and asynchronous task execution. Users can continuously query, guide, and redirect tasks via voice in real time. However, challenges persist in smoothly delegating tasks across different models without introducing latency into live conversation. While systems like GPT-Live-1 demonstrate rapid progress likened to the jump from GPT-3 to GPT-4, complex latency and architecture questions remain open.