Why your Realtime voice agent keeps talking after an interruption

The OpenAI Realtime API cancelled the response. The audio was already on your side of the connection. Stopping generation, stopping playback and fixing the conversation history are three separate jobs.

Duplex Compute · October 13, 2026 · Docs and versions checked October 13, 2026

The Realtime API reference says the server "will produce audio faster than realtime." A three-second reply can arrive in a fraction of that time. By the time the user says "wait, stop", most of the reply may sit in your playback queue, your phone provider's buffer, or a Web Audio schedule. Cancelling the response on the server stops audio that has not been generated yet. It does nothing to audio you already hold.

That is the most common cause of the symptom. The others are a cancellation that never fires, a history that claims the user heard words they did not, and a tool call that keeps running or runs twice.

WebRTC and SIP vs. WebSocket

Who owns the output buffer decides who has to clean it up. The Realtime conversations guide splits the work by transport.

OpenAI recommends WebRTC for playing audio on client devices such as browsers. WebSocket is the usual choice for server-to-server work, such as relaying audio to a phone call. In that setup the queue includes whatever the phone provider has buffered, and it needs flushing too.

What the bug looks like on a timeline

This is an illustrative WebSocket sequence with a client that queues audio and never flushes it. The times are made up to show the order of events. We did not measure them.

time   server                               client playback
0.0s   response.created
0.1s   response.output_audio.delta ...      starts playing reply (6.0 s long)
1.4s   last delta sent; reply fully queued  still playing, 1.3 s heard
2.0s   input_audio_buffer.speech_started    user: "No, the other one"
2.0s   response.done, status cancelled      nothing new to cancel
       (reason: turn_detected)
2.0s                                        still playing: 4.1 s left in queue
6.1s                                        reply finishes; history says the
                                            user heard all 6.0 s

The fix has two parts. Stop and empty the local queue the moment speech_started arrives. Then tell the server how much the user heard.

The WebSocket procedure

1. Listen for input_audio_buffer.speech_started. With interrupt_response on, the server cancels the in-progress response. Its response.done carries status cancelled with reason turn_detected.
2. Stop playback immediately and discard every queued chunk, including chunks scheduled ahead in Web Audio or held by a telephony relay.
3. Record how many milliseconds of the assistant item actually played.
4. Send conversation.item.truncate with that item's item_id, its content_index, and audio_end_ms set to the played duration. The server answers with conversation.item.truncated.
5. Drop any audio delta that arrives later for the cancelled response_id, so a stray chunk cannot restart playback.

If you turned interrupt_response off to control turns yourself, send response.cancel before step 2. The reference says calling it with no response in progress returns an error and leaves the session unaffected. Keep audio_end_ms at or below the audio you received; a larger value also returns an error.

This sketch has not been run. It assumes the default 24 kHz PCM output and a browser AudioContext.

const ctx = new AudioContext({ sampleRate: 24000 });
let playhead = 0;            // AudioContext time where the next chunk will start
let queued = [];             // scheduled AudioBufferSourceNodes
let current = null;          // { itemId, contentIndex, startedAt }
const dropped = new Set();   // response ids cancelled by an interruption

function playDelta(ev) {
  if (dropped.has(ev.response_id)) return;
  const pcm = new Int16Array(Uint8Array.from(atob(ev.delta), c => c.charCodeAt(0)).buffer);
  const buf = ctx.createBuffer(1, pcm.length, 24000);
  const ch = buf.getChannelData(0);
  for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;

  playhead = Math.max(playhead, ctx.currentTime);
  if (!current || current.itemId !== ev.item_id) {
    current = { itemId: ev.item_id, contentIndex: ev.content_index,
                responseId: ev.response_id, startedAt: playhead };
  }
  const src = ctx.createBufferSource();
  src.buffer = buf;
  src.connect(ctx.destination);
  src.start(playhead);
  playhead += buf.duration;
  queued.push(src);
  src.onended = () => { queued = queued.filter(s => s !== src); };
}

function onSpeechStarted() {
  const now = ctx.currentTime;
  const stillPlaying = current && now < playhead;
  for (const s of queued) s.stop();      // flush everything already scheduled
  queued = [];
  playhead = now;
  if (!stillPlaying) return;              // the user heard the whole item
  dropped.add(current.responseId);
  ws.send(JSON.stringify({
    type: "conversation.item.truncate",
    item_id: current.itemId,
    content_index: current.contentIndex,
    audio_end_ms: Math.floor((now - current.startedAt) * 1000),
  }));
  current = null;
}

ws.onmessage = (m) => {
  const ev = JSON.parse(m.data);
  if (ev.type === "response.output_audio.delta") playDelta(ev);
  if (ev.type === "input_audio_buffer.speech_started") onSpeechStarted();
};

Reconcile what the user heard

Truncation fixes the audio in the server's history. Text is coarser. The guide explains that the model cannot align transcript and audio precisely, so conversation.item.truncate removes the transcript for the unplayed portion without giving you a truncated transcript. If your application logs or acts on the assistant's text, mark that item as interrupted and store the played duration next to it. Do not keep the full generated text as if it were said.

This matters most when the cut-off part carried a fact. "Your total is forty dollars, and I've also added the insurance" means something different if the user heard only the first half. The model's next turn sees the truncated audio. Your CRM or order log should see the same thing.

Pending tools and irreversible actions

An interruption stops speech. It says nothing about a function call that is already running. In the Realtime API, a function call arrives with a call_id, your code executes it, and you return a function_call_output with the same call_id. If the user barges in while your handler runs, you decide what happens. OpenAI's GPT-Live guide states the same rule for its delegated backend: work can continue after the caller interrupts, and your application decides whether to finish or cancel it.

Our recommendations, which are design advice rather than API behaviour:

• Execute only complete arguments. Wait for response.function_call_arguments.done or the call in response.done; a cancelled response can leave a partial call.
• Key side effects by call_id. If the confirmation ("I've booked the 10 a.m.") is interrupted, the booking still happened. Never resubmit the action because the sentence about it was cut off.
• Sort tools into safe to cancel (search, lookup) and must finish (payment, booking). Cancel the first kind on interruption. Let the second finish and tell the user the result on the next turn.
• If the user's interruption corrects an argument ("no, Friday"), treat it as a new request. Cancel or reverse the old call explicitly; do not hope the model notices.

Coughs, "uh-huh" and false stops

With interrupt_response on, any detected speech start cancels the reply. A cough or an "uh-huh" can stop the agent mid-sentence. Raising threshold in server VAD helps with background noise. It does not tell backchannels from real interruptions. Frameworks handle this layer above the model: LiveKit offers adaptive interruption handling, a minimum duration or word count, and a false-interruption timeout after which the agent resumes. Pipecat offers a minimum-words start strategy and a model-based backchannel filter. Turn-end tuning is covered in our semantic VAD note.

What Pipecat and LiveKit do for you

On an interruption, Pipecat cancels the in-flight LLM completion, clears TTS buffers, and has the output transport drain unplayed audio. Its context aggregator commits only the text that was actually spoken. For speech-to-speech services such as OpenAI Realtime, the provider detects the barge-in and Pipecat follows its events, but Pipecat still flushes local audio output. Function calls are cancelled on interruption by default. Setting cancel_on_interruption=False makes a call asynchronous instead: the conversation continues and the result is added to context when it arrives.

Pipecat also separates two settings people confuse. Disabling interruptions means speech over the bot is still transcribed and answered after the bot finishes. Muting user input discards that speech. Pick the one that matches what you want the user's words to do.

LiveKit Agents stops the agent on user speech and truncates the conversation history to the part the user heard. With a realtime model doing server-side turn detection, LiveKit reacts to the model's interruption signal and ignores most of its own interruption options; tune the model's VAD instead.

Debugging checklist

1. Log every server event with a timestamp. Confirm input_audio_buffer.speech_started arrives when the user speaks. If it does not, the problem is VAD settings or input audio, not playback.
2. Check interrupt_response in the active session. Confirm a response.done with status cancelled and reason turn_detected follows the speech start. Reason client_cancelled means your own response.cancel did it.
3. Measure your queue. Log seconds of audio scheduled but not yet played at the moment of the interruption. Anything above zero that keeps playing is the bug.
4. Find every buffer: Web Audio schedule, jitter buffer, media server, phone provider. Each one needs a flush.
5. Drop late deltas for the cancelled response_id.
6. On WebSocket, confirm you send conversation.item.truncate and receive conversation.item.truncated. An error usually means a wrong item_id or an audio_end_ms past the audio you received.
7. On WebRTC, do not add your own truncate after an automatic interruption; the server already did it. Use response.cancel plus output_audio_buffer.clear only for manual stops.
8. Ask the agent "what was the last thing you said?" after an interruption. The answer should end where playback stopped.
9. Interrupt during a tool call. Check that safe tools cancel, side-effecting tools run once, and the next turn reports the result.
10. Play a cough, an "uh-huh", a "stop", and a corrected argument at the agent. Write down which ones stop it. Decide which ones should.

Much of this work exists because a turn-based model speaks a reply it has already generated, and your code has to unwind it when the user cuts in. A full-duplex model listens while it speaks; GPT-Live is one example. Duplex Compute is a full-duplex voice API with tool calling at $0.01 per connected minute. Tool side effects stay your application's responsibility on any architecture, so the section on pending tools applies there too. Our full-duplex explainer compares the two designs.

Related notes

• Semantic VAD vs. server VAD: stop cutting off unfinished sentences
• OpenAI Realtime function calling
• GPT-Live vs. Realtime API
• Full-duplex voice AI vs. turn-based
• Voice agent evaluation