Semantic VAD vs. server VAD: stop cutting off unfinished sentences

A caller says "I need to change, um..." and stops to think. Server VAD hears silence and answers. Semantic VAD guesses the thought is unfinished and waits longer.

Duplex Compute · October 13, 2026 · Docs and versions checked October 13, 2026

Three jobs hide behind the phrase "turn detection". Voice activity detection decides whether a stretch of audio contains speech. End-of-turn detection decides whether the speaker has finished a thought. Barge-in handling decides whether speech that arrives while the agent talks should stop it. LiveKit's turns guide describes the same layering: most systems run VAD first, apply endpointing heuristics to its output, and can add a model that reads the meaning of the speech.

Semantic VAD works on the second job. It does not make a half-duplex agent full duplex, and it does not read intent. It predicts whether the words so far sound complete.

What server VAD measures

In the OpenAI Realtime API, server_vad is the default mode and chunks audio on periods of silence. You tune it with three numbers. The API reference gives their defaults.

The reference states the tradeoff directly: with a shorter silence_duration_ms "the model will respond more quickly, but may jump in on short pauses from the user." Server VAD has no idea what the user said. A 600 ms breath after "um" looks the same as 600 ms of silence after "thanks, that's all."

What semantic VAD adds

OpenAI describes semantic_vad as a semantic classifier that scores the probability the user is done speaking. When the score is low, the model waits for a timeout. When it is high, it does not wait. Speech that trails off gets a longer timeout than a definitive statement.

The only tuning knob is eagerness. The reference gives each level a maximum wait.

The maximum timeout is a ceiling for the unsure case. A confident "that's all" can end the turn well before it. Semantic VAD has no threshold or silence_duration_ms; those fields are documented for server_vad only.

One sentence, two detectors

The table below walks one hesitant utterance and one complete sentence through the documented rules. The timings are illustrative arithmetic from the documented settings, not measurements. We did not record audio or run these configurations. Where semantic VAD is involved, the classifier's actual score decides the timing, so the table gives the window the rule allows.

Utterance A (hesitant)
0.0s         1.4s        2.5s                     4.1s
|I need to change, um...| pause |the date of my booking.| silence ...

Utterance B (complete)
0.0s                                   2.4s
|I need to change the date of my booking.| silence ...

A fixed silence threshold forces one tradeoff on every turn: short enough to feel fast is short enough to cut people off. Semantic VAD moves that tradeoff per turn, using the words. It can still be wrong in both directions. A user who pauses after a grammatical fragment ("I want to fly to Boston." ... "On Friday.") gives the classifier a complete-sounding sentence, and the agent can answer before the date arrives.

To produce real numbers, record both utterances, stream them through each configuration, log input_audio_buffer.speech_stopped and the start of the response, and check whether the task still succeeded. Run each clip several times. Our evaluation note covers how to score the result.

The current configuration shape

In the current guide, turn detection lives under session.audio.input.turn_detection and is set with a session.update event. Check any example you copy from an older blog post against that path.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": {
        "turn_detection": {
          "type": "semantic_vad",
          "eagerness": "low",
          "create_response": true,
          "interrupt_response": true
        }
      }
    }
  }
}

For server VAD, swap the type and add the three numbers:

"turn_detection": {
  "type": "server_vad",
  "threshold": 0.5,
  "prefix_padding_ms": 300,
  "silence_duration_ms": 500,
  "create_response": true,
  "interrupt_response": true
}

create_response and interrupt_response apply only in speech-to-speech sessions. Set both to false and VAD keeps emitting speech events while your client decides when to call response.create. That split is useful when your own logic, such as a tool still running, should decide when the agent speaks. Server VAD also accepts idle_timeout_ms, which prompts the model to speak after a long silence; the reference lists it for server_vad only.

With interrupt_response on, a VAD start event cancels the agent's ongoing response. A user's "mm-hm" while the agent talks is speech, so it can stop the reply. Barge-in debugging has its own note: why your Realtime agent keeps talking after an interruption.

LiveKit's detector modes

LiveKit Agents exposes the same choice as a mode on AgentSession. Its turns guide lists five.

The LiveKit turn detector page describes an audio model that combines meaning with intonation, pitch and rhythm. It ships as v1 on LiveKit Inference and v1-mini for local CPU use. With it enabled, the session's endpointing delay drops from 0.5–3.0 s to 0.3–2.5 s. The page's own example shows a text-based detector committing a flight-booking turn three times at mid-turn pauses while the audio model commits once at 6.41 s. That is LiveKit's demonstration, not an independent benchmark.

LiveKit can also run its own detector in front of OpenAI Realtime. Set the model's turn_detection to None, give the session Silero VAD and the turn detector, and the agent decides when each turn ends. LiveKit notes that Gemini Live keeps turn-taking on its own signals and does not support this client-side mode. The open-source alternative to these hosted detectors is a two-stage pipeline; our note on Silero VAD and Smart Turn builds one in Python.

Where semantic VAD stops

Every detector here produces one decision: the user's turn is over, start the reply. Semantic VAD makes that decision better informed. Once the turn is committed, the reply runs until VAD hears new speech, and that speech arrives as an interruption. Backchannels ("uh-huh"), corrections mid-reply and two people talking at once all fall to the barge-in layer.

A full-duplex model removes the single commit point. OpenAI describes GPT-Live as able to listen while speaking. Duplex Compute is a full-duplex voice API with tool calling and configurable behavior, at $0.01 per connected minute. If your agent's failures cluster around pauses, corrections and talk-over, compare that architecture against tuning another threshold; our full-duplex explainer covers the difference.

Related notes

• Silero VAD and Smart Turn: a two-stage end-of-turn detector in Python
• Why your Realtime agent keeps talking after an interruption
• Full-duplex voice AI vs. turn-based
• Pipecat vs. LiveKit
• Voice agent evaluation