Voice agent evaluation: check the final state, not the transcript

A caller says "book a table for six, actually seven." The agent passes if your database holds one reservation for seven and the confirmation came after the write. Everything else in a voice agent evaluation explains why it passed or failed.

Duplex Compute · October 13, 2026 · Sources checked October 13, 2026

Start from one scenario file

A scenario pins down what the caller says, what the agent may do and what must be true afterwards. OpenAI's voice agent evaluation cookbook gives each scenario "an ID, required behavior, forbidden behavior, expected evidence, and stable fixture." Its running example is a corrected date: "August 7, sorry, August 6." The template below applies the same idea to a corrected party size and adds timing anchors and cost fields. Copy it and change the domain.

{
  "id": "booking_party_size_correction_001",
  "description": "Caller corrects the party size mid-sentence.",
  "input": {
    "audio": "fixtures/booking_six_actually_seven.wav",
    "reference_transcript": "Hi, can I book a table for six, actually seven, tomorrow at eight under Rivera?",
    "disfluency": ["self_correction"],
    "audio_source": "human_recording",
    "context": { "today": "2026-10-13", "restaurant": "demo_bistro" }
  },
  "expected": {
    "tools_required": [
      { "name": "check_availability",
        "args": { "date": "2026-10-14", "time": "20:00", "party_size": 7 } },
      { "name": "create_reservation",
        "args": { "name": "Rivera", "date": "2026-10-14", "time": "20:00", "party_size": 7 },
        "max_calls": 1 }
    ],
    "forbidden": [
      { "tool": "create_reservation", "args_match": { "party_size": 6 } },
      { "rule": "more_than_one_reservation_created" },
      { "rule": "spoken_confirmation_before_tool_success" }
    ],
    "final_state": {
      "reservations": [
        { "name": "Rivera", "date": "2026-10-14", "time": "20:00", "party_size": 7 }
      ]
    },
    "spoken_confirmation": { "must_mention": ["seven", "eight"], "after_event": "create_reservation.success" }
  },
  "timing_anchors": {
    "t0": "end_of_caller_speech",
    "first_sound": "first assistant audio after t0, including fillers",
    "first_useful_answer": "first assistant audio stating availability or the booking",
    "task_complete": "reservation row committed"
  },
  "run": { "repeats": 20, "record": ["audio", "events", "tool_calls", "final_state", "usage"] },
  "cost": { "voice_seconds": null, "voice_rate_usd_per_min": null, "backend_usd": null, "tool_usd": null }
}

The forbidden list matters as much as the expected calls. A model that books six and then books seven ends with the right row and a second wrong one. A model that says "you're all set" while the write is still running sounds correct and may be lying.

Grade with state and evidence

The cookbook's guiding measure is task completion. A scenario passes when the assistant "reaches the expected application state, satisfies mandatory requirements, and avoids unauthorized actions." Deterministic checks verify tool calls, arguments, authorization, state and audio timing. An LLM judge scores the conversation itself, with partial credit.

Keep those two kinds of grade apart. The cookbook's own published run shows why. Its recorded booking passed task completion with tool accuracy at 50%, because the agent made one of two expected calls. The same report shows a 75% response rate, and the cookbook explains that the scorer counted the caller's goodbye as two response opportunities. Read the recording before you trust a metric. The cookbook asks you to sort each failure into an assistant failure, an infrastructure or evidence problem, or a grader issue.

Add realism in three steps

OpenAI's voice agents guide calls the stages Crawl, Walk and Run. Crawl streams synthetic speech for one controlled request. Walk replays recorded human audio of the same request, so voices, microphones and pauses change while the task stays fixed. Run puts an independent simulated caller into a continuous multi-turn conversation, where clarification, interruptions and changed requirements show up.

The cookbook harness targets GPT-Live. A separate Realtime evaluation guide covers the Realtime API. The scenario file above does not depend on either. Any stack that exposes tool calls, audio and a database can run it.

Self-corrections deserve their own fixtures. Full-Duplex-Bench v3 uses real human audio annotated for five disfluency categories, with chained API calls across four task domains. Its authors report that "self-correction handling and multi-step reasoning under hard scenarios remain the most consistent failure modes" across the six systems they tested. Our note on reading voice benchmarks explains what its Pass@1 scores do and do not say.

Time three moments, not one

"Response latency" in most harnesses means time to the first assistant audio. The cookbook defines it as the mean time from the end of caller speech to the first qualifying assistant audio, "including a spoken preamble," and notes it "is not time to task completion." A filler such as "let me check" starts that clock and answers nothing. The voice agents guide says to measure such acknowledgments separately from the answer the caller needs.

Voice models are nondeterministic, so run each scenario many times. The voice agents guide compares "the median and 95th percentile across similar calls"; the cookbook reports P50 and P90 per run. Whichever tail you choose, print the count beside it. With 20 runs, the p95 is the slowest run or close to it. One stalled call moves it.

Pass rates need the same care. 14 passes in 20 runs is 70%, and the 95% Wilson interval runs from about 48% to 85%. Two systems at 65% and 75% on 20 runs each have not been separated. Fix the caller audio, model version, prompt, tools and transport, and change one thing per comparison.

Cost per successful task

A cheap run that fails still costs money, and the user will try again. Divide total spend across all attempts by the number of tasks that passed:

cost per success = (voice + backend model + tools + transport, summed over ALL runs)
                   / runs that passed

An illustrative example, not a measurement: 20 runs of 2 minutes each on a voice API priced at $0.05 per connected minute is $2.00 of voice time. Assume $0.01 of backend model and tool spend per run, another $0.20. If 14 runs pass, the cost is $2.20 / 14, or $0.157 per successful booking. If a prompt change leaves spend the same but drops passes to 10, the cost rises to $0.22 per success.

Take the usage numbers from the provider, not from the wall clock. For GPT-Live the cookbook says to keep the final usage.seconds snapshot "rather than summing updates." For token-billed APIs such as Gemini Live, retained context is billed again each turn, so long scenarios cost more per minute than short ones. The voice agent pricing hub has the billing formula for each model.

The same suite compares providers fairly only if every provider runs it. Duplex Compute prices its full-duplex voice API at $0.01 per connected minute. That rate is a voice-session line. Whether it lowers cost per successful task depends on the pass rate your scenarios produce, which only your runs can show.

Related notes

• Reading voice benchmarks: tool calling, reply quality and Pass@1
• Voice agent pricing: calculator and billing formulas
• OpenAI Realtime function calling
• Semantic VAD vs. silence-based turn detection