Nemotron VoiceChat tool calling: what the model supports and what hosts expose
NVIDIA's 11B full-duplex model has its own output channel for tool calls, and NVIDIA's container runs them live over a WebSocket. The cheapest hosted version, .wave's beta, exposes no tools at all.
How the model produces a tool call
NemotronLabs VoiceChat is an 11B end-to-end speech model that listens and speaks at the same time. A fast conformer encodes the caller's audio. An NVIDIA-Nemotron-Nano-9B-v2 backbone predicts text tokens, and a TTS decoder turns them into the agent's speech. A separate output channel predicts tool-calling scripts. NVIDIA calls it the first open full-duplex model to support tool calling while keeping the conversation going during tool execution.
Each tool can carry an "on-hold" message. The model speaks it as soon as it generates the text that triggers the call, so the caller hears "Sure, let me check the weather" instead of silence.
The prompt format is visible in the README. The default template lists tools inside <AVAILABLE_TOOLS>, the model writes calls as <TOOLCALL>[{"name": ..., "arguments": ...}]</TOOLCALL>, and results come back inside <TOOL_RESPONSE>. System prompts and tool responses must be ASCII-only. NVIDIA asks you to turn tool results into short, TTS-friendly ASCII sentences before passing them in, because the model reads them aloud.
Offline demo or live container
The repository offers two ways to run the same Hugging Face checkpoint. Only one of them executes a tool.
| Path | What runs | Tool execution | Good for |
|---|---|---|---|
| Offline inference | A conda environment and offline_voicechat_fc_infer.py on a WAV file | None. --api-response-json supplies a pre-written tool response | Checking which call and arguments the model predicts |
| Inference container | NVIDIA's container with CUDA, Triton and vLLM behind a bidirectional WebSocket | Live: your client runs the function and returns the result | Interactive testing and building a service |
The README says it plainly: offline function calling does not invoke a live tool. A demo that prints <TOOLCALL>[{"name": "generate_random_number", ...}]</TOOLCALL> shows that the model picked a tool. It does not show a working loop.
The live event loop
The container's protocol follows the OpenAI Realtime shape. Its API reference lists three events for tools. You register tools in session.update. The server sends response.function_call_arguments.done with call_id, name and JSON-encoded arguments. You run the function and send conversation.item.create with a function_call_output item that echoes the call_id.
{
"type": "session.update",
"session": {
"tools": [
{
"name": "get_weather",
"description": "Get current weather for a city",
"ack_messages": ["Sure, let me check the weather for you."],
"parameters": {
"type": "object",
"properties": { "city": { "type": "string", "description": "City name" } },
"required": ["city"]
}
}
]
}
}// server to client
{ "type": "response.function_call_arguments.done", "call_id": "call_<id>", "name": "get_weather", "arguments": "{\"city\":\"Austin\"}" }
// client to server
{ "type": "conversation.item.create",
"item": { "type": "function_call_output", "call_id": "call_<id>", "output": "It is 24 degrees and sunny in Austin." } }ack_messages is the on-hold message: the model speaks one of them while it waits. When you port OpenAI code, note that the deploy guide's flow shows the model resuming once the output arrives, with no response.create step. Sending a function_call_output when no tools were registered returns an error with code tools_not_set.
Documented tool limits
NVIDIA's README has a long known-limitations section. These are the parts that affect tool calling.
| Limitation | What to do in the application |
|---|---|
| A maximum of 5 tools per session is recommended; more may degrade performance | Group related actions behind one tool, or swap tool sets between sessions |
| Simultaneous multi-tool calls are not reliable | Design flows that need one call per step |
| The user cannot interrupt the agent during tool execution | Keep tools fast; put long work behind a quick "started" result |
| Long tool responses can delay speech | Return short sentences; use ack_messages |
| Wrong or skipped tools, invented arguments, mis-spoken results | Validate every argument; confirm writes; check the stored state, not the spoken sentence |
| In mixed chat and tool conversations, the model may answer from its own knowledge instead of calling a tool | Do not let a spoken answer stand in for a lookup the business depends on |
| Trained on audio context windows of no more than 2 minutes; longer context may not be retained | Keep task state in your application and pass what matters back in tool results |
| Runaway continuation can occur, including in tool-calling sessions | Watch for agent turns without user input; end or reset the session |
The default system prompt already pushes against invented arguments: "Tool-call arguments must be values the user spoke. If a required argument is missing, ask the user; never guess." It also tells the model not to retry a failed call and to say the API has an issue. Prompt rules lower the rate of bad calls. Your handler still has to reject them.
The no-interruption limit shapes booking flows most. A caller who says "a table for six" and then "actually, seven" while book_table runs cannot cut in. The correction arrives after the first result. If the handler created a reservation, the second call has to update it. Key writes on the task, as the Realtime function calling note shows, so a corrected call never books twice.
What self-hosting takes
The README and the container prerequisites require an NVIDIA GPU with at least 80 GB of memory. The prerequisites list A100, H100, RTX 6000 Pro and B200, on x86_64 Linux with Docker and the NVIDIA Container Toolkit. The NeMo code is Apache 2.0. The model is licensed under OpenMDW 1.1. The checkpoint has one fixed voice and no voice cloning.
NVIDIA does not publish how many simultaneous conversations one GPU sustains. The client has a --num-streams flag for load tests, which is the tool to measure it with. GPU cost per conversation-hour is the GPU's hourly price divided by measured concurrent sessions times average utilization; the self-hosting note works through that formula.
What .wave's hosted beta exposes
.wave hosts the model as nemotron-voicechat at $0.005 per minute, an introductory beta price billed while the session is open. Its limits section is short. Sessions are capped at 120 seconds. The model speaks in one voice, aria. Custom instructions, text input and tools are not supported.
The checkpoint's tool channel does not reach this API. A host decides which model features it wraps, and here it wrapped speech in and speech out. When you compare open voice models by price, read the host's limits section next to the model card. A benchmark score for the model's tool use says nothing about a hosted endpoint that cannot accept a tool definition.
| Option | Tools | Session length | Cost basis |
|---|---|---|---|
| Offline script | Predicted only, with a canned response | One WAV file | Your GPU time |
| NVIDIA container, self-hosted | Live, executed by your client | Model trained on context up to 2 minutes | An 80 GB GPU, idle or busy |
| .wave beta | Not supported | 120 seconds | $0.005 per minute |
Before you build on it
Run one live call against the container and check the stored result. Start a deliberately slow tool and try to interrupt it, so you hear what callers will hear. Then correct an argument mid-request and count the records you end up with. Those are the experiments the documentation leaves open; this note did not run them.
If you want a hosted full-duplex API with tool calling and longer conversations, without running an 80 GB GPU, Duplex Compute prices sessions at $0.01 per connected minute.
Related notes
- Self-hosted voice AI vs. managed APIs
- Low-cost voice APIs with tool calling
- OpenAI Realtime function calling
- Full-duplex voice AI vs. turn-based
Want full duplex and tools without the GPU?
Duplex Compute is a hosted full-duplex voice API with tool calling at $0.01 per connected minute. Tell us your expected voice hours, the tools your agent calls, and the provider or model you use now.
Request access