Turn-taking

A voice conversation is a negotiation over who holds the floor. The server runs voice activity detection (VAD) and turn endpointing for you; this page covers the events it emits, the knobs that tune it, and the escape hatches when you want manual control.

The default loop

1. The user speaks. The server emits user-started-speaking, streams transcript deltas, then user-stopped-speaking.
2. The endpointer decides the turn is over and the model responds: bot-llm-started, bot-tts-started, bot-started-speaking, agent audio, bot-stopped-speaking, turn-complete.
3. If the user starts talking while the agent is speaking, the agent is interrupted — playback stops and the model yields the floor (barge-in).

You don't configure any of this to get a working conversation. You tune it when the defaults don't fit the room.

Interruption sensitivity

interruption_sensitivity on the agent config controls how readily user audio barges in: "default", "high", or "low".

• high — the agent yields at the first hint of user speech. Good for fast, cooperative assistants in quiet rooms.
• low — the agent holds the floor through short interjections and background noise. Good for phone calls, speakerphones, and noisy environments where echoes and "mm-hm" backchannels shouldn't cut the agent off.
• default — the provider's tuned middle ground.

Providers map this to their own VAD parameters. It governs the start of speech — how easily the user takes the floor. How long the server then waits before deciding they've finished is endpointing, tuned separately.

Provider endpointing

Endpointing is the gap between the user's last word and the agent's first — the server has to be sure the user is done, not just pausing. That certainty costs time, and on a steady-state conversation it is usually the largest single component of perceived latency. Each provider exposes it differently, so the knobs live on that provider's model block.

Gemini offers two detectors, selected by turn_detection:

• "cosmo_vad" (also the default when unset) runs Cosmo's semantic turn detection: a classifier decides whether the utterance reads as finished, so a mid-thought pause doesn't end the turn and a completed one doesn't pay a full silence window. The cosmo_vad block tunes it: pause_ms (silence that triggers the end-of-turn inference), prefix_ms (audio kept from before speech was detected), and max_hold_ms (total silence after which the turn ends regardless of the classifier's verdict — the bound on how long a mid-thought verdict can hold the turn open).
• "server_vad" endpoints on a fixed silence window, tunable in three ways: end_of_speech_sensitivity ("high" closes the turn sooner, "low" waits for more confidence), silence_duration_ms (how much silence ends the turn), and prefix_padding_ms (how much audio is kept from before speech was detected, so an opening syllable isn't clipped). These three knobs are read only with "server_vad".

from cosmo_ai import CosmoVadConfig, GeminiModel

agent = client.agent(
    instructions="…",
    model=GeminiModel(
        turn_detection="cosmo_vad",
        cosmo_vad=CosmoVadConfig(pause_ms=250, max_hold_ms=900),
    ),
)

from cosmo_ai import GeminiModel

agent = client.agent(
    instructions="…",
    model=GeminiModel(
        turn_detection="server_vad",
        end_of_speech_sensitivity="high",
        silence_duration_ms=200,
    ),
)

OpenAI offers two detectors, selected by turn_detection:

• "server_vad" (the default) times a fixed silence window — silence_duration_ms and prefix_padding_ms tune it.
• "semantic_vad" runs a classifier that closes the turn as soon as the utterance reads as complete, so a speaker who trails off doesn't pay the full silence window. eagerness paces it: "high" answers sooner, "low" waits longer for the user to continue, "auto" is the provider default.

import { OpenAIModel } from 'cosmo-ai';

const agent = client.agent({
  instructions: '…',
  model: OpenAIModel({ turnDetection: 'semantic_vad', eagerness: 'high' }),
});

Grok runs one detector, the fixed silence window, so silence_duration_ms and prefix_padding_ms always apply and there is nothing to select between:

import { GrokModel } from 'cosmo-ai';

const agent = client.agent({
  instructions: '…',
  model: GrokModel({ turnDetection: 'server_vad', silenceDurationMs: 200 }),
});

xAI exposes no semantic endpointer, and Cosmo's own detector needs to take the turn away from the provider — which the Grok integration does not permit — so "semantic_vad" and "cosmo_vad" are both rejected there at session start rather than quietly downgraded to the window.

OpenAI GPT Live is full-duplex: it listens while it speaks and decides itself when each turn starts and ends, so the block carries no detector and no window. interruption_sensitivity has nothing to bind to either — barge-in is the model's own call. What the block does tune is the backend Responses model GPT Live hands tool calls and reasoning to — it cannot call tools itself:

import { OpenAILiveModel } from 'cosmo-ai';

const agent = client.agent({
  instructions: '…',
  model: OpenAILiveModel({
    responsesModel: 'gpt-5-nano',
    reasoningEffort: 'low',
    verbosity: 'low',
    toolChoice: 'required',
  }),
});

responsesInstructions gives that model its own brief; unset, it reads the agent's instructions. toolChoice: 'required' makes every delegated turn call a tool, which is what a UI-driving agent usually wants; verbosity: 'low' keeps what comes back for the voice model to say short. The voice model is billed per second of session; the backend model's tokens are billed separately under its own name. GPT Live is audio-only — video and screen frames sent to a session on it are discarded.

delegation picks who does the work the voice model hands off. The default, 'responses', is the backend model above. 'client' hands each request to your application instead: the session emits delegation_created with what the user said, no tools run, and you answer through appendThinking (background the model keeps to itself), appendCommentary (something to say now, in its own words) and appendInstructions (how to behave from here on), passing the event's delegationId. 'cosmo' runs the request on Cosmo's workspace agent on the server, with the workspace's tools and skills, and narrates it back over the same channels; the event still reaches you, and the three appends still steer the model. The voice model keeps talking with the user while the work runs.

Two things about hand-offs are worth knowing before you build on them. GPT Live hands off on speech, not on text — a sendText turn to a GPT Live session produces no reply at all, in any delegation mode, so a delegation path has to be tested by speaking. And a hand-off does not always carry what the user said: the model hands off mid-turn, and transcript arrives empty. The SDK fills it with the session's last user turn when that happens, so your backend still has something to act on; each turn stands in for at most one hand-off, so two blank hand-offs in a row never replay the same instruction, and a hand-off raised before the user's first word still arrives empty.

Each knob belongs to one detector. On OpenAI, sending eagerness alongside server_vad, or a silence window alongside semantic_vad, is rejected at session start rather than silently ignored. The same rule holds on Gemini once a detector is named: "cosmo_vad" with a server_vad knob, or "server_vad" with a cosmo_vad block, is rejected. "semantic_vad" is an OpenAI value and rejected on Gemini, where "cosmo_vad" is the semantic detector — and "cosmo_vad" is likewise rejected on OpenAI.

Tighter endpointing is a trade, not a free win: every millisecond you cut is a millisecond less evidence that the user actually finished, so turns start fragmenting across natural mid-thought pauses. Tune against real recordings from the room the agent will run in.

Noise cancellation

Background voices triggering false barge-ins, or showing up in the transcript? audio.noise_cancellation on the agent config applies background-voice cancellation to inbound audio before it reaches the model. It is off by default. Use voice_focus when one person should be heard and the room should not — cafés, open offices, TV in the background; it reduces both false barge-ins and transcription of bystanders. Use denoise when several people share the microphone, since voice_focus would treat all but one of them as background.

The isolator sits ahead of the model, so the filtered signal is also what endpointing reads — it costs some barge-in responsiveness. On clean audio it buys little and can move turn boundaries: enable it for noisy environments, leave it off for quiet-room or headset use, and measure end-of-turn latency on your own recordings before leaving it on.

Manual turn boundaries

If your app knows better than the endpointer when a turn ends — push-to-talk, a hardware button, a form submit — send the boundary yourself:

await session.sendActivityEnd();

await session.set_muted(False)     # open the floor
# … user holds the talk button …
await session.set_muted(True)
await session.send_activity_end()  # "the turn is over, respond now"

try await session.sendActivityEnd()

activity-end marks the end of the user's turn; it doesn't end the session. Combined with mute, this gives you full push-to-talk semantics: unmute while the button is held, then mute + activity-end on release.

Text turns have explicit boundaries by nature — send_text(...) is always a complete turn.

Silence handling

What should happen when the user says nothing for 30 seconds? Declare it, don't poll for it — a server-side silence-timeout hook runs even if your process dies mid-call:

from cosmo_ai.hooks import SilenceTimeout, Say, EndCall

agent = client.agent(
    instructions="…",
    hooks=[
        SilenceTimeout(timeout_seconds=30, action=Say(prompt="Gently check whether the caller is still there."), max_count=2, reset_mode="on_user_speech"),
        SilenceTimeout(timeout_seconds=90, action=EndCall(farewell="I'll let you go — call back anytime.")),
    ],
)

The client observes each firing via the user-speech-timeout session event, which reports the silence duration, the trigger count, and the action the server already took.

Recommended defaults by use case