Wire protocol
All control-plane message types on the Cosmo Realtime data channel.
Wire protocol
All control messages ride a LiveKit reliable data channel as JSON, discriminated by a type string. Binary audio and video ride separate tracks and are not described here. Messages carry no per-message id — correlation rides session_id (server-assigned on ready) and, for tools, tool_call_id.
EXTERNAL_PROTOCOL_VERSION = "1.0" — carried in the version field of session-config and ready. Either side may refuse an incompatible major version (version_mismatch). See Protocol version.
For the narrative version of this page — which events to use for which UI — see Realtime events.
Forward compatibility. SDKs surface a message with an unrecognized type as an explicit unknown event and keep the session alive — never as a terminal decode error. The server rejects unknown inbound types with error (code=invalid_message).
Client → server messages
type | Fields | Purpose |
|---|---|---|
session-config | version, agent, session | Session-start payload. Carried on the POST /session/start request body — see POST /session/start for the full field reference. The server answers with ready on the data channel once the agent is up. |
mute | muted: bool | Toggle the mic gate. While muted the client drops outbound audio frames. |
send-text | content: string | Send a text turn instead of audio; the model answers it. |
send-context | content: string (1–4096 chars) | Hand the model context without asking it to reply. Delivered on the provider's pre-turn channel (turn_complete=false), so no turn is opened and nothing is spoken; the model reads it as background on its next answer. For live application state. A note outside the length bounds is rejected at the wire boundary and never reaches the model. |
send-image | mime_type: string = "image/jpeg", data: string (base64), stream_id: string = "video.input.default" | One image frame (screen share, camera, any visual input). stream_id labels concurrent video streams. |
activity-end | — | End-of-turn signal for manual-VAD turn-taking. Distinct from end (whole-session teardown). See Turn-taking. |
bind-input | — | Bind the agent's audio input to this client, sent after publishing audio. The server pins the sender's participant identity; a client can only bind its own input. Observers that publish no audio never send this. |
tool_job_result | job_id: string, tool_name: string, status: "completed" | "failed", result?: object, summary?: string, error?: string | Terminal result of a long-running (background) client tool, delivered off-band. summary / error are the model-facing text; result is structured data for logging. |
end | — | User ended the session. The server tears down the upstream session and closes. |
ping | — | Heartbeat; server replies pong. |
envelope-chunk | envelope_id: string, seq: int, total: int, data: string (base64) | Framing carrier for oversized client messages — see Envelope chunking. Emitted by the SDK, never directly. |
turn-context (per-turn desktop context) is a first-party Cosmo client message on the same channel; it is not part of the public SDK surface.
Server → client messages
Session lifecycle
ready
Sent after the upstream session is established and the agent is ready.
| Field | Type | Description |
|---|---|---|
version | string | Server protocol version ("1.0"). |
session_id | UUID | Server-assigned session id. Persist it and pass it back as experimental.resume_session_id on a fresh session-config to resume after a disconnect. |
agent | object | null | Resolved catalog-agent summary (name, effective tools[]) when the session referenced one; null for inline sessions. Informational only. |
rejected_tools | {name, reason}[] | Client-tool specs the server refused (sanitization, schema caps, name collisions). The session still starts without them. |
cosmo | object | null | First-party extension block; null for a fully agnostic session. |
max_session_seconds | int | null | Effective server-enforced duration cap; null = no cap. |
reconnecting
Server is transparently rotating the upstream model session. Transport and session state survive the swap.
| Field | Type | Description |
|---|---|---|
version | string | Protocol version. |
seconds_remaining | float | null | Optional ETA hint. |
session-ending-soon
The server will end this session shortly (e.g. the max-duration cap). The session keeps running until session-ended.
| Field | Type | Description |
|---|---|---|
seconds_remaining | float | Time until cutoff. |
reason | string | Stable slug (e.g. max_session_duration). |
session-ended
The server ended the session on purpose, published best-effort just before the transport tears down — the room close may still arrive first. Every SDK folds this into a guaranteed terminal session-ended item on its event stream.
| Field | Type | Description |
|---|---|---|
reason | string | Stable slug; same vocabulary as session-ending-soon.reason. |
error
| Field | Type | Description |
|---|---|---|
code | RealtimeErrorCode | See error codes. |
message | string | Human-readable reason. |
fatal | bool | true: the session is dead — tear down and reconnect. false: this turn failed but the session continues. |
pong
Reply to ping. No fields.
Transcript and text
transcript
Streaming transcript for either speaker. For the assistant in audio sessions this is the audio transcription — the words the listener actually heard.
| Field | Type | Description |
|---|---|---|
role | "USER" | "ASSISTANT" | Speaker. |
text | string | While is_final is false: the new fragment since the previous event (append). On is_final: true: the cumulative full transcript for the turn (replace). |
is_final | bool | Terminates the turn's stream. |
model-text
Streaming text-channel fragment from the model — distinct from transcript: text the model emits alongside its audio, not a transcription of it. In text-only sessions it carries the response; in audio sessions it may include narration that was never spoken.
| Field | Type | Description |
|---|---|---|
text | string | Fragment. |
is_final | bool | Defaults false. |
turn-complete
Marks the end of a turn so the client can finalize a transcript bubble.
| Field | Type | Description |
|---|---|---|
role | "USER" | "ASSISTANT" | Whose turn ended. |
Speech and processing markers
Field-less, informational — drive UI indicators, never mic gating.
type | When |
|---|---|
user-started-speaking / user-stopped-speaking | Server-side VAD detected user voice activity start / stop. |
bot-started-speaking / bot-stopped-speaking | First / last audio frame of the assistant turn left the server. |
bot-llm-started / bot-llm-stopped | Model began / finished generating (before audio is on the wire — useful for "thinking…" states). |
bot-tts-started / bot-tts-stopped | Assistant TTS audio frames started / stopped flowing. |
user-speech-timeout
A server-runtime silence hook fired; the server already performed action. Observability only.
| Field | Type | Description |
|---|---|---|
session_id | string | Session the hook fired in. |
silence_ms | int | Measured user silence. |
trigger_count | int | How many times this hook has fired. |
max_count | int | Configured cap. |
action | Say | EndCall | What the server did ({type: "say", text?, prompt?} or {type: "end_call", farewell?}). |
Tools
Server-executed tools surface as a three-event lifecycle sharing tool_call_id (the upstream's function-call id):
tool-call
Model decided to invoke a server-executed tool.
| Field | Type |
|---|---|
tool_call_id | string |
name | string |
tool-dispatch-started
Server-side handler began executing — lets UIs show a "dispatching…" state for slow tools.
| Field | Type |
|---|---|
tool_call_id | string |
name | string |
tool-result
Server-side tool finished.
| Field | Type | Description |
|---|---|---|
tool_call_id | string | Correlates the triple. |
ok | bool | Success. |
summary | string | null | Short human-readable line. |
tool-invocation
Server asks the connected client to run a tool locally — sent only for tools declared via kind: "client" specs at session start. The result returns out of band (transport RPC), not as a data-channel send.
| Field | Type | Description |
|---|---|---|
request_id | string | Dispatch correlation id. |
tool_call_id | string | Upstream function-call id (correlates with the observability triple). |
name | string | Tool name. |
args | object | Decoded arguments. |
origin | "realtime" | "server" | Which producer authored the invocation (voice model vs. a server tool runtime). |
executable | bool | false marks an informational mirror whose execution runs out-of-band; true (default) means the recipient runs the tool. |
First-party cosmo.* events
cosmo.usage
Cumulative token usage for the live session, split by direction and modality. All fields are ints defaulting to 0: input_text_tokens, input_image_tokens, input_audio_tokens, input_cached_tokens, output_text_tokens, output_audio_tokens, total_tokens.
cosmo.session-state
Live session state after a set_state tool write. Carries the full canonical state, not a delta. See Session state.
| Field | Type | Description |
|---|---|---|
state | object | Full canonical state after the merge. |
updated_keys | string[] | Keys touched by the write. |
warnings | string[] | Advisory schema findings the model also saw in its tool result. |
stage | string | null | Hoisted from state["stage"]. |
Chunked envelope
server-envelope-chunk
Mirror of envelope-chunk for the server→client direction: envelope_id, seq, total, data (base64). SDKs reassemble transparently and re-dispatch the inner message.
RealtimeErrorCode values
| Value | Meaning |
|---|---|
auth_failed | Credential invalid or expired. |
workspace_forbidden | Credential valid but workspace access denied. |
voice_disabled | Voice is not enabled for this workspace. |
upstream_disconnect | Upstream model session disconnected unexpectedly. |
internal_error | Unrecognized server-side error. |
invalid_message | Client sent a malformed or unknown control message. |
version_mismatch | Client protocol version incompatible with the server. |
Envelope chunking
Every message must be valid JSON (UTF-8). Oversized messages are split into envelope-chunk / server-envelope-chunk fragments; data is base64 over the UTF-8 bytes of the inner message JSON, so byte-boundary splits never land mid-codepoint. The receiving side buffers by envelope_id, reassembles in seq order (0..total-1), and dispatches the inner message as if it arrived whole. See Envelope chunking.