Wire protocol
All control-plane message types on the Cosmo Realtime data channel.
All control messages ride a LiveKit reliable data channel as JSON, discriminated by a type string. Binary audio and video ride separate tracks and aren't described here. Messages carry no per-message id — correlation rides session_id (server-assigned on ready) and, for tools, tool_call_id.
The protocol carries no version number — it evolves additively, and every session-config identifies its client in the required sdk field ({name, version}; the SDKs stamp it automatically, and a direct REST caller supplies its own). See Protocol compatibility.
For the narrative version — which events to use for which UI — see Realtime events.
Every frame is a single JSON object discriminated by type. A server transcript frame, verbatim:
{
"type": "transcript",
"role": "ASSISTANT",
"text": "You're all set for Tuesday.",
"is_final": false
}Forward compatibility. SDKs surface a message with an unrecognized type as an explicit unknown event and keep the session alive — never as a terminal decode error. The server drops an inbound message it can't parse or route, logs it, and leaves the session running; it sends no error frame back, so a client that mistypes a message gets silence rather than a rejection.
Client → server messages
Each row is one client → server message: its discriminator, its fields, and when to send it. session-config rides the session-start HTTP request; everything else rides the data channel.
type | Fields | Purpose |
|---|---|---|
session-config | sdk, agent, session | Session-start payload. Carried on the POST /session/start request body — see POST /session/start for the full field reference. The server answers with ready on the data channel once the agent is up. |
mute | muted: bool | Toggle the mic gate. While muted the client drops outbound audio frames. |
send-text | content: string | Send a text turn instead of audio; the model answers it. |
send-context | content: string (1–4096 chars) | Hand the model context without asking it to reply. Delivered on the provider's pre-turn channel (turn_complete=false), so no turn is opened and nothing is spoken; the model reads it as background on its next answer. For live application state. A note outside the length bounds is rejected at the wire boundary and never reaches the model. |
send-image | mime_type: string, data: string (base64), stream_id: string | One image frame (screen share, camera, any visual input). All three are required; the SDKs default mime_type to image/jpeg and stream_id to video.input.default in their own signatures. stream_id labels concurrent video streams. |
activity-end | — | End-of-turn signal for manual-VAD turn-taking. Distinct from end (whole-session teardown). See Turn-taking. |
bind-input | — | Bind the agent's audio input to this client, sent after publishing audio. The server pins the sender's participant identity; a client can only bind its own input. Observers that publish no audio never send this. |
tool_job_result | job_id: string, tool_name: string, status: "completed" | "failed", result?: object, summary?: string, error?: string | Terminal result of a long-running (background) client tool, delivered off-band. summary / error are the model-facing text; result is structured data for logging. |
end | — | User ended the session. The server tears down the upstream session and closes. |
ping | — | Heartbeat; server replies pong. |
envelope-chunk | envelope_id: string, seq: int, total: int, data: string (base64) | Framing carrier for oversized client messages — see Envelope chunking. Emitted by the SDK, never directly. |
Server → client messages
Server messages group into lifecycle, transcript, tool, first-party, and chunking families.
Session lifecycle
These frames mark the start, interruption, and end of a session.
ready
Sent after the upstream session is established and the agent is ready.
| Field | Type | Description |
|---|---|---|
session_id | UUID | Server-assigned session id. Persist it and pass it back as experimental.resume_session_id on a fresh session-config to resume after a disconnect. |
agent | object | null | Resolved catalog-agent summary when the session referenced one; null for inline sessions. Informational only. tools[] is a flat list of strings mixing two vocabularies: each client tool's name, then each server tool's wire kind — so ["lookup_order", "web_search"], not the model-facing cosmo_web_search. |
rejected_tools | {name, reason}[] | Tools that were valid but are unavailable here — a server tool this deployment isn't configured for, or one gated off for this workspace. They are dropped and the session starts without them. A malformed tool spec never reaches this list; it rejects the whole start with 422 invalid_tool_config. |
max_session_seconds | int | null | Effective server-enforced duration cap; null = no cap. |
reconnecting
Server is transparently rotating the upstream model session. Transport and session state survive the swap.
| Field | Type | Description |
|---|---|---|
seconds_remaining | float | null | Optional ETA hint. |
session-ending-soon
The server ends this session shortly (for example, the max-duration cap). The session keeps running until session-ended.
| Field | Type | Description |
|---|---|---|
seconds_remaining | float | Time until cutoff. |
reason | string | Stable slug (for example, max_session_duration). |
session-ended
The server ended the session on purpose, published best-effort just before the transport tears down — the room close may still arrive first. Every SDK folds this into a guaranteed terminal session-ended item on its event stream.
| Field | Type | Description |
|---|---|---|
reason | string | Stable slug; same vocabulary as session-ending-soon.reason. |
error
A mid-session failure report. In practice the runtime emits upstream_disconnect and internal_error; the other enum values are reserved — see the code table for what each one signals and where its condition actually surfaces.
| Field | Type | Description |
|---|---|---|
code | ErrorCode | See error codes. |
message | string | Human-readable reason. |
fatal | bool | true: the session is dead — tear down and reconnect. false: this turn failed but the session continues. |
pong
Reply to ping. No fields.
Transcript and text
These frames carry what was said and what the model wrote.
transcript
Streaming transcript for either speaker. For the assistant in audio sessions this is the audio transcription — the words the listener actually heard.
| Field | Type | Description |
|---|---|---|
role | "USER" | "ASSISTANT" | Speaker. |
text | string | While is_final is false: the new fragment since the previous event (append). On is_final: true: the cumulative full transcript for the turn (replace). |
is_final | bool | Terminates the turn's stream. |
model-text
Streaming text-channel fragment from the model — distinct from transcript: text the model emits alongside its audio, not a transcription of it. In text-only sessions it carries the response; in audio sessions it may include narration that was never spoken.
| Field | Type | Description |
|---|---|---|
text | string | Fragment. |
is_final | bool | Defaults false. |
turn-complete
Marks the end of a turn so the client can finalize a transcript bubble.
| Field | Type | Description |
|---|---|---|
role | "USER" | "ASSISTANT" | Whose turn ended. |
Speech and processing markers
Field-less, informational — drive UI indicators, never mic gating.
type | When |
|---|---|
user-started-speaking / user-stopped-speaking | Server-side VAD detected user voice activity start / stop. |
bot-started-speaking / bot-stopped-speaking | First / last audio frame of the assistant turn left the server. |
bot-llm-started / bot-llm-stopped | Model began / finished generating (before audio is on the wire — useful for "thinking…" states). |
bot-tts-started / bot-tts-stopped | Assistant TTS audio frames started / stopped flowing. |
user-speech-timeout
A server-runtime silence hook fired; the server already performed action. Observability only.
| Field | Type | Description |
|---|---|---|
session_id | string | Session the hook fired in. |
silence_ms | int | Measured user silence. |
trigger_count | int | How many times this hook has fired. |
max_count | int | Configured cap. |
action | Say | EndCall | What the server did ({type: "say", text?, prompt?} or {type: "end_call", farewell?}). |
Tools
Server-executed tools surface as a three-event lifecycle sharing tool_call_id (the upstream's function-call id):
tool-call
Model decided to invoke a server-executed tool.
| Field | Type |
|---|---|
tool_call_id | string |
name | string |
tool-dispatch-started
Server-side handler began running — lets UIs show a "dispatching…" state for slow tools.
| Field | Type |
|---|---|
tool_call_id | string |
name | string |
tool-result
Server-side tool finished.
| Field | Type | Description |
|---|---|---|
tool_call_id | string | Correlates the triple. |
ok | bool | Success. |
summary | string | null | Short human-readable line. |
tool-invocation
Server asks the connected client to run a tool locally — sent only for tools declared with kind: "client" specs at session start. The result returns out of band (transport RPC), not as a data-channel send.
| Field | Type | Description |
|---|---|---|
request_id | string | Dispatch correlation id. |
tool_call_id | string | Upstream function-call id (correlates with the observability triple). |
name | string | Tool name. |
args | object | Decoded arguments. |
origin | "realtime" | "server" | Which producer authored the invocation (voice model vs. a server tool runtime). |
executable | bool | false marks an informational mirror whose execution runs out-of-band; true (default) means the recipient runs the tool. |
First-party cosmo.* events
These frames report Cosmo-specific telemetry rather than conversation content.
cosmo.usage
Cumulative token usage for the live session, split by direction and modality. All fields are ints defaulting to 0: input_text_tokens, input_image_tokens, input_audio_tokens, input_cached_tokens, output_text_tokens, output_audio_tokens, total_tokens.
cosmo.session-state
Live session state after a set_state tool write. Carries the full canonical state, not a delta. See Session state.
| Field | Type | Description |
|---|---|---|
state | object | Full canonical state after the merge. |
updated_keys | string[] | Keys touched by the write. |
warnings | string[] | Advisory schema findings the model also saw in its tool result. |
stage | string | null | Hoisted from state["stage"]. |
Chunked envelope
Messages whose JSON exceeds 12,000 bytes arrive split across these frames, 8,000 bytes at a time.
server-envelope-chunk
Mirror of envelope-chunk for the server→client direction: envelope_id, seq, total, data (base64). SDKs reassemble transparently and re-dispatch the inner message.
ErrorCode values
Every ErrorCode value, what it signals, and how its condition is delivered today. Only upstream_disconnect and internal_error currently arrive as error frames; the rest are reserved enum values whose conditions surface elsewhere.
| Value | Meaning | Delivery today |
|---|---|---|
upstream_disconnect | Upstream model session disconnected unexpectedly. | error frame. |
internal_error | Unrecognized server-side error. | error frame. |
auth_failed | Credential invalid or expired. | Reserved — a bad credential rejects the session start over HTTP (401) before a data channel exists. |
workspace_forbidden | Credential valid but workspace access denied. | Reserved — rejects the session start over HTTP (401/403). |
voice_disabled | Realtime voice isn't available for this session. | Reserved — voice unavailability rejects the session start with HTTP 503. |
invalid_message | Client sent a malformed or unknown control message. | Reserved — the runtime logs and drops an unparseable inbound message; no error frame is sent. |
version_mismatch | Client incompatible with the server. | Reserved — current servers don't send it; an older server may reject the session start over HTTP (400) with it. |
See Errors for the session-start rejection catalog and recovery guidance.
Envelope chunking
Every message must be valid JSON (UTF-8). Oversized messages are split into envelope-chunk / server-envelope-chunk fragments; data is base64 over the UTF-8 bytes of the inner message JSON, so byte-boundary splits never land mid-codepoint. The receiving side buffers by envelope_id, reassembles in seq order (0..total-1), and dispatches the inner message as if it arrived whole. See Envelope chunking.
An oversized send-image crossing the wire as three fragments, sharing one envelope_id:
{ "type": "envelope-chunk", "envelope_id": "d3adbeef-…", "seq": 0, "total": 3, "data": "eyJ0eXBlIjoic2VuZC1pbWFnZSIs…" }
{ "type": "envelope-chunk", "envelope_id": "d3adbeef-…", "seq": 1, "total": 3, "data": "…" }
{ "type": "envelope-chunk", "envelope_id": "d3adbeef-…", "seq": 2, "total": 3, "data": "…" }Concatenating the base64-decoded data fields in seq order yields the inner frame — here {"type": "send-image", "mime_type": "image/jpeg", "data": "…", "stream_id": "video.input.default"} — which the receiver dispatches as if it had arrived whole.