Realtime events
Every message on the data channel, grouped by direction and family — the protocol at a glance.
All control traffic is JSON on a reliable LiveKit data channel, discriminated by a type field. This page is the map; each SDK exposes the same events with language-native names (TypeScript session.on('transcript', …), Python RealtimeTranscriptDelta, Swift case .transcript).
Two framing rules apply to everything below:
- Oversized messages are chunked. Anything above the ~15 KiB data-channel limit travels as
envelope-chunk/server-envelope-chunkframes and is reassembled before your handler fires. You never see chunks. See Envelope chunking. - Unknown types are not errors. SDKs surface unrecognized
typevalues as an explicitunknownevent and keep going. New server events never break old clients.
Client → server
| Type | Purpose |
|---|---|
session-config | First frame after joining the room: protocol version + agent config + session params. The server replies with ready. |
mute | Toggle the server-side microphone gate. |
send-text | A text turn instead of audio — the agent answers it. |
send-context | Context the agent should have without being asked anything: no turn, no speech, no transcript entry. For live application state. |
send-image | One base64 image frame (mime_type, stream_id). See Image input. |
activity-end | Manual end-of-turn signal when you're running your own turn detection. See Turn-taking. |
bind-input | Bind the agent's audio input to this participant (the SDK sends it when you publish audio). |
tool_job_result | Deferred result of a background client tool (job_id, status, result). |
end | Graceful goodbye; the server tears down the upstream session. |
ping | Heartbeat; server replies pong. |
envelope-chunk | Framing carrier for oversized client messages. |
Server → client
Session lifecycle
| Type | Fired when | Payload highlights |
|---|---|---|
ready | Agent is live and listening | session_id, resolved agent summary, rejected_tools, max_session_seconds |
reconnecting | Server is rotating the upstream model session; a brief pause, not a failure | seconds_remaining |
session-ending-soon | Wall-clock cap approaching | seconds_remaining, reason |
session-ended | Server ended the session deliberately | reason (e.g. max_session_duration) |
error | Something went wrong | code, message, fatal — non-fatal errors do not end the session |
pong | Reply to ping | — |
Every SDK guarantees a terminal session-ended item as the last event on the stream, synthesized locally if the transport died before the server could say goodbye.
Transcripts and model text
| Type | Fired when | Payload highlights |
|---|---|---|
transcript | Speech transcribed, either speaker | role, text, is_final — streaming events append; the final event replaces the accumulated text |
model-text | The model emits text alongside (or instead of) audio | text, is_final |
turn-complete | A turn ended; finalize UI state | role |
Which one to display for which UI is covered in Transcripts.
Speech activity and model processing
These are informational, type-only markers — ideal for driving avatars, level meters, and "thinking…" indicators:
| Type | Meaning |
|---|---|
user-started-speaking / user-stopped-speaking | Server VAD detected user voice start/stop |
bot-started-speaking / bot-stopped-speaking | First/last audio frame of the assistant turn |
bot-llm-started / bot-llm-stopped | Model began/finished generating |
bot-tts-started / bot-tts-stopped | Speech synthesis began/finished |
user-speech-timeout | A server silence-timeout hook fired: silence_ms, trigger_count, max_count, and the action the server already took |
Tools
Four events cover both dispatch directions — see Tools:
| Type | Fired when |
|---|---|
tool-call | The model decided to invoke a tool |
tool-dispatch-started | The server-side handler began executing |
tool-result | The handler finished (ok, summary) |
tool-invocation | The server asks this client to run a local tool (request_id, name, args) |
The three observability events share a tool_call_id so you can render a timeline per invocation.
First-party extensions
Cosmo-specific events are namespaced cosmo.*:
| Type | Purpose |
|---|---|
cosmo.usage | Cumulative token usage, split by input/output and text/audio/image/cached |
cosmo.session-state | The durable session state after a cosmo.set_state write (state, updated_keys, stage) — see Session state |
Ordering guarantees
readyalways precedes transcripts, tool events, and speech markers.- The three tool observability events arrive in order for a given
tool_call_id, but events from different tool calls interleave. - Transcript deltas for one turn arrive in order;
turn-completefollows the final transcript of that turn. - The terminal session-ended item is always last; nothing follows it.
Wire reference
Exact schemas for every message live in the wire protocol reference. The protocol carries version (currently 1.0) on session-config and ready; see Protocol version for how breaking changes are signaled.