Transport

Cosmo Realtime uses two separate channels: a LiveKit room for media and a reliable data channel inside that room for JSON control messages. The session handshake happens over REST before the room is joined.

Architecture

Client                         Cosmo Server                 Realtime model
  │                                 │                             │
  │──POST /session/start ──────────►│                             │
  │◄── { livekit_url, token } ──────│                             │
  │                                 │                             │
  │══ LiveKit Room (WebRTC) ════════│                             │
  │   ├── RTP audio track           │──── model session (WS) ────►│
  │   │   (mic → server → TTS)      │◄─── audio + events ─────────│
  │   └── Reliable data channel     │                             │
  │       (JSON control messages)   │                             │
  │                                 │                             │
  ▼                                 ▼                             ▼

REST handshake

POST /api/v1/external/realtime/session/start is the only HTTP call the SDK makes per session. It carries the session-config payload and returns:

{
  "session_id": "abc123",
  "room_name": "cosmo-room-xyz",
  "livekit_url": "wss://livekit.example.com",
  "token": "eyJ..."
}

The token is a LiveKit JWT scoped to this room. The SDK passes it straight to room.connect().

LiveKit room

The LiveKit room carries two kinds of traffic:

Video tracks (screen share, camera) also ride the LiveKit room when active — see Video.

Reliable data channel

All JSON control messages travel over a single reliable, ordered SCTP data channel. LiveKit exposes this as room.localParticipant.publishData(payload, reliable: true) on the outbound side and the data_received room event on the inbound side.

The channel has a practical message size limit around 15 KiB. Messages larger than this threshold are transparently chunked into server-envelope-chunk (server→client) or envelope-chunk (client→server) packets and reassembled by the peer.

┌────────────────────────────────────────────────────────────┐
│ Large JSON message (e.g. a 40 KiB send-image frame)        │
│                                                            │
│  chunk 0: { type:"envelope-chunk", seq:0, total:3, data: }│
│  chunk 1: { type:"envelope-chunk", seq:1, total:3, data: }│
│  chunk 2: { type:"envelope-chunk", seq:2, total:3, data: }│
│                                                            │
│  → peer reassembles by envelope_id, re-dispatches          │
└────────────────────────────────────────────────────────────┘

You never need to handle chunking manually. The SDK does it for every outbound message.

Message flow (abbreviated)

Client → Server                    Server → Client
────────────────────────────────   ────────────────────────────
session-config (HTTP, not channel) ready
mute                               transcript (streaming)
send-text                          model-text (streaming)
send-image                         turn-complete
activity-end                       speaking / llm / tts markers
bind-input                         tool-call
tool_job_result                    tool-dispatch-started
ping                               tool-result
end                                tool-invocation
                                   reconnecting
                                   session-ending-soon
                                   session-ended
                                   error
                                   pong

The full map, including the first-party cosmo.* events, is in Events.

Why the split?

Audio and control are on separate paths because their requirements differ:

• Audio needs low latency, jitter tolerance, and congestion control → RTP is the right fit.
• Control needs reliable delivery and ordering → SCTP with reliable: true.
• Chunked images need to ride the control channel so they arrive in-order with respect to other JSON messages — a separate media track would arrive out of order.

Keeping them in the same LiveKit room means NAT traversal, DTLS setup, and ICE negotiation happen once. The data channel piggybacks on the already-open DTLS association.

Pitfalls

• The SDK's automatic envelope chunking kicks in at ~15 KiB. Images larger than a few megabytes will produce many chunks and add latency — resize before sending.
• The data channel does not open instantaneously after the room join. The SDK defers ready dispatch until the channel is open, so handlers registered right after agent.start() still receive it.
• Do not call room.localParticipant.publishData() directly. Use the SDK's send methods — they handle serialization, envelope chunking, and error logging.

See also

• Sessions — what the handshake returns
• Video — screen share and camera over LiveKit video tracks
• Lifecycle — the event sequence over the data channel