Transport

A Cosmo Realtime session runs on one of two carriers. webrtc, the default, splits the session across a LiveKit room for media and a reliable data channel inside that room for JSON control messages; websocket carries the whole session — audio and protocol — over a single socket against an open-source cosmo-server on your own machine. Both speak the same published protocol: ready, transcripts, tool events, and session-ended decode identically in every SDK, so application code does not change with the carrier.

Carrier selection

The client picks its carrier at construction — transport: 'websocket' in TypeScript, transport="websocket" in Python, transport: .websocket in Swift — and webrtc is the default everywhere (livekit remains a deprecated alias of webrtc). TypeScript on Node and Python also read the COSMO_TRANSPORT environment variable when the option is omitted.

Managed Cosmo serves the WebRTC carrier. The WebSocket carrier runs against a cosmo-server started with COSMO_TRANSPORT=websocket — the server serves one carrier per process, so a client asking for the lane it doesn't run is refused at session start. Each carrier's per-language surface is on the reference pages: TypeScript, Python, Swift.

The WebRTC carrier

The WebRTC carrier uses two separate channels: a LiveKit room for media and a reliable data channel inside that room for JSON control messages. The session handshake happens over REST before the room is joined.

Client                         Cosmo Server                 Realtime model
  │                                 │                             │
  │──POST /session/start ──────────►│                             │
  │◄── { livekit_url, token } ──────│                             │
  │                                 │                             │
  │══ LiveKit Room (WebRTC) ════════│                             │
  │   ├── RTP audio track           │──── model session (WS) ────►│
  │   │   (mic → server → TTS)      │◄─── audio + events ─────────│
  │   └── Reliable data channel     │                             │
  │       (JSON control messages)   │                             │
  │                                 │                             │
  ▼                                 ▼                             ▼

This is also the carrier that recovers: the transport retries a transient drop, and the server rotates the upstream model session under a live call — Reconnects covers both layers.

REST handshake

POST /api/v1/external/realtime/session/start is the only HTTP call the SDK makes per session. It carries the session-config payload and returns:

{
  "session_id": "abc123",
  "room_name": "cosmo-9f3c1ad84be7205c6d18e4b2",
  "livekit_url": "wss://livekit.example.com",
  "token": "eyJ..."
}

The token is a LiveKit JWT scoped to this room. The SDK passes it straight to room.connect().

LiveKit room

The LiveKit room carries two kinds of traffic:

Video tracks (screen share, camera) also ride the LiveKit room when active — see Video.

Reliable data channel

All JSON control messages travel over a single reliable, ordered SCTP data channel. LiveKit exposes this as room.localParticipant.publishData(payload, reliable: true) on the outbound side and the data_received room event on the inbound side.

The channel tops out around 16 KiB end to end, and the SDK keeps a margin under it: messages whose JSON exceeds 12,000 bytes are transparently split into server-envelope-chunk (server→client) or envelope-chunk (client→server) packets carrying 8,000 bytes each, and reassembled by the peer.

┌────────────────────────────────────────────────────────────┐
│ Large JSON message (e.g. a 40 KiB send-image frame)        │
│                                                            │
│  chunk 0: { type:"envelope-chunk", seq:0, total:3, data: }│
│  chunk 1: { type:"envelope-chunk", seq:1, total:3, data: }│
│  chunk 2: { type:"envelope-chunk", seq:2, total:3, data: }│
│                                                            │
│  → peer reassembles by envelope_id, re-dispatches          │
└────────────────────────────────────────────────────────────┘

You never need to handle chunking manually. The SDK does it for every outbound message.

Why the split?

Audio and control are on separate paths because their requirements differ:

• Audio needs low latency, jitter tolerance, and congestion control → RTP is the right fit.
• Control needs reliable delivery and ordering → SCTP with reliable: true.
• Chunked images need to ride the control channel so they arrive in-order with respect to other JSON messages — a separate media track would arrive out of order.

Keeping them in the same LiveKit room means NAT traversal, DTLS setup, and ICE negotiation happen once. The data channel piggybacks on the already-open DTLS association.

The WebSocket carrier

The WebSocket carrier is the one-process shape for local development: the same cosmo-server process serves the API, holds the provider session, and bridges it to the client, with no media server or worker between them. It binds loopback by default and is meant for a developer's own machine.

The handshake mirrors the room lane: the SDK sends the same session-config payload to POST /api/v1/external/realtime/session/ws-start, and the response names a socket URL and a single-use subprotocol instead of a room and a token. The SDK opens the WebSocket with that subprotocol, and everything else rides the one connection:

Turn detection is the provider's own — this server runs no detector of its own — and outbound audio is paced at roughly playback rate, so an interruption cuts off quickly instead of draining audio already parked in the client's buffer. Envelope chunking applies to text frames exactly as it does on the data channel.

The carrier trades resilience and media capability for that simplicity:

• A dropped socket ends the session. There is no reconnection layer: a network blip that closes the socket finishes the session, and the app starts a new one.
• The provider session's lifetime is the session's lifetime. The room carrier rotates the upstream model session under a live call; the socket carrier does not, so the upstream session reaching its boundary ends the call the same way.
• Video stays on the room carrier. Camera and screen-share tracks never ride the socket (Video), and background client tools are refused at session start. Single image frames (send-image) travel on either carrier.

Message flow (abbreviated)

Client → Server                    Server → Client
────────────────────────────────   ────────────────────────────
session-config (HTTP, not channel) ready
mute                               transcript (streaming)
send-text                          model-text (streaming)
send-image                         turn-complete
activity-end                       speaking / llm / tts markers
bind-input                         tool-call
tool_job_result                    tool-dispatch-started
ping                               tool-result
end                                tool-invocation
                                   reconnecting
                                   session-ending-soon
                                   session-ended
                                   error
                                   pong

The full map, including the first-party cosmo.* events, is in Events.

Pitfalls

• The SDK's automatic envelope chunking kicks in at 12,000 bytes. Images larger than a few megabytes produce many chunks and add latency — resize before sending.
• Readiness can be announced before your handler is attached — the transport is still settling as the agent comes up. You don't need to race it: agent.start() resolves at ready, and a handler or consumer attached afterwards is still delivered the event.
• Don't call room.localParticipant.publishData() directly. Use the SDK's send methods — they handle serialization, envelope chunking, and error logging.