Envelope chunking

The LiveKit reliable data channel has a practical per-packet ceiling of roughly 16 KiB end to end, imposed by SCTP, browser constraints, and the SFU layer. The SDK keeps a margin under it: a message whose JSON exceeds SAFE_PACKET_BYTES (12,000) is split into *-envelope-chunk frames carrying CHUNK_RAW_BYTES (8,000) of inner payload each, which the peer buffers and reassembles before dispatching. Both constants live in src/transport/envelope.ts and are mirrored in the Swift reassembler.

Callers emit a single logical message and receive a single logical message. The chunking layer is below the event surface.

When chunking fires

The most common trigger is an oversized image frame sent with sendImage(). The send-image message carries a base64-encoded JPEG directly in JSON; a 640×480 JPEG at moderate quality is roughly 30–80 KiB serialized — well above the threshold.

send-image payload (80 KiB) ÷ 8,000 bytes per chunk
  → chunk 0:  envelope-chunk { seq: 0,  total: 11, data: "..." }
  → chunk 1:  envelope-chunk { seq: 1,  total: 11, data: "..." }
  → …
  → chunk 10: envelope-chunk { seq: 10, total: 11, data: "..." }
                                           ↓ reassembled on the server
                                        single send-image

Text-only turns and control frames (mute, ping, transcript deltas, tool events) are almost always under 1 KiB and never trigger chunking in practice.

Wire frame schema

Client → server:

{
  "type": "envelope-chunk",
  "envelope_id": "<shared-uuid-across-all-chunks-in-this-envelope>",
  "seq": 0,
  "total": 6,
  "data": "<base64-encoded UTF-8 fragment>"
}

Server → client:

{
  "type": "server-envelope-chunk",
  "envelope_id": "<shared-uuid>",
  "seq": 0,
  "total": 3,
  "data": "<base64-encoded UTF-8 fragment>"
}

data is base64 over the UTF-8 bytes of the inner message JSON. Base64 keeps every chunk ASCII-safe — splitting raw UTF-8 at byte boundaries can land mid-codepoint and corrupt the decode on the receiver. The receiving side base64-decodes, concatenates, then JSON-parses to recover the original message.

How each SDK chunks and reassembles

RealtimeClient.sendImage()
  → transport.send(send-image message)
    → envelope.ts: chunk when the JSON exceeds SAFE_PACKET_BYTES
      → livekit_transport: publish each chunk

livekit_transport: raw data packet
  → envelope.ts: EnvelopeReassembler.consume()
    → if all chunks arrived: emit reassembled server message
      → RealtimeClient.handleServerMessage()

# Simplified from python/src/cosmo_ai/session/_engine.py in cosmo-ai
async def _handle_payload(self, payload: bytes | str) -> None:
    raw = json.loads(payload)
    if raw["type"] == "server-envelope-chunk":
        chunk = ServerEnvelope.model_validate(raw)
        reassembled = self._reassemble_envelope(chunk)
        if reassembled is not None:
            await self._handle_payload(reassembled)  # recurse once complete
        return
    # normal dispatch path: validate the typed event model and enqueue it
    event = _SERVER_EVENT_BY_TYPE[raw["type"]].model_validate(raw)
    self._emit(event)

// From Sources/CosmoRealtime/Session/RealtimeSession.swift in cosmo-swift-sdk
let result = await reassembler.consume(
    envelopeId: envelopeId,
    seq: seq,
    total: total,
    data: chunkData
)
switch result {
case .pending:
    break  // waiting for more chunks
case .complete(let assembled):
    await _receiveFrame(assembled)  // re-enter the normal dispatch path
case .invalid(let reason):
    // Surface on the forward-compatibility variant; never terminal.
    eventsContinuation.yield(.unknown(rawType: "server-envelope-chunk", payload: data))
}

Impact on image streaming

Chunking applies to sendImage(), not to video tracks. A ~60 KiB frame inflates to roughly 80 KiB once base64-encoded, so it splits into 11 chunks — the worked example above. Resize before encoding if you send stills on an interval — a 320×240 frame usually lands under the threshold and ships as a single packet. Sizing and format guidance for frames lives in Image input.

For continuous capture, prefer addVideoStream(): the track rides RTP, skips the data channel, and never chunks. Lowering fps there reduces bandwidth but has no bearing on envelope chunking.

// TypeScript — continuous camera track, no envelope chunking
const handle = await session.addVideoStream(cameraStream, {
  kind: 'camera',
  fps: 1,
});

# Python — sending a resized JPEG
import base64

# Resize to 320×240 before encoding to stay under the threshold.
data = base64.b64encode(resized_jpeg_bytes).decode()
await session.send_image(data=data, mime_type="image/jpeg", stream_id="camera")

send_image is the supported path — it applies the same chunking as any other oversized frame, so you never assemble envelopes yourself.

See also

• Build a voice React app — addVideoStream usage
• Image input — sizing and format guidance for one-shot frames
• Debugging — correlating chunked sessions in server logs