Envelope chunking

The LiveKit reliable data channel has a practical per-packet cap of approximately 15 KiB (imposed by SCTP + browser constraints + the SFU layer). The SDK handles this transparently: any message whose JSON exceeds the threshold is wrapped into a sequence of *-envelope-chunk frames that the peer buffers and reassembles before dispatching.

Callers emit a single logical message and receive a single logical message. The chunking layer is below the event surface.

When chunking fires

The most common trigger is an oversized image frame. The send-image message carries a base64-encoded JPEG directly in JSON; a 640×480 JPEG at moderate quality is roughly 30–80 KiB serialized — well above the threshold.

send-image payload (80 KiB)
  → chunk 0: envelope-chunk { seq: 0, total: 6, data: "..." }
  → chunk 1: envelope-chunk { seq: 1, total: 6, data: "..." }
  → chunk 2: envelope-chunk { seq: 2, total: 6, data: "..." }
  → chunk 3: envelope-chunk { seq: 3, total: 6, data: "..." }
  → chunk 4: envelope-chunk { seq: 4, total: 6, data: "..." }
  → chunk 5: envelope-chunk { seq: 5, total: 6, data: "..." }
                                           ↓ reassembled on the server
                                        single send-image

Text-only turns and control frames (mute, ping, transcript deltas, tool events) are almost always under 1 KiB and never trigger chunking in practice.

Wire frame schema

Client → server:

{
  "type": "envelope-chunk",
  "id": "<message-uuid>",
  "envelope_id": "<shared-uuid-across-all-chunks-in-this-envelope>",
  "seq": 0,
  "total": 6,
  "data": "<base64-encoded UTF-8 fragment>"
}

Server → client:

{
  "type": "server-envelope-chunk",
  "id": "<message-uuid>",
  "envelope_id": "<shared-uuid>",
  "seq": 0,
  "total": 3,
  "data": "<base64-encoded UTF-8 fragment>"
}

data is base64 over the UTF-8 bytes of the inner message JSON. Base64 keeps every chunk ASCII-safe — splitting raw UTF-8 at byte boundaries can land mid-codepoint and corrupt the decode on the receiver. The receiving side base64-decodes, concatenates, then JSON-parses to recover the original message.

TypeScript SDK

Chunking is handled in transport/envelope.ts. Outbound messages are serialized and split if needed before being published to the LiveKit data channel. Inbound server-envelope-chunk frames are buffered in an EnvelopeReassembler instance keyed by envelope_id.

RealtimeClient.addVideoStream()
  → transport.send(send-image message)
    → envelope.ts: chunk if > 15 KiB
      → livekit_transport: publish each chunk

On the inbound side:

livekit_transport: raw data packet
  → envelope.ts: EnvelopeReassembler.consume()
    → if all chunks arrived: emit reassembled server message
      → RealtimeClient.handleServerMessage()

Callers never see envelope-chunk in their on() handlers.

Python SDK

RealtimeSession._handle_payload() checks the type field before dispatching. On server-envelope-chunk, it calls _reassemble_envelope() which buffers chunks in self._envelope_buffers keyed by envelope_id and calls _handle_payload() recursively once all seq slots are filled.

# Simplified from sdks/cosmo-realtime/python/src/cosmo_ai/session/_engine.py
async def _handle_payload(self, payload: bytes | str) -> None:
    raw = json.loads(payload)
    if raw["type"] == "server-envelope-chunk":
        chunk = RealtimeServerEnvelope.model_validate(raw)
        reassembled = self._reassemble_envelope(chunk)
        if reassembled is not None:
            await self._handle_payload(reassembled)  # recurse once complete
        return
    # normal dispatch path: validate the typed event model and enqueue it
    event = _SERVER_EVENT_BY_TYPE[raw["type"]].model_validate(raw)
    self._emit(event)

The buffers are bounded so a misbehaving server cannot grow them without limit: at most 64 in-flight envelopes (a new envelope evicts the oldest), at most 1024 chunks per envelope, and at most 4 MiB accumulated per envelope — an envelope over any cap is dropped whole. A chunk that fails base64 validation surfaces as an UnknownEvent and never kills the stream.

Swift SDK

The Swift SDK uses an actor-isolated EnvelopeReassembler that enforces:

• Maximum 8 in-flight envelopes at a time.
• Maximum 4 MiB total across all in-flight envelopes.
• 30-second TTL — stale envelopes are swept on each new chunk arrival.

// From sdks/cosmo-realtime/swift/Sources/CosmoRealtime/Session/RealtimeSession.swift
let result = await reassembler.consume(
    envelopeId: envelopeId,
    seq: seq,
    total: total,
    data: chunkData
)
switch result {
case .pending:
    break  // waiting for more chunks
case .complete(let assembled):
    await _receiveFrame(assembled)  // re-enter the normal dispatch path
case .invalid(let reason):
    // Surface on the forward-compatibility variant; never terminal.
    eventsContinuation.yield(.unknown(rawType: "server-envelope-chunk", payload: data))
}

Invalid envelopes (bad base64, seq/total mismatch, byte cap exceeded) are logged, surfaced as .unknown events, and discarded without crashing the session.

Impact on image streaming

If you stream camera frames at high frequency and large size, you will generate many envelope chunks. For a 640×480 camera stream at 1 fps with ~60 KiB frames, expect roughly 4–5 chunks per frame. Reduce frame size or lower fps in addVideoStream options to stay comfortably under the threshold:

// TypeScript — low-bandwidth camera stream
const handle = await session.addVideoStream(cameraStream, {
  kind: 'camera',
  fps: 1,          // 1 frame per second
});

# Python — sending a resized JPEG
import base64

# Resize to 320×240 before encoding to stay under the threshold.
data = base64.b64encode(resized_jpeg_bytes).decode()
await session.send_image(data=data, mime_type="image/jpeg", stream_id="camera")

send_image is the supported path — it applies the same chunking as any other oversized frame, so you never assemble envelopes yourself.

Next steps

• Build a voice React app — addVideoStream usage
• Coach example — high-frequency camera streaming pattern
• Debugging — correlating chunked sessions in server logs