Video

The Cosmo agent can see. There are two ways to send visual input: RealtimeClientImage JSON frames over the control data channel, and LiveKit video tracks for continuous streams like screen share or camera.

What it is

RealtimeClientImage is a single image frame sent as base64-encoded bytes inside a JSON message on the data channel. Use it for one-shot captures: a photo, a screenshot at a point in time, or a frame sampled from a video stream.

LiveKit video tracks are continuous WebRTC video streams. Use them for real-time screen share or camera input where the agent needs to see motion or a live view.

The two paths are not mutually exclusive — you can have a screen share track active and also send periodic RealtimeClientImage frames.

JSON image frames

RealtimeClientImage fields:

The runtime translates this to an InboundImage and forwards it to the realtime model. How the model consumes it depends on the provider — see Provider differences below.

TypeScript — send a canvas snapshot:

const canvas = document.getElementById('preview') as HTMLCanvasElement;
const blob = await new Promise<Blob>((resolve) =>
  canvas.toBlob(b => resolve(b!), 'image/jpeg', 0.8)
);
const arrayBuffer = await blob.arrayBuffer();
const base64 = btoa(String.fromCharCode(...new Uint8Array(arrayBuffer)));

await client.sendImage({ data: base64, mimeType: 'image/jpeg', streamId: 'camera' });

Python — send a PIL image:

import base64
from io import BytesIO
from PIL import Image

img = Image.open("screenshot.png").convert("RGB")
buf = BytesIO()
img.save(buf, format="JPEG", quality=80)
b64 = base64.b64encode(buf.getvalue()).decode()

await client.send_image(
    data=b64,
    mime_type="image/jpeg",
    stream_id="screen_share_main",
)

Swift — send a captured image:

import CosmoRealtime

let jpeg: Data = /* your JPEG bytes */
let b64 = jpeg.base64EncodedString()
try await client.send(image: b64, mimeType: "image/jpeg", streamId: "screen_share_main")

LiveKit video tracks (screen share)

The TypeScript SDK exposes startScreenShare() and stopScreenShare() on RealtimeClient. These use navigator.mediaDevices.getDisplayMedia() and publish the resulting video track to the LiveKit room.

// Start screen share
await client.startScreenShare();

// Stop screen share
await client.stopScreenShare();

// Check state
const state = client.getScreenShareState();
// { kind: 'inactive' | 'requesting' | 'active' | 'error' }

The Swift SDK exposes the same surface, with one key difference: the caller drives the capture loop. Use ScreenCaptureKit (or any other source that produces CMSampleBuffers) and pipe frames into the SDK.

import CosmoRealtime
import ScreenCaptureKit

try await client.startScreenShare()

// In your SCStreamOutput callback:
func stream(_ stream: SCStream, didOutputSampleBuffer sampleBuffer: CMSampleBuffer, of type: SCStreamOutputType) {
    client.pushScreenShareFrame(sampleBuffer)
}

// Later:
await client.stopScreenShare()

startScreenShare() creates the LiveKit video track but defers the SFU publish until the first pushScreenShareFrame — LiveKit's BufferCapturer needs at least one frame to resolve dimensions before publishing. stopScreenShare() is idempotent.

The Python SDK mirrors Swift — caller-driven capture, deferred publish, idempotent stop. Frames are livekit.rtc.VideoFrames.

import livekit.rtc as rtc

await client.start_screen_share(width=1920, height=1080)

# In your capture callback, for each frame:
frame = rtc.VideoFrame(width=1920, height=1080, type=rtc.VideoBufferType.RGBA, data=raw_rgba_bytes)
await client.push_screen_share_frame(frame)

# Later:
await client.stop_screen_share()

The screen share state is also available via useMediaState():

import { useMediaState } from 'cosmo-ai';

function ScreenShareButton() {
  const client = useRealtimeClient();
  const media = useMediaState();
  const active = media.screen.kind === 'active';
  return (
    <button onClick={() => active ? client.stopScreenShare() : client.startScreenShare()}>
      {active ? 'Stop Screen Share' : 'Share Screen'}
    </button>
  );
}

Envelope chunking for large images

RealtimeClientImage frames can easily exceed the ~15 KiB data-channel limit. The SDK's envelope system automatically chunks them. A 200 KiB JPEG produces roughly 14 chunks that the server reassembles before passing to the model. This is transparent to your code.

For best latency, resize images before sending:

• JPEG at 80% quality
• Maximum 1280×720 px for screenshots
• Maximum 640×480 px for camera frames

Provider differences (Gemini vs OpenAI)

Cosmo's voice runtime is provider-neutral — a session runs on either Gemini Live or OpenAI GPT Realtime (see Sessions for how the provider is chosen). Visual input behaves differently on each:

This is transparent to your code: send RealtimeClientImage frames or publish a screen-share track the same way regardless of provider. The runtime applies the per-provider sampling gate — on OpenAI, frames you send faster than the cadence above are dropped before they reach the model, so you are never billed for frames the model would not have used.

Frame sampling is the standard way to do "video" with OpenAI Realtime — it is what the ChatGPT app itself does. It suits screen-share Q&A well (the screen is mostly static while the user talks) but it is not continuous video. Sessions that need true streaming video (meeting-bot participant cameras) stay on Gemini.

When to use each path

Pitfalls

• Sending frames faster than the provider's cadence does not improve agent perception — Gemini processes frames asynchronously, and on OpenAI sub-cadence frames are dropped by the sampling gate (see Provider differences). Sample at the source rather than flooding the channel.
• The stream_id field is informational. Use distinct values ("camera", "screen_share_main") for multi-stream sessions so logs are readable, but the model does not parse it.
• startScreenShare() requires transportState === 'ready'. Calling it before ready throws RealtimeNotReadyError.
• All three SDKs expose sendImage / send_image / send(image:) for one-shot JSON frames, and startScreenShare() / start_screen_share() for continuous LiveKit video tracks.
• Swift and Python defer the LiveKit publish until the first pushScreenShareFrame / push_screen_share_frame arrives, so you must keep feeding frames or the publish never completes. TypeScript publishes immediately via getDisplayMedia().
• Python's push_screen_share_frame() accepts a livekit.rtc.VideoFrame; supply it from your own capture loop (e.g. pyobjc ScreenCaptureKit bridge on macOS, mss + manual encoding elsewhere).

See also

• Transport — how the data channel and video tracks coexist in the LiveKit room
• Sessions — context_items for pre-session workspace file context (distinct from live video)
• Audio — the mic-side of the same LiveKit room