Video

The Cosmo agent can see. You can send visual input two ways: ClientImage JSON frames over the control data channel, and LiveKit video tracks for continuous streams like screen share or camera.

Frames and streams

ClientImage is a single image frame sent as base64-encoded bytes inside a JSON message on the data channel. Use it for one-shot captures: a photo, a screenshot at a point in time, or a frame sampled from a video stream.

LiveKit video tracks are continuous WebRTC video streams. Use them for real-time screen share or camera input where the agent needs to see motion or a live view.

The two paths aren't mutually exclusive — you can have a screen share track active and also send periodic ClientImage frames.

JSON image frames

ClientImage fields:

The runtime forwards the frame to the realtime model. How the model consumes it depends on the provider — Gemini accepts native realtime frames, OpenAI takes rate-limited conversation items — see Provider differences below.

const canvas = document.getElementById('preview') as HTMLCanvasElement;
const blob = await new Promise<Blob>((resolve) =>
  canvas.toBlob(b => resolve(b!), 'image/jpeg', 0.8)
);
const arrayBuffer = await blob.arrayBuffer();
const base64 = btoa(String.fromCharCode(...new Uint8Array(arrayBuffer)));

await client.sendImage({ data: base64, mimeType: 'image/jpeg', streamId: 'camera' });

import base64
from io import BytesIO
from PIL import Image

img = Image.open("screenshot.png").convert("RGB")
buf = BytesIO()
img.save(buf, format="JPEG", quality=80)
b64 = base64.b64encode(buf.getvalue()).decode()

await client.send_image(
    data=b64,
    mime_type="image/jpeg",
    stream_id="screen_share_main",
)

import CosmoRealtime

let jpeg: Data = /* your JPEG bytes */
let b64 = jpeg.base64EncodedString()
try await session.send(image: b64, mimeType: "image/jpeg", streamId: "screen_share_main")

LiveKit video tracks

A video track is a continuous WebRTC stream, published once and left running. Two APIs open one:

• addVideoStream(stream, options) publishes a camera or canvas stream you already hold, at the frame rate you ask for.
• startScreenShare() prompts for a display surface and publishes that.

Both ride RTP rather than the data channel, so neither is envelope-chunked and neither is affected by the message size threshold. The screen-share half — per-language setup, the deferred publish on the first pushed frame, and the redaction and consent questions that come with capturing a whole display — lives on Screen share.

Video tracks need the WebRTC transport: on the websocket transport, starting a video or screen-share publish refuses with video_unsupported (stopping one stays a harmless no-op) — one-shot ClientImage frames travel on either transport.

Envelope chunking for large images

ClientImage frames easily exceed the 12,000-byte chunking threshold. The SDK's envelope system splits them into 8,000-byte chunks automatically — a ~60 KiB JPEG base64-expands to roughly 80 KiB and splits into 11 chunks (the worked example in Envelope chunking) — and the server reassembles them before passing the frame to the model. This is transparent to your code. Video tracks don't go through this path — they ride RTP and are never chunked.

For best latency, resize images before sending:

• JPEG at 80% quality
• Maximum 1280×720 px for screenshots
• Maximum 640×480 px for camera frames

Provider differences

Cosmo's voice runtime is provider-neutral — a session runs on one of five providers (gemini, openai, openai_mini, openai_live, grok), selected by the agent's model; leave it unset and the workspace default applies. Visual input behaves differently across them.

The two families that accept visual input:

A session can also resolve to a voice-only model — Grok, OpenAI GPT Live (openai_live), or any voice-only model reached through the plain model string, a workspace default, or a catalog agent's stored config — that supports no vision at all. Visual input is then discarded before the model rather than erroring the session, so don't build a visual flow on an agent pinned to one.

This is transparent to your code: send ClientImage frames or publish a screen-share track the same way regardless of provider. The runtime applies the per-provider sampling gate — on OpenAI, frames you send faster than the cadence above are dropped before they reach the model, so you're never billed for frames the model would not have used.

Frame sampling is the standard way to do "video" with OpenAI Realtime — it's what the ChatGPT app itself does. It suits screen-share Q&A well (the screen is mostly static while the user talks) but it's not continuous video. Sessions that need true streaming video (meeting-bot participant cameras) stay on Gemini.

When to use each path

The following table maps each visual-input scenario to the path that fits it.

Frames for the vision locators

detect_objects and point_at_object never take an image — your job is to make sure the session has a recent frame to give them. Both supply paths above count, and you can use either or both:

Stream it. Any live camera or screen-share track works. The freshest frame is already on the server when the call arrives.

Or send it. A ClientImage sent with sendImage feeds the locators too — a recent one is preferred over the track frame. Periodic screenshots on a setInterval cadence (the pattern from the table above) keep the locators fed without holding a track open.

Frames expire: one older than 15 seconds no longer counts, whichever way it arrived. The screen tools' capture handler never feeds the locators — captures belong to cosmo_screen_locate alone, so a session that declares screenLocateTool(…) but supplies no frames still has none here. With no recent frame the locators return status: "unavailable", with a summary telling the model to ask the user to share their screen or turn on their camera. They never answer from memory — or from an expired frame.

Pitfalls

• Sending frames faster than the provider's cadence doesn't improve agent perception — Gemini processes frames asynchronously, and on OpenAI sub-cadence frames are dropped by the sampling gate (see Provider differences). Sample at the source rather than flooding the channel.
• The stream_id field is informational. Use distinct values ("camera", "screen_share_main") for multi-stream sessions so logs are readable, but the model doesn't parse it.
• startScreenShare() requires a live session. It is usable the moment agent.start() resolves; calling it after the session ends throws SessionStateError with code not_connected.
• All three SDKs expose sendImage / send_image / send(image:) for one-shot JSON frames, and startScreenShare() / start_screen_share() for continuous LiveKit video tracks.
• Swift and Python defer the LiveKit publish until the first pushScreenShareFrame / push_screen_share_frame arrives, so you must keep feeding frames or the publish never completes. TypeScript publishes immediately through getDisplayMedia().
• Python's push_screen_share_frame() accepts a livekit.rtc.VideoFrame; supply it from your own capture loop (for example, pyobjc ScreenCaptureKit bridge on macOS, mss + manual encoding elsewhere).
• Python and Swift publish one video track at a time, so a camera stream and a screen share cannot run together — starting either while the other is live is refused rather than silently replacing it, with SessionStateError code video_publish_already_active. TypeScript carries several: its transport keys publications by stream_id, and a screen share alongside a camera is a supported arrangement there — getVisionInputStatus() reports both sources as live. The code is declared in all three so a branch ports unchanged; only Python and Swift raise it.
• Python's add_video_stream() returns a VideoStreamHandle: stream.push(frame) publishes, remove_video_stream(stream) stops. It takes the same livekit.rtc.VideoFrame as the screen-share path, from whatever capture loop you already have — the SDK opens no camera. push is safe to call off the event loop, so the handle can be driven straight from a capture thread. Both VideoStreamHandle and SessionStateError import from cosmo_ai.