Image input

The lightest way to give the agent eyes is one frame at a time: a photo the user picked, a canvas snapshot, a periodic webcam still. Images travel as base64 JSON messages on the control channel — no video track, no extra permissions beyond what your app already captured.

await session.sendImage({ data: base64Jpeg, mimeType: 'image/jpeg' });
await session.sendText('What am I looking at?');

await session.send_image(data=base64_jpeg, mime_type="image/jpeg")
await session.send_text("What am I looking at?")

try await session.send(image: base64Jpeg, mimeType: "image/jpeg")
try await session.send(text: "What am I looking at?")

The frame lands in the model's context; you can reference it in the same turn ("what's this?") or later ones ("compare it with the label I showed you earlier").

Choose stream ids

Every frame carries a stream_id (default "video.input.default"). Frames with the same id replace each other as "the current view" of that stream; distinct ids let you keep separate visual channels — say, a document camera and a product photo — individually addressable.

Sending stills on an interval through one stream_id is a perfectly good poor-man's video: 1 frame every 1–2 seconds is enough for "watch what I'm doing" experiences at a fraction of the bandwidth of a track. When you need real motion or the platform is already producing a MediaStream, use a video track instead.

Keep frames small

Downscale before sending — 1280×720 for screenshots, 640×480 for camera frames, the same ceilings video streaming uses. Past them you are paying for pixels the model never sees.

Both providers normalize a frame before tokenizing it, so resolution beyond their working size buys no comprehension:

• Gemini Live spends a fixed token budget per frame, set by the session's media resolution rather than by the frame's pixel count. A larger frame does not cost more tokens, and does not show the model more.
• OpenAI's tile-based vision path fits the image into 2048×2048, then scales the shortest side to 768 — a working long edge near 1200px on a 16:10 screen. Everything beyond is discarded before tokenization.

A 1280px long edge sits just above the point where extra pixels stop doing anything on either. What overshooting costs is bytes, not tokens: a 1920px screen capture is roughly 1.8× the payload of the same frame at 1280, and every frame over 12 KB is split across multiple envelope chunks — a 320×240 frame fits a single packet with none — so oversized stills add visible latency before the model can react.

• Compress before sending — JPEG around 70–80% quality is plenty for scene understanding.
• Raise the ceiling only for a task that measurably needs the detail, such as dense OCR or fine-grained UI grounding.

In Swift, pass the CGImage and the SDK bounds it for you — this is the preferred call, since it downscales before the pixels are ever encoded or base64-inflated:

try await session.send(image: cgImage)

ImageDownscale.recommendedMaxLongEdge (1280) and ImageDownscale.recommendedQuality (0.8) are the defaults; both are parameters when a task needs something else. ImageDownscale.encodeJPEG(image:maxLongEdge:quality:) is public if you want the bounded base64 without sending it.

The base64 overloads cannot downscale before encoding, so an oversized payload is decoded and re-encoded at the recommended bound, and the re-encode is logged. A payload that exceeds the server's ingress limit and cannot be downscaled is rejected with its measured size.

Examine full resolution on demand

Downscaling caps what every frame costs; the examine_image server tool removes the reason to overshoot the cap. With it granted, the model examines the freshest published frame at full resolution — but only when it decides the moment needs the detail. You stream small, cheap frames continuously and pay high-resolution prices on the handful of turns that earn it — the dense document finally held up to the camera, the one control the question is about — instead of on every frame of the call.

Grant it when the agent answers questions about what is on screen; the model should reach for it only when a turn needs the detail — small text, or what the user is hovering over — because each call re-reads the frame at full resolution and adds a few seconds before the answer.

const agent = client.agent({ tools: [examineImageTool()] });

from cosmo_ai import examine_image_tool

agent = client.agent(instructions="…", tools=[examine_image_tool()])

let agent = try client.agent(instructions: "…", tools: [.examineImageTool()])

detect_objects and point_at_object follow the same on-demand shape for grounding: they locate a named object in the frame server-side, returning boxes or points the model can hand to the draw tools to put the answer on the user's screen.

Set the retention policy

Frames become part of the session's model context, and — like audio and transcripts — are persisted only if the session records. Set store_video: false for sessions whose imagery must not be retained but whose conversation should still record, or store_recording: false to retain nothing at all; see Recording and privacy.