Image input

The lightest way to give the agent eyes is one frame at a time: a photo the user picked, a canvas snapshot, a periodic webcam still. Images travel as base64 JSON messages on the control channel — no video track, no extra permissions beyond what your app already captured.

await session.sendImage({ data: base64Jpeg, mimeType: 'image/jpeg' });
await session.sendText('What am I looking at?');

await session.send_image(data=base64_jpeg, mime_type="image/jpeg")
await session.send_text("What am I looking at?")

try await session.send(image: base64Jpeg, mimeType: "image/jpeg")
try await session.send(text: "What am I looking at?")

The frame lands in the model's context; you can reference it in the same turn ("what's this?") or later ones ("compare it with the label I showed you earlier").

Streams

Every frame carries a stream_id (default "video.input.default"). Frames with the same id replace each other as "the current view" of that stream; distinct ids let you keep separate visual channels — say, a document camera and a product photo — individually addressable.

Sending stills on an interval through one stream_id is a perfectly good poor-man's video: 1 frame every 1–2 seconds is enough for "watch what I'm doing" experiences at a fraction of the bandwidth of a track. When you need real motion or the platform is already producing a MediaStream, use a video track instead.

Resolution and size

• Compress before sending — JPEG around 70–80% quality is plenty for scene understanding.
• Oversized frames are handled automatically by envelope chunking, but a multi-megabyte still adds visible latency before the model can react. Downscale to ≤1080p unless the task needs fine detail.
• For fine detail on demand, opt into the examine_image server tool (ExamineImageTool() in Python, { kind: 'examine_image' } in TypeScript, .examineImage in Swift): the model examines the freshest published frame at full resolution only when it decides it needs to, instead of you paying for high-res on every frame. The detect_objects and point_at_object kinds locate named objects in the frame, returning boxes or points.

Privacy

Frames become part of the session's model context, and — like audio and transcripts — are persisted only if the session records. Set store_recording: false for sessions whose imagery must not be retained; see Recording & privacy.