Screen share

Screen share turns the agent into an over-the-shoulder copilot: it watches the live screen and talks about what it sees — walking a user through an unfamiliar app, reviewing a document together, debugging alongside a developer.

The screen travels as a LiveKit video track (like camera video), not as control-channel messages. The browser or OS handles the capture-permission prompt.

TypeScript (browser)

await session.startScreenShare();   // browser shows the picker
// … the agent now sees the shared surface …
await session.stopScreenShare();

Track the lifecycle for your UI:

const state = client.getScreenShareState();  // 'inactive' | 'requesting' | 'active' | 'error'

A denied picker surfaces as a screen_denied error event — treat it as a normal user choice, not a failure. getVisionInputStatus() tells you whether the model has a fresh frame, which is the honest signal for a "👁 agent can see your screen" indicator.

Swift (macOS / iOS)

Swift splits capture from transport, because on Apple platforms you own the capture pipeline (ScreenCaptureKit on macOS, ReplayKit on iOS):

try await session.startScreenShare()          // create the track

// From your capture callback (any thread):
session.pushScreenShareFrame(sampleBuffer)    // nonisolated, safe from capture threads

await session.stopScreenShare()

The track publishes on the first frame you push, not at startScreenShare() — so a capture pipeline that never produces frames never publishes an empty track. Two hooks matter in production:

• setScreenShareFrameProcessor(_:) — transform frames before they leave the device (redaction, watermarking, cropping a region).
• onScreenShareFailed(_:) — observe SFU rejections or codec failures without tearing down the session.

Python

The Python SDK has no screen-capture arm (server processes rarely have a screen). Send periodic image frames of whatever surface you render, or run screen share from a browser/Swift client in the same session.

Redaction and consent

Screen content is the most sensitive input modality — it can contain anything the user has open.

• Capture the narrowest surface the task needs: a window or tab rather than the whole display (the browser picker offers this; on macOS, filter in your ScreenCaptureKit config).
• Redact before transport where you control frames (Swift's frame processor); in the browser, prefer sharing a specific tab.
• Say clearly in your UI when the screen is visible to the agent, and make stopping one tap. store_recording: false keeps shared screens out of persisted artifacts — see Recording & privacy.

For the agent to not just see but point at and click things on screen, see Screen interaction.