Screen interaction

Screen share lets the agent see; screen interaction lets it act: "click the export button", "highlight the field the user should fill in next". The division of labor is strict — your app implements four platform primitives, and the server does the hard part (grounding the model's natural-language target to a concrete element with a vision model).

The client side is a four-method contract:

interface ScreenInteraction {
  capture(): Promise<ScreenCapture>;
  activate(element: ScreenElement, capture: ScreenCapture, action: ScreenAction): Promise<boolean>;
  highlightElement(
    element: ScreenElement,
    capture: ScreenCapture,
    opts: { label: string; placement: ScreenPlacement; affordance: ScreenAffordance },
  ): Promise<boolean | ScreenHighlightResult>;
  highlightRegion(
    region: ScreenRegion,
    opts: { element?: ScreenElementHint; label: string; placement: ScreenPlacement; affordance: ScreenAffordance },
  ): Promise<boolean | ScreenHighlightResult>;
}

• capture() returns a ScreenCapture: a JPEG of the current screen plus a list of ScreenElements — indexed regions with a role, a frame (x, y, w, h), and optional title / label / value. Build the element list from whatever your platform offers: the accessibility tree, your own component registry, or a DOM walk.
• activate(element, capture, action) performs a real interaction on one grounded element, from the same capture it was chosen in.
• highlightElement(element, capture, …) marks or points at a grounded element instead of acting — a learning spotlight, a coach-mark, a tour overlay. The affordance says what gesture the mark should suggest (pointer, click, double_click, drag_show, …).
• highlightRegion(region, …) marks a region the caller located itself — no capture, no grounding round trip, so it draws immediately. Marking may skip the freshness check because a stale marker is harmless; a stale click is not, which is why acting always goes through a grounded capture.

Return true when done, false to decline (the model will try a different approach), and reject only for unexpected errors. The highlight methods may instead return a ScreenHighlightResult reporting whether the mark actually landed on a real control — the signal the caller needs to know when to re-target.

The flow

1. The model decides to interact with the screen and calls the server's screen tool.
2. The server calls your capture() and receives the screenshot + element list.
3. A vision model grounds the request ("the blue Export button") to a specific ScreenElement.
4. The server calls your activate(...) or highlightElement(...) with that element (or highlightRegion(...) when it already knows the coordinates).
5. The result flows back to the model, which narrates or continues.

Your app never parses model output or does visual matching — you only ever execute a concrete element + action, which keeps the dangerous part (deciding what to touch) server-side and the trusted part (actually touching it) in code you wrote.

Opting in

Pass your implementation when building the client:

const client = new RealtimeClient({ token, screenInteraction: myScreenInteraction });

Available in TypeScript today.

Safety

activate performs real actions in your app, so treat it like any other privileged tool surface:

• Only include elements in capture() that the agent may legitimately touch — the element list is an allowlist by construction. Omit destructive controls entirely, or return false from activate for them.
• Prefer highlightElement + a human click for anything irreversible.
• Actions land in the tool event stream like every other dispatch, so your session timeline records what the agent touched and when.