Screen tools

Screen share lets the agent see; the screen tools let it act: "click the export button", "highlight the field to fill in next". The split is strict — the server finds the element, your app touches it.

Four slots, added to an agent's tools like any other:

import {
  screenClickElementTool, screenHighlightElementTool, screenHighlightBoxTool,
  clicked, notClicked, landedOnControl, landedOnEstimate, notShown,
} from 'cosmo-ai/tool/screen';

const agent = client.agent({
  instructions: 'help the user drive their machine',
  tools: [
    screenLocateTool(request => grabScreen(request)),
    screenClickElementTool(({ element, action }) => {
      if (!canControlTheDesktop()) return notClicked('I need accessibility access');
      press(element.frame, action);
      return clicked;
    }),
  ],
});

from cosmo_ai.tools import (
    ScreenClickOutcome,
    ScreenClickTarget,
    screen_click_element_tool,
    screen_locate_tool,
)


def on_click(target: ScreenClickTarget) -> ScreenClickOutcome:   # sync or async
    if not can_control_the_desktop():
        return ScreenClickOutcome(clicked=False, reason="I need accessibility access")
    press(target.element.frame, target.action)      # button, single or double
    return ScreenClickOutcome(clicked=True)


agent = client.agent(
    instructions="help the user drive their machine",
    tools=[screen_locate_tool(grab_screen), screen_click_element_tool(on_click)],
)

• screenLocateTool(capture) is a server-tool opt-in that carries configuration, because unlike the zero-config ones the locator has to be told how to see. capture returns a ScreenCapture: a JPEG of the current screen plus a list of ScreenElements — indexed regions with a role, a frame, and optional title / label / value. Build it from whatever your platform offers: the accessibility tree, your own component registry, a DOM walk. Declaring it is what asks for the locator, and it is the one entry here the model never calls — the server does, mid-cosmo_screen_locate.
• screenClickElementTool(onClick) receives a ScreenClickRequest — the resolved element, the capture it came from, and the action (button, single or double). Return clicked or notClicked(reason).
• screenHighlightElementTool(onHighlight) highlights a located element instead of touching it — a coach-mark, a tour overlay. interaction says which gesture to suggest (pointer, click, double_click, drag_show, …). Return landedOnControl, or notShown(reason).
• screenHighlightBoxTool(onHighlight) highlights a box the model located itself, as fractions of the shared surface. No capture and no lookup, so it draws immediately, which makes it the default way to point. Return landedOnControl when you snapped onto a real control, landedOnEstimate when all you had was the model's box.

Both highlights answer in one shape, { shown, exact }, so the model runs the same check either way. exact: false is what sends it to the locator — and from a located element the answer is always landedOnControl, since the grounder picked it out of a real accessibility list.

Your capture handler receives a ScreenCaptureRequest; it carries no options today and exists so future capture options can arrive without changing your handler's signature. Captures serve cosmo_screen_locate alone — the vision locators read the session's own recent frames (a live track, or a sent ClientImage), never this handler's captures.

Walk through the flow

1. The model calls cosmo_screen_locate with a description ("the blue Export button").
2. The server calls your capture and receives the screenshot + element list.
3. A vision model resolves the description against both and returns candidates to the model, each carrying an opaque found_element handle.
4. The model picks one — using the conversation, which the grounder cannot see — and passes that handle to cosmo_sdk_screen_click_element or cosmo_sdk_screen_highlight_element.
5. The SDK resolves the handle back to the element it names and calls your handler with it.

Your app never parses model output and never does visual matching. It executes a concrete element plus an action, which keeps deciding what to touch on the server and actually touching it in code you wrote.

A handle names one element of one capture, so it means nothing beside any other, and there are no fields in it to edit. If the screen moved on and the capture expired — or the model assembled a handle rather than passing one back — the SDK declines with a reason the agent can say out loud rather than touching the wrong thing.

Guard real actions

screenClickElementTool performs real actions, so treat it like any other privileged surface:

• Only include elements in capture that the agent may legitimately touch — the element list is an allowlist by construction. Omit destructive controls, or return notClicked(...) for them.
• Prefer screenHighlightElementTool plus a human click for anything irreversible.
• Clicking is gated by workspace policy on top of whatever you declare: a session without desktop control starts without cosmo_sdk_screen_click_element and reports the drop on ready.rejected_tools.
• Every call lands in the tool event stream, so the session timeline records what the agent touched and when.

Available in all three SDKs. The spelling follows each language — TypeScript's screenClickElementTool / screenHighlightElementTool / screenHighlightBoxTool with the screenLocateTool(capture) opt-in; Swift's .screenLocateTool(capture:) and its ScreenClickTool / ScreenHighlightTool / ScreenHighlightBoxTool; Python's screen_locate_tool(capture) / screen_click_element_tool / screen_highlight_element_tool / screen_highlight_box_tool in cosmo_ai.tools — but the names on the wire, the model-facing text, and the decode and reply shapes are identical, pinned by the shared client-tool vectors. Supplying the capture handler is the host's job in every one; the SDK owns everything either side of it.