Tools

Tools are how the agent acts on the world. The taxonomy is small and explicit:

These are declared in the agent's tools list and merged into one function-calling surface for the model.

Client tools

Declare a name, description, and JSON-Schema parameters; attach a handler. The SDK validates the model's arguments, runs your handler, and returns the result — with hooks able to deny or rewrite the call first.

import { clientTool } from 'cosmo-ai/tool';
import { zodInput } from 'cosmo-ai/tool/zod';
import { z } from 'zod/v4';

const getWeather = clientTool({
  name: 'get_weather',
  description: 'Current weather for a city',
  input: zodInput(z.object({ city: z.string().describe('City name') })),
  handler: async ({ city }) => ({ tempC: 21.5 }),
});

from typing import Any
from pydantic import BaseModel, Field
from cosmo_ai import RealtimeClient, tool

class WeatherInput(BaseModel):
    city: str = Field(description="City name")

@tool
async def get_weather(input: WeatherInput) -> dict[str, Any]:
    """Current weather for a city."""
    return {"temp_c": 21.5}

agent = client.agent(instructions="…", tools=[get_weather])

struct WeatherArgs: Decodable, Sendable { let city: String }

let getWeather = try AgentTool.clientTool(
    name: "get_weather",
    description: "Current weather for a city",
    input: .object(properties: ["city": .string(description: "City name")], required: ["city"])
) { (args: WeatherArgs) in
    ["temp_c": .double(21.5)]
}

Throw to report a failure

A handler that returns has reported success, whatever the payload says. A result shaped like { ok: false, error: … } is a successful call whose result happens to say no: the model hears a completed action, the agent talks about work that never happened, and the dispatch is recorded as a success.

Throw instead, and say why:

handler: async ({ id }) => {
  const element = lastLook.get(id);
  if (element === undefined) {
    throw new Error(`No element ${id} in the last look. Look at the screen first.`);
  }
  drawHighlight(element);
  return { highlighted: id };
},

@tool
async def highlight(input: HighlightInput) -> dict[str, Any]:
    """Highlight an element from the last look at the screen."""
    element = last_look.get(input.id)
    if element is None:
        raise RuntimeError(
            f"No element {input.id} in the last look. Look at the screen first."
        )
    draw_highlight(element)
    return {"highlighted": input.id}

struct HighlightArgs: Decodable, Sendable { let id: String }
struct HighlightFailed: LocalizedError { let errorDescription: String? }

let highlight = try AgentTool.clientTool(
    name: "highlight",
    description: "Highlight an element from the last look at the screen",
    input: .object(properties: ["id": .string(description: "Element id")], required: ["id"])
) { (args: HighlightArgs) in
    guard let element = lastLook[args.id] else {
        throw HighlightFailed(
            errorDescription: "No element \(args.id) in the last look. Look at the screen first."
        )
    }
    drawHighlight(element)
    return ["highlighted": .string(args.id)]
}

The message is model-facing prose the agent says out loud, not an error code — write it as something the user can act on ("the camera is off — ask them to turn it on"), the same way the draw renderers phrase a refusal. Any thrown error works; the SDK turns it into the call's error reply and records the dispatch as failed.

Two consequences follow. Whatever the exception's message contains reaches the model verbatim, so don't let it carry a path, a credential, or a stack — catch and re-throw with a sentence you chose. And background tools report the same way: a throw before the ack becomes the call's error reply, after it becomes job.fail on your behalf.

Validate arguments with schemas

Tool schemas use a restricted JSON-Schema dialect (no pattern/format, no oneOf, no recursive models; nesting depth at most 6, at most 64 properties across the whole schema, 8 KiB serialized cap). Violations throw ToolDefinitionError at construction time — at import in Python, at tool() in TypeScript, at define in Swift — never mid-call. So does an invalid tool name or a missing or overlong description: .code is a closed ToolDefinitionErrorCode naming which rule was broken.

When the model sends malformed arguments, the SDK rejects them before your handler runs and returns a sanitized INVALID_INPUT error to the model (paths and constraints, never the submitted values). Your handler only ever sees validated, typed input.

Keep the reply small

One serialized reply — the {ok, result, error} envelope, not your result on its own — is capped at 15 KiB. A result over that is not dropped: the SDK shortens it and delivers what fits.

Shortening is structural, so the model always receives well-formed JSON. Long strings are trimmed and terminated with … [truncated]; when the overflow is in the shape rather than the text — a long array, many keys — top-level entries are dropped largest-first instead. Either way the result gains one key:

{
  "temp_c": 21.5,
  "cosmo_sdk_truncated": {
    "note": "partial result — do not answer as if it were complete; narrow the request or say what is missing.",
    "kept_bytes": 15102,
    "original_bytes": 61440
  }
}

The note is what stops the model answering from a partial reply as though it were the whole one. The byte pair is what lets it tell losing a little from losing almost everything — the two call for different answers. The key carries the reserved cosmo_sdk_ prefix, so it never collides with one of yours. Error text and a background tool's ack note are shortened the same way.

Strings are trimmed on Unicode scalar boundaries, so a result truncates identically whichever SDK you ship and a multi-byte character is never split.

Treat the cap as a floor, not a budget. A handler that knows which bytes matter — the newest log lines, the top-ranked hits — should bound its own payload, because the generic pass cannot know what to keep.

Return an answer, not a snapshot

A realtime session inverts REST instincts. There is no cache tier under the conversation: everything you push into a tool reply is re-read, and re-billed, on every subsequent turn. A pull is paid once, and only when the model wants it.

So a tool reply is an answer to a question, not a picture of your application. Prefer several narrow tools the model calls on demand over one tool returning everything it might conceivably need:

The anti-pattern has a signature: a reply sized to the transport ceiling instead of to the question — budget = 15 KiB - 1024. Computing against the cap means the tool is answering a question nobody asked, and paying for it every turn.

Run tools in the background

A regular client tool blocks the conversation until it returns, and the server abandons the call after 10 seconds. If the work takes longer than a beat, make it a background tool: the handler acks immediately (the agent can say "working on it…" and keep talking) and delivers the result later through a job handle.

Background tools ride the WebRTC transport: on the websocket transport, session start refuses an agent that declares one with background_tools_unsupported.

The declaration and the wire shape are identical to a regular client tool. The handler signature is the whole difference: it takes a second argument, a ClientToolJob, and returns nothing.

const exportReport = backgroundClientTool({
  name: 'export_report',
  description: 'Export the quarterly report',
  input: zodInput(z.object({ quarter: z.string() })),
  handler: async ({ quarter }, job) => {
    job.ack('Starting the export');
    const url = await runExport(quarter);        // takes a minute
    await job.complete({ result: { url }, summary: 'The report is ready.' });
  },
});

@tool(background=True)
async def export_report(input: ExportInput, job: ClientToolJob) -> None:
    """Export the quarterly report."""
    await job.ack(note="Starting the export")
    url = await run_export(input)          # takes a minute
    await job.complete(result={"url": url}, summary="The report is ready.")

struct ExportArgs: Decodable, Sendable { let quarter: String }

let exportReport = try AgentTool.backgroundClientTool(
    name: "export_report",
    description: "Export the quarterly report",
    input: .object(properties: ["quarter": .string()], required: ["quarter"])
) { (args: ExportArgs, job: ClientToolJob) in
    await job.ack("Starting the export")
    let url = try await runExport(args.quarter)   // takes a minute
    try await job.complete(result: ["url": .string(url)], summary: "The report is ready.")
}

ClientToolJob is the handle for one invocation, not a kind of tool. ack is what releases the reply; the note you pass is what the model says at acceptance. complete and fail deliver the outcome, which is injected into the conversation whenever it lands, however many turns later that is.

Three rules the handler has to respect:

• Ack, then work. Everything before the ack still blocks the conversation, so ack first and do the work after.
• Always finish. A handler that returns without acking is answered as an error, not as an inline result. One that acks and then returns without completing is failed for you, so the call is never left hanging.
• A throw is reported wherever it happens. Before the ack it becomes the call's error reply; after the ack it becomes job.fail on your behalf.

The terminal result is capped like any tool reply: 8 KiB of result, 2048 characters of summary or error. Say the answer in summary; put anything large where the model can fetch it, such as a URL.

Server tools

Server tools execute on Cosmo's backend; each built-in is its own typed, zero-config spec — the server owns the model-facing declaration, and you write no handler:

const agent = client.agent({ tools: [webSearchTool()] });

from cosmo_ai import web_search_tool

agent = client.agent(instructions="…", tools=[web_search_tool()])

let agent = try client.agent(instructions: "…", tools: [.webSearchTool()])

The session-state pair — cosmo.view_state and cosmo.set_state (session state) — has no typed kinds and cannot be enabled from the SDK: a generic kind: "server" name reference is rejected with a 422 at connect. Run a catalog agent configured with them instead.

Use the tools the SDK ships

A handful of client tools ship with the SDK itself. They run in your app like any other client tool, but the SDK owns the name, description, schema, decode and reply shape — you supply only the one function that does the work. Their names live under the reserved cosmo_sdk_ prefix, and a tool of your own carrying it is rejected where you declared it: the SDK owns those names, so a collision would swap one for something the model was told behaves differently. Every other name stays free, including the natural one an SDK tool shortens to.

Draw on the user's live view

cosmo_sdk_draw_box and cosmo_sdk_draw_point are the renderer half of the locate-then-draw pair. A locator (detect_objects / point_at_object) returns candidates to the model; the model picks the one matching what it's looking at and passes it here, and your handler draws it over the user's camera or screen preview. The locators name these tools in their own descriptions, so declaring them is what turns "here is where it's" into something the user can see — without them the model describes the position out loud instead.

import { drawBoxTool, notShown, shown } from 'cosmo-ai';

const agent = client.agent({
  tools: [
    detectObjectsTool(),
    drawBoxTool((request) => {
      if (!isCameraStreaming()) return notShown('the camera is off — ask the user to turn it on');
      showBox(request);
      return shown;
    }),
  ],
});

from cosmo_ai import detect_objects_tool
from cosmo_ai.tools import DrawBoxRequest, DrawOutcome, draw_box_tool

def on_draw(request: DrawBoxRequest) -> DrawOutcome:
    if not is_camera_streaming():
        return DrawOutcome(shown=False, reason="the camera is off — ask the user to turn it on")
    show_box(request.box, label=request.label)
    return DrawOutcome(shown=True)

agent = client.agent(tools=[detect_objects_tool(), draw_box_tool(on_draw)])

config.tools = [.detectObjectsTool(), .drawBoxTool { request in
    guard isCameraStreaming() else {
        return .notShown("the camera is off — ask the user to turn it on")
    }
    showBox(request)
    return .shown
}]

Three things hold across every SDK:

• Coordinates are normalized to the frame the model was shown — [0,1], upper left origin — so you map them onto your preview the same way you map a locator's own box. Out-of-range values are clamped, so a model that overshoots the frame edge still yields a drawable annotation.
• Malformed arguments never reach your handler. They surface to the model as the invocation's error.
• Answer honestly. The refusal reason is model-facing prose the agent says out loud, not an error code — a box reported as shown but invisible leaves the model talking about something the user can't see.

The point renderer follows the same contract with a point instead of a box. The two exist side by side because they answer different questions: a box around a leaf includes everything behind it, where a marked point says one thing.

Serve a skill body

cosmo_sdk_load_skill is the one tool in this family you never construct: attaching skills is what declares it. The menu of skill names rides in the instructions, and when the model calls this tool the SDK answers with the matching body. It carries the reserved prefix for the same reason the renderers do — the model is told how it behaves, so a tool of your own may not claim the name. The plain load_skill name stays free.

Place the result on screen

A locator returns coordinates normalized to the frame the model was shown, not to your view. Mapping one to the other has to account for how the video is laid out — object-fit cover versus contain, and front-camera mirroring. Every SDK exports the arithmetic so you don't repeat it: boxRect / pointPosition in TypeScript, box_rect / point_position in cosmo_ai.tools, and NormalizedBox.rect(in:...) / NormalizedPoint.point(in:...) in Swift. See Render locator results for the full mapping story — content modes, letterboxing, and selfie mirroring.

Watch dispatch

Every tool invocation — client or server — emits an observability sequence sharing one tool_call_id:

1. tool-call — the model decided to call the tool
2. tool-dispatch-started — the handler began executing
3. tool-result — finished: ok plus a short summary

Client-tool executions additionally arrive as tool-invocation (the actual dispatch to your handler). Render a live tool timeline from these events — see Handle tool calls.

Guard execution

Pre-tool-use hooks run before every client-tool handler and can deny the call or rewrite its arguments — the right place for confirmation gates, argument clamps, and "never touch production" rules that must hold no matter what the model decides. Add production guardrails works through the common patterns.