Server-side tools

A server tool runs on Cosmo's backend. You do not write a handler — the code is not yours. You opt in with a typed, zero-config spec on the agent; the server owns the model-facing declaration.

Contrast with a client tool, which runs in your process and needs a handler. See Tools for the full taxonomy.

Opting in

Each built-in server tool is its own typed kind — web search (web_search), full-resolution frame examination (examine_image), and the object locators (detect_objects / point_at_object). Add the opt-in to the agent's tools:

from cosmo_ai import WebSearchTool

agent = client.agent(
    instructions="You are a research assistant.",
    tools=[WebSearchTool()],
)

const agent = client.agent({
  instructions: 'You are a research assistant.',
  tools: [{ kind: 'web_search' }],
});

let session = try await RealtimeSession.start(
    .init(apiKey: apiKey, baseURL: baseURL),
    config: SessionConfig(tools: [.webSearch])
)

Mixing tool kinds

tools is one list and takes any mixture. A server tool and a client tool sit side by side; the model sees a single tool surface and does not distinguish where execution happens.

agent = client.agent(
    tools=[
        get_weather,     # client — your handler
        WebSearchTool(), # server — Cosmo's
    ],
)

Availability

Whether a tool can actually run is workspace- and deployment-dependent, so treat a missing tool as a configuration question rather than a bug in your code.

The signal is rejected_tools on the ready event, with a reason per entry:

from cosmo_ai import RealtimeReady

async for event in session:
    if isinstance(event, RealtimeReady) and event.rejected_tools:
        print("rejected:", event.rejected_tools)

In TypeScript that is rejectedTools on the ready event, or via useToolCalls() for the call stream once running.

This is the first thing to check when a tool "does nothing".

Observing execution

A server tool produces the same lifecycle as any other call — tool-call, tool-dispatch-started, tool-result — correlated by tool_call_id:

from cosmo_ai import RealtimeToolCall, RealtimeToolResult

async for event in session:
    match event:
        case RealtimeToolCall():
            print(f"→ {event.name}")
        case RealtimeToolResult():
            print(f"← {'ok' if event.ok else 'failed'}: {event.summary}")

These are observational. There is nothing to respond to: the backend executes and replies on its own.

Next steps

• Tools — client vs server vs background
• Handling tool calls — writing a client tool
• Sessions — where tools ride on the handshake