Server tools

A server tool runs on Cosmo's backend. You don't write a handler — the code isn't yours. You opt in with a typed, zero-config spec on the agent; the server owns the model-facing declaration.

Contrast with a client tool, which runs in your process and needs a handler. See Tools for the full taxonomy.

Opt in

Each built-in server tool is its own typed kind — web search (web_search), full-resolution frame examination (examine_image), the object locators (detect_objects / point_at_object), speaker diarization (speaker_log), and hang-up (end_call; see In-call tools). The one kind that carries configuration is the screen-element locator, screen_locate: it grounds against a screenshot only your app can take, so its opt-in takes a capture handler (Screen tools). The object locators return coordinates to the model and draw nothing themselves; declare the SDK renderer they name alongside them so the result reaches the user's screen (Tools). Add the opt-in to the agent's tools:

const agent = client.agent({
  instructions: 'You are a research assistant.',
  tools: [webSearchTool()],
});

from cosmo_ai import web_search_tool

agent = client.agent(
    instructions="You are a research assistant.",
    tools=[web_search_tool()],
)

let client = RealtimeClient(apiKey: apiKey)
let agent = try client.agent(tools: [.webSearchTool()])
let session = try await agent.start()

Identify who said what

Add speakerLogTool() to let the agent attribute speech from a shared microphone. Cosmo runs diarization alongside the voice model and exposes cosmo_who_said_what to it. This is available across voice providers, including GPT Live; your app does not need a Deepgram account or a tool handler.

import { speakerLogTool } from 'cosmo-ai';

const agent = client.agent({
  instructions: 'Ask everyone to introduce themselves. Check who said what before crediting an answer.',
  tools: [speakerLogTool()],
});

from cosmo_ai import speaker_log_tool

agent = client.agent(
    instructions="Ask everyone to introduce themselves. Check who said what before crediting an answer.",
    tools=[speaker_log_tool()],
)

let agent = try client.agent(
    instructions: "Ask everyone to introduce themselves. Check who said what before crediting an answer.",
    tools: [.speakerLogTool()]
)

The tool reads the last 30 seconds by default; the model can request 1–90 seconds. Results contain lines with speaker, text, and seconds_ago, plus speakers and first_heard (each label's first words). Labels such as S0 and S1 distinguish voices; the agent associates them with names from introductions. They are not verified identities. For an unclear attribution, have the agent ask who spoke.

The transcript stays on the server until the model calls the tool. It does not add speaker labels to the SDK's ordinary transcript events. An unavailable log returns status: "unavailable"; an empty window returns status: "ok" with no lines. Check the ready event for a rejected speaker_log opt-in as described below.

Diarization is opt-in and adds a provider charge for audio streamed, including silence, across each tracked microphone. It accrues even when the model does not call the tool. See Provider costs for how usage is measured.

Mix tool kinds

tools is one list and takes any mixture. A server tool and a client tool sit side by side; the model sees a single tool surface and doesn't distinguish where execution happens.

agent = client.agent(
    tools=[
        get_weather,     # client — your handler
        web_search_tool(), # server — Cosmo's
    ],
)

Check availability

Whether a tool can actually run is workspace- and deployment-dependent, so treat a missing tool as a configuration question rather than a bug in your code.

The signal is rejected_tools on the ready event, with a reason per entry:

from cosmo_ai import ReadyEvent

async for event in session:
    if isinstance(event, ReadyEvent) and event.rejected_tools:
        for tool in event.rejected_tools:
            print(f"{tool.name} won't fire: {tool.reason}")

Each entry's name echoes the identifier you declared, so it maps back to the line of config that asked for it. For a server-tool opt-in that is its kind — web_search for web_search_tool() — not the cosmo_-prefixed name the model sees. For a client tool it is the name you gave the tool.

In TypeScript that's rejectedTools on the ready event, or through useToolCalls() for the call stream once running.

This is the first thing to check when a tool "does nothing".

Observe execution

A server tool produces the same lifecycle as any other call — tool-call, tool-dispatch-started, tool-result — correlated by tool_call_id:

from cosmo_ai import ToolCallEvent, ToolResultEvent

async for event in session:
    match event:
        case ToolCallEvent():
            print(f"→ {event.name}")
        case ToolResultEvent():
            print(f"← {'ok' if event.ok else 'failed'}: {event.summary}")

These are observational. There is nothing to respond to: the backend runs and replies on its own.

Next steps

• Tools — client vs server vs background
• Handling tool calls — writing a client tool
• Sessions — where tools ride on the handshake