Cosmo Realtime SDK
ReleasesMigration guidesPython

Upgrade Python to 0.6

Every breaking change in `cosmo-ai-sdk` 0.6.0, with the replacement for each.

Breaking changes when moving cosmo-ai-sdk from v0.5.1 to v0.6.0. The changelog has the full release notes; more than one version behind, chain the pages.

  • Tools are built by calling a constructor, and every constructor returns AgentTool. WebSearchTool() and its siblings are replaced by web_search_tool(), examine_image_tool(), detect_objects_tool(), point_at_object_tool(), end_call_tool(); draw_box, draw_point, screen_locate, screen_click_element, screen_highlight_element and screen_highlight_box gain a _tool suffix. The tool models are no longer exported — declare a hand-written JSON Schema with client_tool(...) / background_client_tool(...) instead of constructing ClientTool directly, and annotate with AgentTool, which is now the single discriminated union (RealtimeToolSpec is gone).

    # before
    client.agent(tools=[WebSearchTool(), ClientTool(name="lookup", description="…", parameters=schema, handler=handle)])
    # after
    client.agent(tools=[web_search_tool(), client_tool(name="lookup", description="…", parameters=schema, handler=handle)])
  • client_tool(...) and background_client_tool(...) validate at construction, matching @tool: the name grammar, the reserved cosmo_sdk_ prefix, the description, and the schema dialect. A declaration the server would refuse now fails where you wrote it instead of arriving as a ready.rejected_tools entry at connect.

  • Server-event models no longer carry the type and id fields. type was a constant restating the class (isinstance is the idiom, and decode never read the field), and id was a client-generated UUID that correlated with nothing — the wire defines neither on server events. Events now expose exactly the wire payload, matching the TypeScript and Swift SDKs field-for-field. Code branching on event.type == "…" switches to isinstance(event, …Event); code logging event.id drops it.

    # before
    async for event in session:
        if event.type == "transcript":
            print(event.text)
    # after
    async for event in session:
        if isinstance(event, TranscriptDeltaEvent):
            print(event.text)
  • audio.noise_cancellation takes a mode instead of a boolean — 'off', 'denoise' or 'voice_focus'. The new one is 'denoise': it removes non-speech noise and keeps every voice, which is what a microphone several people share needs. 'voice_focus' is the previous behaviour, and keeps only the speaker it judges primary — on a shared microphone that treats the second person as background and filters them out.

    true and false remain valid on the wire, so a session started by an already-published SDK version is unaffected.

    true becomes 'voice_focus', false becomes 'off'. A two-person setup that was passing true and losing the quieter speaker wants 'denoise':

    AudioConfig(noise_cancellation=NoiseCancellation.DENOISE)
  • Passing a workspace API key (cosmo_…) as token is refused at construction in every SDK. That parameter takes a minted end-user token; a key there authenticates anyway, so the mistake used to work — and shipped the key with whatever app carried it. Pass the key as the API-key parameter, or mint a token for the user with mint_token and pass that. Acts-as-user tokens (cosmo_pat_…) are unaffected.

    # before
    RealtimeClient(token=api_key)
    # after
    RealtimeClient(api_key=api_key)
  • model_options is gone; its provider block moves onto model, which now takes either the model string it always took or one provider block naming the provider once — a model that disagrees with its knobs is unrepresentable. The provider types drop the Options suffix (GeminiModelOptions → GeminiModel, and likewise for OpenAI, OpenAI-mini and Grok), each block's turn_detection accepts only the detectors its provider offers (Cosmo-VAD tuning moves to CosmoVadConfig on the block's cosmo_vad field), and the union is named RealtimeModel: RealtimeModelOptions is renamed, taking either the string or the block, with RealtimeModelBlock for the block alone. A block with no model id runs the provider's default, and model: "gemini" still selects a provider by name. The server keeps accepting model_options from existing releases — the old pair folds into model server-side — so upgrading the SDK is not coupled to a backend deploy.

    Move the block to model and fold the old model string into its model_id:

    # before
    client.agent(model="gemini-live", model_options=GeminiModelOptions(temperature=0.7))
    # after
    client.agent(model=GeminiModel(model_id="gemini-live", temperature=0.7))
  • SkillParseError is now SkillError and carries a code naming which failure it was, so a caller can tell them apart without reading the message. The codes are not_a_directory, cannot_read, missing_frontmatter, unterminated_frontmatter, malformed_frontmatter_line, duplicate_frontmatter_key, missing_description and duplicate_skill_name — a SkillErrorCode enum in Python and Swift, a union of the same values in TypeScript. The old name described only the five parsing failures, while the type has always also covered a bad path and a duplicate skill name. Match on err.code; err.message is the sentence alone, and an unreadable skills directory now raises SkillError(cannot_read) where it used to escape as a bare PermissionError.

    Breaking: Attaching skills reads the same in every SDK. Swift takes a directory through the skills: argument itself — client.agent(skills: .directory(skillsURL)) — instead of loadSkills(fromDirectory:), which is no longer public; .directory(_:) is a factory on [Skill], so inline skills are unchanged and the two compose with +. Swift's skills is optional rather than defaulting to an empty array. parseSkillMd takes defaultName directly in TypeScript rather than wrapped in an options object, and parse_skill_md is now public in Python for SKILL.md text you already hold. The skill-assembly internals — resolveSkills, skillsMenuText, buildLoadSkillTool, LoadSkillWiring, loadSkillToolName, UnknownSkillError — are no longer public in Swift; the wire name the tool registers under is unchanged, so a hook matching cosmo_sdk_load_skill keeps working.

  • Every MCP failure now raises McpError, carrying a code naming which one it was, so a caller can tell them apart without reading the message. The codes are not_a_file, cannot_read, invalid_json, missing_servers, invalid_server_entry, missing_command, invalid_args, invalid_env, invalid_cwd, duplicate_server_name, connection_failed, invalid_response, server_error and tool_error, plus extra_not_installed — an McpErrorCode enum. It replaces McpConfigError, and covers connection and tool-call failures as well as config, so one except McpError spans the whole concept. Match on error.code; the message is the sentence alone. McpError is no longer a ValueError — a dead subprocess is no ValueError. Two classes fold into codes: McpToolError, a bare RuntimeError outside the error family, becomes tool_error, and McpExtraNotInstalledError becomes extra_not_installed. The second is breaking for anyone catching ImportError or ExtraNotInstalledError around a missing [mcp] install — catch McpError and match the code instead. ExtraNotInstalledError is removed with it: MCP was the only extra that raised it, so it was a base class for a family of none.

    Breaking: Attaching MCP servers reads the same in both SDKs. Swift takes servers through the mcp: argument itself — client.agent(mcp: .configFile(configURL)) — instead of McpRegistry, which is no longer public; .configFile(_:) is a factory on [McpStdioServer], so inline servers are unchanged and the two compose with +. catalogAgent now throws, since duplicate server names are rejected when the agent is built rather than mid-call. The MCP internals — McpRegistry, ConnectedMcp, SkippedTool, MCPToolInfo, MCPCallResult, MCPTransport, MCPTransportFactory, defaultMCPTransportFactory and parseMcpConfig — are no longer public in Swift.

    Fixed: Swift now rejects malformed .mcp.json fields it previously accepted in silence. args that is not an array, or holds a boolean or an object, raises invalid_args instead of being coerced through string conversion — a true became the argument "1". An env that is not an object of strings raises invalid_env rather than being dropped, which had launched the server without the variables it was configured with; a non-string cwd raises invalid_cwd on the same footing. Duplicate server names are now rejected in Swift as they already were in Python. In Python, an unreadable config file raises McpError(cannot_read) where it used to escape as a bare PermissionError, an invalid document is invalid_json rather than sharing one message with an unreadable one, and "args": null means absent, as it already did for env and cwd. Both SDKs run the same mcp-config-vectors.json conformance file.

    Fixed: The runtime codes now say what actually failed. A dead subprocess reports connection_failed in Python where it used to report server_error, and a reply the SDK cannot decode reports invalid_response, which nothing raised before — the three are read from the exception the mcp package raises rather than collapsed into one. In Swift, a well-formed .mcp.json whose root is not an object reports missing_servers instead of claiming the text is not valid JSON, and a config inside a directory the process cannot traverse reports cannot_read instead of not_a_file — fileExists answers false for a permission wall exactly as it does for an absent file, so the read classifies it now.

    Breaking: A number in args is accepted only when it is whole and fits in a signed 64-bit integer, and is written in decimal. 1.0 and 1 are indistinguishable once decoded and a larger integer reached the process in scientific notation, so neither had a spelling both SDKs agreed on; quote the value instead. Both SDKs also walk a document's entries in name order now, which fixes the order servers are attached in and which malformed entry is reported when more than one is bad — Python previously followed document order.

    Fixed: A .mcp.json whose bytes are not UTF-8 now raises McpError coded cannot_read in both SDKs. Python raised a bare UnicodeDecodeError, which is a ValueError rather than an OSError and so escaped the error family entirely; Swift reported not_a_file about a file that is there.

  • MintTokenError.code is now a closed MintTokenErrorCode naming what the SDK saw — request_failed, invalid_response, request_rejected or missing_api_key — and the server's own rejection slug moves to server_code, set only when the code is request_rejected. The two were previously the same field, so code could hold either the SDK's category or anything the server sent, down to a synthetic http_<status>, with no way to tell which. Match on err.code for what happened to the request and read err.server_code for why the server refused. Handlers comparing code against "transport_error" or against a server slug such as "auth_failed" need updating; str(err) is now the message alone, without the code: prefix.

    Breaking: A TokenSource that cannot produce a token now raises TokenSourceError rather than MintTokenError, with its own TokenSourceErrorCode — request_failed, request_rejected, invalid_response or fetcher_failed. Resolving a token source is not part of mint_token: it happens beneath every authenticated call — verify, mint_token, session start, dial and usage reads all resolve it first, and it re-resolves on expiry and after a 401 — so the failure surfaced under the name of one operation it mostly had nothing to do with. token_source_failed is gone from MintTokenErrorCode accordingly, and the four new codes say which part failed where one bucket said only that something did. A refused redirect is request_failed in every SDK — it never reached a token endpoint, so there is no rejection to report; Python previously reported it as an http_<status> rejection.

    Breaking: TypeScript gets the same two errors. MintTokenErrorCode and TokenSourceErrorCode are literal unions rather than aliases of string, both errors take { code, message, serverCode }, and a malformed TokenSource.endpoint URL now throws a TypeError rather than an SDK error — argument validation is not part of the error family, which is what the Python SDK already did.

    Breaking: Swift gets the same two errors, as structs replacing the MintTokenError enum. MintTokenError.rejected(code:detail:), .transport(message:) and .invalidResponse(message:) are gone; construct or match MintTokenError(code:message:serverCode:) with a MintTokenErrorCode instead, and expect TokenSourceError where a TokenSource fetch used to raise a mint error. mint_token now refuses a client built with a minted token or a token source before the request goes out, with code missing_api_key, rather than letting the server answer 401 — and TokenSource.custom rejects a fetcher returning an empty jwt instead of sending it as a bearer.

    # before
    if err.code == "auth_failed": reauthenticate()
    # after
    if err.code == "request_rejected" and err.server_code == "auth_failed": reauthenticate()
  • A TokenSource.custom fetcher now resolves with the same shape mint_token returns — a MintedToken — in every SDK. Python no longer accepts a plain {jwt, expires_at} mapping (construct MintedToken, which still parses an RFC 3339 expires_at string), and TypeScript no longer accepts a string expiresAt (pass a Date; the FetchedToken type is removed — annotate with MintedToken). Python's direct TokenSource(...) construction now validates identically to custom instead of bypassing it. Swift is unchanged.

    # before
    async def fetch(): return {"jwt": jwt, "expires_at": expires_at}
    # after — construct MintedToken (still parses an RFC 3339 string)
    async def fetch(): return MintedToken(jwt=jwt, expires_at=expires_at)
  • AmbienceConfig and the agent's audio.ambience field are removed from every SDK, along with AmbienceTrack. The field never produced audible ambience on any session started through this API — it was accepted and validated, then dropped — so removing it changes no behavior. Delete ambience from your agent's audio block; the rest of the block is unchanged.

  • The five backend calls share an ApiError base — MintTokenError, TokenSourceError, VerifyError, UsageError and DialError all descend from it, so one catch covers any of them while catching a specific one still says which call it was. server_code moves to the base, because a rejection slug belongs to whichever backend answered rather than to the call that asked.

    VerifyError, UsageError and DialError gain closed code enums in place of a bare string: request_failed, request_rejected, invalid_response, plus invalid_request on dial and usage for a call the SDK refuses to make. Where a server slug was the code, it is now server_code and code is request_rejected. Swift gains DialError, which it did not have, and its UsageError and VerifyError become structs carrying code rather than case-carrying enums.

  • One CredentialsError with the closed CredentialsErrorCode replaces six spellings of the same failure — TypeScript's CredentialError (singular), Python's CredentialsError plus its NotFound, File, Expired and Mismatch subclasses, and Swift's case-carrying enum. The five resolution codes are the slugs the cross-SDK vectors already pinned; conflicting_credentials, api_key_in_token_slot and insecure_base_url cover the construction-time guards, which previously threw ValueError or TypeError. CredentialsError is also a ValueError, so existing handling keeps working.

  • Every error carries message, so except RealtimeError as e / catch let e as RealtimeError can read it without narrowing to a concrete type first. In Swift message is now a RealtimeError requirement, which a type conforming to the protocol outside the SDK must add. In Python RealtimeError and its argument-less subclasses — NotConnectedError, VideoPublishAlreadyActiveError — now take the message as their one positional argument. str(error) is unchanged, including the "code: message" form the session, dial, usage, verify and tool-schema errors render.

  • Registering a hook that cannot work now throws HookError in every SDK, with the closed HookErrorCode — malformed_matcher, invalid_hook, server_hook_not_allowed. These previously raised ValueError and TypeError, so neither was catchable as RealtimeError. HookError is exported from cosmo_ai.hooks, beside the hooks it describes. HookError is a ValueError, so an except ValueError around hook declaration keeps firing; the two cases that raised TypeError no longer do.

  • AudioUnavailableError and AudioPublishAlreadyActiveError now require the message they always carried, so AudioUnavailableError() with no arguments raises TypeError where it used to build an error with no text. Pass the message positionally, as every raise site already did.

  • session-ending-soon now decodes to the typed SessionEndingSoonEvent, carrying seconds_remaining and a stable reason slug, instead of surfacing through the unknown-event fallthrough. Code that matched UnknownEvent with raw_type == "session-ending-soon" and read the raw payload must match SessionEndingSoonEvent instead — the unknown-event arm no longer fires for this frame. The session keeps running until session-ended, so use the warning to have the agent wrap up or show a countdown.

  • agent.start() now resolves when the session is ready — the server's handshake has landed — instead of at transport join, so every session method works the moment it returns. This closes a silent-loss window: a send in the join→ready gap previously reached a data channel no agent was subscribed to yet, and was dropped with no error. A session whose ready handshake never arrives within 40 seconds is torn down and start() raises SessionStartError coded ready_timeout; a room that closes before ready raises it coded handshake_failed, carrying the server's boot-failure error frame code and message when one preceded the close. Cancelling a pending start tears the session down and re-raises CancelledError, so asyncio.timeout and task cancellation abort a start cleanly. Readiness is also read from the agent's cosmo.ready participant attribute, so a session that joins after the agent came up — a mid-call observer, a reconnect — still observes it.

  • The screen-capture handler has one shape — it always receives the ScreenCaptureRequest. The zero-argument form is gone: in Python screen_locate_tool's handler must accept the request (lambda request: ...; ignore it if unneeded), and in Swift the handler type is the top-level ScreenCaptureHandler — the request-taking signature ({ request in ... } or { _ in ... }), matching the Python and TypeScript name — with the nested ScreenLocateTool.Handler and RequestHandler names removed. TypeScript already had this shape and is unchanged. Migrate deliberately: a zero-argument handler now fails at call time, and in Python the arity error's text would reach the model as the locator's spoken reason.

  • ScreenCaptureRequest no longer carries wantsElements (wants_elements in Python); the request is an empty envelope, and in Swift its initializer is init(). Capture handlers should always collect elements — the server has always requested them, so nothing changes at runtime. A handler that read the flag can simply drop the check.

  • Declaring screen_locate on the websocket transport now refuses at session start with SessionStartError (code: "config", server_code: "screen_locate_unsupported") — the locator's capture payload travels as a byte stream, a channel the single-socket carrier does not have. Previously TypeScript silently skipped registration and Python and Swift captured the screen and then failed to deliver it; now the capture handler never runs. The Python background-tools refusal gains the matching server_code: "background_tools_unsupported".

  • ErrorCode, SessionStatus, UsageStatus and CredentialKind now accept a value the server added after your package shipped, instead of failing the payload that carried it — previously an unrecognized error code cost the whole error event, and an unrecognized status failed the usage request along with the counters you asked for. Swift switches over these enums need a default or @unknown default now that they carry an unknown(String) case and are no longer @frozen; in Python both kinds are members of the enum, so use value in list(TheEnum) rather than isinstance to tell them apart. TranscriptRole is unchanged.

    In Python a server-added value is an enum member like any other, so declared versus added is a list check:

    if event.code not in list(ErrorCode): log_unknown(event.code)
  • Every way a start can fail now raises one SessionStartError, whose closed SessionStartErrorCode names how far the attempt got — transport, invalid_response, join_failed, config, busy, entitlement, version_mismatch, voice_disabled, rejected, handshake_failed, ready_timeout. Switch on code where you used to branch on a type or read an HTTP status, and read the server's own rejection slug from server_code beside it. It replaces VersionMismatchError. In Python the base error's code — previously open, carrying the server's own slug or a synthetic http_<status> — closes to the enum, with the slug moving to server_code.

    # before — code carried the server's own slug
    try: session = await agent.start()
    except SessionStartError as e:
        if e.code == "concurrent_session_limit": wait_and_retry()
        else: raise
    # after — code is the closed SessionStartErrorCode; the slug is server_code
    try: session = await agent.start()
    except SessionStartError as e:
        if e.code == "busy": wait_and_retry()
        else: raise
  • SessionStartError.detail is a SessionStartRejection in every SDK — the server's structured reason for refusing a start, which no SDK carried in full before. Each group of fields belongs to one server code: limit / active for concurrent_session_limit, granted_minutes / used_minutes for free_minutes_exhausted, balance_cents / top_up_path for insufficient_credits, meter / included / used / reset_at for quota_exceeded, and provider / allowed_providers / plan / upgrade_path for provider_not_entitled. A field the server adds that the SDK does not name is kept rather than dropped.

  • Calling a session method the session cannot serve now throws SessionStateError with the closed SessionStateErrorCode — not_connected, already_started, audio_publish_already_active, video_publish_already_active, screen_share_unavailable, invalid_payload. It replaces NotConnectedError, AudioPublishAlreadyActiveError and VideoPublishAlreadyActiveError. A send issued before ready and one issued after the session ended both report not_connected.

  • The session state machine's value type has one name in every SDK — SessionState. RealtimeSessionState is renamed; fields, kinds, and behaviour are unchanged, so the migration is the rename alone.

  • ToolSchemaError is now ToolDefinitionError, and it covers the whole declaration — a bad tool name and a missing or overlong description throw it too, where a bare ValueError was raised before. It is now catchable as RealtimeError like every other SDK error, and is still a ValueError, so existing handling keeps working. code is the closed ToolDefinitionErrorCode rather than a string.

  • A tool-call validation failure reports its issues as ToolInputIssue in every SDK — path, code, constraint — where Python had raw dictionaries keyed loc/type/ctx, Swift nested the type inside the error, and TypeScript carried path as an array of segments. path is now the dotted form (address.city, items[2].sku) everywhere, the same string the INVALID_INPUT message renders.

    TypeScript exports ToolInputIssue from cosmo-ai/tool: it is the type ToolInputValidationError.issues carries, so a caller reading them has to be able to name it.