Cosmo Realtime SDK

Choose your architecture

Where the SDK runs, which credential it holds, and how audio reaches the agent — decide before you write code.

Every Cosmo Realtime session has the same shape: a REST call creates the session, the SDK joins a LiveKit room, audio flows as WebRTC tracks, and typed JSON events flow on a data channel. What varies is where the SDK runs, which credential it holds, and whose microphone joins the room. Pick the shape first; the code follows.

Cosmo Realtime system architectureYour app, running the Cosmo SDK, makes a session-start REST call with a credential to the Cosmo backend. The backend creates the session and dispatches the agent. The app then exchanges WebRTC audio tracks and typed events on a data channel with a LiveKit room; the agent exchanges audio and events with the same room.Your appCosmo SDKTypeScript · Python · SwiftCosmo backendRESTLiveKit roomWebRTCAgentmodel + voice① session-start (credential)creates the session,dispatches the agent② audio (WebRTC)typed events(data channel)audio + events

The four shapes

1. Client-direct (browser or native app)

The end user's device runs the SDK and connects straight to Cosmo. Your server's only job is minting a short-lived JWT per user.

Client-direct deployment shapeThe end-user device runs the SDK with a minted JWT and exchanges audio and events directly with Cosmo. Your server only mints short-lived JWTs for the device.End-user deviceyour app + SDK + minted JWTCosmosession · room · agentYour serverJWT mint endpointaudio + eventsshort-lived JWT
  • Lowest audio latency — media goes device ↔ Cosmo directly.
  • Client tools run on the device: update UI, read device state.
  • The device holds only a per-user token, never your API key.
  • Use cosmo-ai in the browser or CosmoAI (Swift) on iOS/macOS.

2. Server-side agent (headless)

Your backend runs the SDK with a workspace API key. There may be no human microphone at all — the session is driven by text turns, custom audio sources, or a phone leg.

Server-side agent deployment shapeYour backend runs the SDK with a workspace API key and exchanges text turns, custom audio, and typed events with Cosmo. No end-user device joins the session.Your backendSDK + workspace API keyCosmosession · room · agentaudio + text turnstyped events
  • Tools execute in trusted infrastructure next to your databases and secrets.
  • Python is the natural fit (cosmo-ai-sdk, asyncio); TypeScript works in Node.
  • Custom audio in/out: start_audio_stream(...) to feed frames from a source of your own, agent_audio() to consume the agent's 48 kHz PCM.

3. Phone agent

A server-side session plus a SIP leg: start the session, then dial() an E.164 number. The callee joins the same LiveKit room as a SIP participant; the agent neither knows nor cares that the voice arrived by phone.

Phone agent deployment shapeYour backend starts a session and calls dial(). Cosmo bridges a SIP leg to the callee’s phone, which joins the same room as the agent.Your backendSDK + API keyCosmosession · room · agentCallee’s phonePSTNstart + dial()SIP leg
  • Declare call-control tools explicitly: end_call lets the agent hang up. See In-call tools.
  • Tune the audio path for phone conditions: interruption_sensitivity, audio.noise_cancellation, silence-timeout hooks for dead air.

4. Client app + catalog agent

The device connects directly (shape 1), but the agent's persona lives server-side as a catalog agent managed in the dashboard. Ship a thin client that names the agent; iterate on instructions, voice, and tools without an app release.

Client app plus catalog agent deployment shapeThe end-user device runs a thin client with a minted JWT and connects directly to Cosmo. The agent’s persona, voice, and tools live in a catalog agent managed in the dashboard and are resolved at session start.End-user devicethin client + minted JWTCosmosession · room · agentDashboardcatalog agent: persona · voice · toolsaudio + eventsresolved at session start

Decision table

RequirementRecommended shape
Lowest-latency browser assistantClient-direct, TypeScript + minted end-user token
Native iOS / macOS experienceClient-direct, Swift + minted end-user token
Tools that touch private databases or secretsServer-side Python agent, or client-direct + server tools
Phone agent (outbound)Server-side session + dial()
Iterate on persona without shipping app updatesCatalog agent + thin client
Load testing, batch processing, simulationsServer-side Python with custom audio sources

Credentials by shape

CredentialLooks likeLivesShapes
Workspace API key (realtime scopes)cosmo_…your server / laptop onlyserver-side, phone, prototyping
Minted end-user JWTeyJ…end-user devicesclient-direct, catalog thin clients
Provisioning key (user_tokens:mint only)cosmo_…your token-minting endpointmints JWTs; can't join sessions itself

The minting flow is three lines of server code — see End-user credentials.

Model pipeline: A provider choice, not an architecture choice

Whether the agent runs a native speech-to-speech model or a modular STT → LLM → TTS pipeline is selected per agent via model, not by changing your app's architecture. An unavailable model is rejected explicitly at session start rather than silently substituted; provider-specific knobs (Gemini thinking_level, …) ride on model's provider block. Text-only agents set audio.output: false.

Stateful sessions

Sessions are stateful and server-anchored: the conversation, tool state, and recording live with the session, not the connection. A dropped network recovers in place (reconnects); an ended session can be continued with resume_session_id (session limits and resume).

Developer dashboard

Where to manage API keys, end users, usage, and billing.

Concepts

The realtime model behind every SDK — how a session works, what flows through it, and what happens when something goes wrong.

On this page

The four shapes1. Client-direct (browser or native app)2. Server-side agent (headless)3. Phone agent4. Client app + catalog agentDecision tableCredentials by shapeModel pipeline: A provider choice, not an architecture choiceStateful sessions