Choose your architecture
Where the SDK runs, which credential it holds, and how audio reaches the agent — decide before you write code.
Every Cosmo Realtime session has the same shape: a REST call creates the session, the SDK joins a LiveKit room, audio flows as WebRTC tracks, and typed JSON events flow on a data channel. What varies is where the SDK runs, which credential it holds, and whose microphone joins the room. Pick the shape first; the code follows.
The four shapes
1. Client-direct (browser or native app)
The end user's device runs the SDK and connects straight to Cosmo. Your server's only job is minting a short-lived JWT per user.
end-user device ──JWT──► Cosmo backend ──► LiveKit room ◄── agent
▲ │
└── your server mints the JWT ───────────┘- Lowest audio latency — media goes device ↔ Cosmo directly.
- Client tools run on the device: update UI, read device state.
- The device holds only a per-user token, never your API key.
- Use
cosmo-aiin the browser orCosmoRealtime(Swift) on iOS/macOS.
2. Server-side agent (headless)
Your backend runs the SDK with a workspace API key. There may be no human microphone at all — the session is driven by text turns, custom audio sources, or a phone leg.
- Tools execute in trusted infrastructure next to your databases and secrets.
- Python is the natural fit (
cosmo-ai-sdk, asyncio); TypeScript works in Node. - Custom audio in/out:
publish_audio_source(...)to feed frames,agent_audio()to consume the agent's 48 kHz PCM.
3. Phone agent
A server-side session plus a SIP leg: start the session, then dial() an E.164 number. The callee joins the same LiveKit room as a SIP participant; the agent neither knows nor cares that the voice arrived by phone.
- Declare call-control tools explicitly:
cosmo.end_callto hang up, a transfer-call tool to hand off to a human queue. - Tune the audio path for phone conditions:
interruption_sensitivity,audio.noise_cancellation, silence-timeout hooks for dead air.
4. Client app + catalog agent
The device connects directly (shape 1), but the agent's persona lives server-side as a catalog agent managed in the dashboard. Ship a thin client that names the agent; iterate on instructions, voice, and tools without an app release.
Decision table
| Requirement | Recommended shape |
|---|---|
| Lowest-latency browser assistant | Client-direct, TypeScript + minted JWT |
| Native iOS / macOS experience | Client-direct, Swift + minted JWT |
| Tools that touch private databases or secrets | Server-side Python agent, or client-direct + server tools |
| Phone agent (outbound) | Server-side session + dial() |
| Iterate on persona without shipping app updates | Catalog agent + thin client |
| Load testing, batch processing, simulations | Server-side Python with custom audio sources |
Credentials by shape
| Credential | Looks like | Lives | Shapes |
|---|---|---|---|
Workspace API key (realtime:use) | cosmo_… | your server / laptop only | server-side, phone, prototyping |
| Minted end-user JWT | eyJ… | end-user devices | client-direct, catalog thin clients |
Provisioning key (user_tokens:mint only) | cosmo_… | your token-minting endpoint | mints JWTs; cannot join sessions itself |
The minting flow is three lines of server code — see End-user credentials.
Model pipeline: a provider choice, not an architecture choice
Whether the agent runs a native speech-to-speech model or a modular STT → LLM → TTS pipeline is selected per agent via model and model_options, not by changing your app's architecture. An unavailable model is rejected explicitly at session start rather than silently substituted; provider-specific knobs (Gemini thinking_level, Ultravox turn_endpoint_delay_seconds, …) ride on model_options. Text-only agents set audio.output: false.
Stateful sessions
Sessions are stateful and server-anchored: the conversation, tool state, and recording live with the session, not the connection. A dropped network recovers in place (reconnects); an ended session can be continued with resume_session_id (session limits & resume).