Cosmo Realtime SDK

Start here

Set up Cosmo from platform.askcosmo.ai/docs/quickstart/cli, then build me a voice agent.

Then install an SDK — npm install cosmo-ai or pip install cosmo-ai-sdk — and a client built with no arguments finds its own credential. Prefer to name your own installer? brew install socratic-ai/tap/cosmo, uv tool install cosmo-cli, and pipx install cosmo-cli all install the CLI. Full walkthrough: Set up with the CLI.

Build realtime voice and multimodal AI into any application. Make it talk. Give it eyes and tools. Put it in your product. Ship it reliably.

The Cosmo Realtime SDK gives your app a live voice-and-vision agent in one call. Audio travels over LiveKit WebRTC tracks; typed JSON events (transcripts, tool calls, errors) travel over a reliable data channel in the same room. You never talk to LiveKit directly — the SDK handles the media plumbing.

Three official clients ship from the same wire protocol:

Every SDK follows the same three-tier model: a client holds your credential, an agent is a reusable persona (instructions, voice, tools), and a session is one live run. See Clients, agents, and sessions.

Make it talk

import { RealtimeClient } from 'cosmo-ai';

const client = new RealtimeClient({ token: endUserJwt }); // or { apiKey } server-side
const agent = client.agent({
  instructions: 'You are a terse voice assistant.',
  greeting: 'Hi — how can I help?',
});

const session = await agent.start();
session.on('transcript', (item) => {
  if (item.isFinal) console.log(`[${item.role}] ${item.text}`);
});

import asyncio, os
from cosmo_ai import RealtimeClient, TranscriptDeltaEvent

async def main() -> None:
    client = RealtimeClient(api_key=os.environ["COSMO_API_KEY"])
    agent = client.agent(instructions="You are a terse voice assistant.")

    async with agent.start() as session:
        await session.set_microphone_enabled(True)
        await session.set_speaker_enabled(True)
        async for event in session:
            match event:
                case TranscriptDeltaEvent() if event.is_final:
                    print(f"[{event.role.value}] {event.text}")

asyncio.run(main())

import CosmoRealtime

let client = RealtimeClient(apiKey: apiKey)
let agent = try client.agent(instructions: "You are a terse voice assistant.")
let session = try await agent.start()

for try await event in session.events {
    if case .transcript(let delta) = event, delta.isFinal {
        print("[\(delta.role == .user ? "user" : "assistant")] \(delta.text)")
    }
}

What gets handled for you

• Session negotiation — POST /api/v1/external/realtime/session/start returns a LiveKit room + token; the SDK joins automatically.
• Audio I/O — the SDK publishes your microphone and plays back the agent's audio. You never manage WebRTC tracks.
• Envelope chunking — control messages over 12,000 bytes are split for the data channel on the way out and reassembled from inbound chunks, transparently in both directions.
• Reconnects — transient network drops recover in place; the server rotates upstream model sessions without ending yours.
• Forward compatibility — unknown event types surface as explicit unknown events, never as decode errors.

What makes an agent shippable

Plumbing gets a demo talking; these are what carry it to production:

• Skills — one agent, many tasks. Each procedure loads just-in-time when a conversation needs it, so scope grows without the prompt — or the per-turn cost — growing with it, and reliability holds as the task list gets long.
• Hooks — guarantees in code, not prompts: deny or rewrite any tool call before it runs, put a classifier where it can actually stop an injected action, and audit every outcome — denials included. Worked patterns: Add production guardrails.
• Native vision tools — stream small, cheap frames; the model examines full resolution only when the moment needs it, so you buy intelligence per answer, not per frame.
• Server hooks — silence handling and call-ending policy the backend runs on your behalf, even if your process dies mid-call.

The developer journey

The sections in the sidebar follow this order, top to bottom.

1. Build one working conversation — set up the CLI, then pick a quickstart.
2. Choose the shape of your app — browser, native, server-side, or phone.
3. Understand the realtime model — sessions, events, turn-taking.
4. Add tools and multimodal input — tools, hooks, skills, image input, screen share.
5. Put it on the phone — telephony.
6. Go deeper — follow a guide or study a runnable example.
7. Ship it reliably — production guardrails, end-user credentials, session limits, usage and limits.

Choose a path

The following table points each starting goal at its quickstart.

Not sure which shape fits? Read Choose your architecture first.