Cosmo Realtime SDK
ReferenceREST API

Session start

POST /session/start — the REST call that starts a Cosmo Realtime session and mints the LiveKit join credentials.

POST /api/v1/external/realtime/session/start starts a realtime session from a session-config body and returns the LiveKit join credentials. The response contains everything the client needs to join the room; all further control traffic rides the wire protocol on the room's data channel.

The SDKs call this internally (agent.start()). Use the raw endpoint only for custom integrations.

Schema and try-it: this endpoint in the API explorer.


Authentication

Bearer token in the Authorization header — either of:

  • a workspace API key carrying the realtime:start scope (server-side secret; the deprecated realtime:use umbrella implies it), or
  • a minted end-user JWT (from POST /api/v1/external/auth/token), safe on end-user devices.
Authorization: Bearer cosmo_...

See API keys, Scopes, and End-user credentials.


Request

POST /api/v1/external/realtime/session/start
Content-Type: application/json
Authorization: Bearer cosmo_...

The body is the external protocol's session-config payload: an agent block (the persona) plus a session block (per-run options).

{
  "type": "session-config",
  "sdk": { "name": "cosmo-ai-sdk", "version": "X.Y.Z" },
  "agent": {
    "type": "inline",
    "instructions": "You are a concise support agent.",
    "voice": { "name": "Puck" },
    "tools": [{ "kind": "web_search" }]
  },
  "session": { "max_session_seconds": 1800 }
}

Top level

FieldTypeRequiredDefaultDescription
type"session-config"no"session-config"Discriminator literal.
sdk{name, version}yes—SDK identity of the caller. The SDKs stamp it automatically; a direct REST caller supplies its own name and version.
agentcatalog | inlinenoneutral inline agentThe persona — see the two variants below, discriminated on type.
sessionobjectno{}Per-run options — see Session params.

agent — catalog variant

Run a workspace catalog agent by machine handle; the stored config runs verbatim. Only per-run ride-alongs may accompany the launch — any other field is a schema error (extra="forbid"), not a runtime rejection.

FieldTypeRequiredDescription
type"catalog"yesDiscriminator.
namestringyesWorkspace-unique machine handle (hyphen slug). Unknown or cross-workspace names reject the start.
inputsRecord<string,string>noPer-run values substituted into the resolved prompt's {{key}} placeholders.
toolstool spec[]noThe session's tool set — client-executed specs and server-tool opt-ins. A stored agent carries no tools of its own, so these are the whole set, not an addition to one.
voiceobjectnoPer-run voice: name (the override id — the one cosmetic exception to "stored config runs verbatim") and speaking_style ("how to speak" text appended to the system prompt after the persona, max 8192 chars).

agent — inline variant

Define the persona in the request. Catalog-only fields (name, inputs) are structurally absent here.

FieldTypeRequiredDefaultDescription
type"inline"yes—Discriminator.
instructionsstringnoserver defaultSystem instructions. Capped at 131,072 characters (instructions_too_long).
modelstring | objectnoserver defaultA provider family alias ("gemini", "openai", "openai_mini", "openai_live", "grok") or a concrete model id as a plain string ("gemini-3.8-live" runs Gemini 3.8 Live; Gemini's default is gemini-3.1-flash-live-preview), or one provider block (below). Unknown or unavailable values are rejected (model_unavailable).
model (block form)objectnoprovider defaultsDiscriminated on provider, each carrying an optional model_id that pins the concrete model within that provider: gemini (temperature, max_output_tokens, thinking_level, include_thoughts, turn_detection — unset and cosmo_vad run Cosmo's semantic turn detection, server_vad opts into Gemini's silence-window detection, semantic_vad is rejected (turn_detection_unsupported) — plus end_of_speech_sensitivity, silence_duration_ms, prefix_padding_ms, read only with server_vad, and the cosmo_vad object (pause_ms, prefix_ms, max_hold_ms, each 0–5000) tuning the semantic detector), openai (turn_detection, plus eagerness for semantic_vad or silence_duration_ms / prefix_padding_ms for server_vad), openai_mini (no knobs today), openai_live (the backend Responses model GPT Live delegates tool calls to: responses_model, responses_instructions, reasoning_effort — minimal / low / medium / high — verbosity — low / medium / high — tool_choice — auto / required / none — parallel_tool_calls, max_output_tokens (16–32768), service_tier — auto / default / flex / priority; delegation — responses (default) / client / cosmo — who does the work the voice model hands off, and under client or cosmo the session emits delegation-created and accepts delegation-append frames, no tools may be declared (delegation_tools_unsupported), and cosmo needs a credential with the resources:read scope (cosmo_delegation_scope_required); no detector knobs, the model owns its turn-taking), grok (turn_detection — server_vad only — plus silence_duration_ms / prefix_padding_ms). A model_id belonging to another provider is rejected (model_unavailable); pairing an OpenAI knob with the other detector is rejected (turn_detection_knob_mismatch); naming semantic_vad or cosmo_vad on the Grok block is rejected (turn_detection_unsupported).
model_optionsobjectno—Deprecated — merged into model, and no longer in the published schema (this row is its remaining documentation). Still accepted from older SDK releases: the block folds into model at validation, with a string model becoming the block's model_id (a provider mismatch between the two is rejected, model_unavailable). New requests pass the provider block as model and never send this.
voiceobjectnoupstream picksname (provider-specific prebuilt voice id) and speaking_style ("how to speak" text, as on the catalog variant).
audioobjectnoall defaultsThe audio pipeline: output (bool, default true — false runs the session text-only; rejected when the resolved model is speech-to-speech only), noise_cancellation (off / denoise / voice_focus, default off — which filter cleans the user's inbound audio; denoise keeps every voice, voice_focus keeps only the primary speaker. true / false are still accepted here, on the wire, as the pre-mode spelling of voice_focus / off, for SDK versions that predate the modes).
toolstool spec[]nononeTool set for the session — see Tool specs.
greetingstringnowait for userOpening line the assistant speaks as soon as the model session opens. Max 4000 chars. A resumed session never re-greets.
interruption_sensitivity"low" | "default" | "high"no"default"How readily user audio barges in over the assistant.
hooksSilenceTimeout[]nononeDeclarative server hooks (max 16): {trigger: "user.speech.timeout", timeout_seconds, action, max_count?, reset_mode?, name?} with a say or end_call action. See Hooks.

Tool specs

Each entry in tools is discriminated on kind:

kindFieldsDescription
clientname, description, parameters (JSON Schema, restricted dialect)Client-executed tool, self-described at start. name must match ^[a-z][a-z0-9_]{2,63}$ and may not start with cosmo_ (reserved for server tools; the cosmo_sdk_ prefix is the one exception, for client tools an SDK ships).
web_search / examine_image / detect_objects / point_at_object / screen_locate / end_callnoneZero-config opt-ins to server-executed tools; the server owns the model-facing declaration.
catalognameReserved for referencing a workspace tool by handle. Not accepted yet — a session carrying one is rejected with invalid_tool_config.

A kind outside this table fails schema validation and is rejected as a validation_error, like any other malformed body. The retired kind: "server" is different: it still parses (the schema knows it), then rejects with invalid_tool_config — the same path as catalog.

Session params

FieldTypeDefaultDescription
max_session_secondsint (60–14400)server limitRequested wall-clock cap. The server resolves the effective cap as the minimum of this and its own limits; the effective value is echoed on ready.
store_recordingboolrecordsfalse writes no recording artifacts for this run. See Recording and privacy.
store_audioboolstore_recording, else recordsPersist this run's audio. Wins over store_recording. Narrowing only: a run may store less than the account's consents allow, never more.
store_transcriptboolstore_recording, else recordsPersist this run's transcript and tool-event artifacts. Same contract as store_audio.
store_videoboolstore_recording, else recordsPersist this run's screen-share video and screenshots. Same contract as store_audio.
experimental.resume_session_idUUID—Resume the named prior session. Experimental — may change shape between releases.

Response

200 OK

{
  "livekit_url": "wss://lk.example.com",
  "token": "<livekit-access-token>",
  "room_name": "cosmo-9f3c1ad84be7205c6d18e4b2",
  "session_id": "8f7e6d5c-4b3a-2190-aaaa-bbbbccccdddd"
}
FieldTypeDescription
livekit_urlstringLiveKit room URL to connect to.
tokenstringShort-lived participant join token. Don't cache it.
room_namestringLiveKit room name.
session_idstringServer-minted session identifier. Pass back as experimental.resume_session_id to resume; include in support requests.
timingsobjectOptional server-side session-start phase breakdown (ms).

After joining the room, wait for the agent's readiness before the first send. It is announced two ways: a ready frame broadcast on the data channel, and the cosmo.ready participant attribute on the agent — room state, so a client that joins or subscribes late still reads it. Latch on whichever you see first. The attribute carries the same payload as the frame unless that payload is oversized, in which case it degrades to the session id and duration cap alone — so read rejected_tools and agent from the frame when you have it. See Lifecycle.


Errors

Every non-2xx response carries the same envelope:

{
  "error": {
    "type": "api_error",
    "code": "unknown_agent",
    "message": "No agent named 'support-agnt' in this workspace.",
    "errors": null
  }
}
FieldDescription
typeError family: api_error, validation_error, or internal_error.
codeMachine-readable slug for typed rejections; absent otherwise.
messageHuman-readable reason.
errorsField-level violations (loc, type, msg) on validation_error responses. Payload values are never echoed back.

Notable rejections:

Statuscode / typeCause
401 / 403api_errorCredential absent, invalid, or lacking the realtime:start scope.
401api_error"No access to workspace '<id>' from this host" — the credential is valid, but its workspace is not served on the Cosmo host this request reached. Every external bearer credential (API key or minted token) works only against the host its workspace was created on; point COSMO_BASE_URL (or the client's base URL) back at that host.
400sdk_unsupportedThe SDK identity names a version below the supported floor; the message carries the minimum version and the upgrade command.
400version_mismatchReserved — current servers don't send it; older servers may refuse an incompatible client with it.
400api_error (no code)audio.output: false requested with a model that is speech-to-speech only.
402free_minutes_exhaustedThe workspace's free voice grant is spent. See Limits.
422validation_errorBody failed schema validation — including the legacy flat shape (persona fields at the top level instead of under agent / session), sending a provider block as both model and the deprecated model_options, and unknown fields on a catalog agent block.
422invalid_tool_configOne or more tool specs failed validation. The message lists every problem, joined by ; .
422unknown_agentThe catalog name doesn't resolve in this workspace.
422agent_config_unavailableThe catalog agent exists but its stored configuration couldn't be loaded.
422greeting_too_longA catalog agent's greeting exceeds 4000 characters after inputs substitution.
422model_unavailableUnknown model id, one not available to this workspace/server, or a model_id that isn't a model of the block's own provider.
422instructions_too_longinstructions exceeds 131,072 characters.
422delegation_tools_unsupportedTools declared on a GPT Live model block with delegation client or cosmo. The voice model has no tool channel when it hands work off.
403cosmo_delegation_scope_requireddelegation: "cosmo" from a credential without the resources:read scope. The workspace agent searches and reads workspace resources for the session. Minted end-user tokens cannot carry that scope.
429concurrent_session_limitThe workspace already has its maximum number of live sessions. See Limits.
503—Realtime backend temporarily unavailable. Retry with backoff.
422turn_detection_knob_mismatchA model block knob belongs to the turn detector that isn't selected — on OpenAI, or on Gemini once a detector is named (a server_vad knob with cosmo_vad, or the cosmo_vad block with server_vad).
422turn_detection_unsupportedA detector the provider cannot run: "semantic_vad" on the Gemini block (OpenAI-only; leave turn_detection unset or send "cosmo_vad" for semantic detection, "server_vad" for the silence window), or "cosmo_vad" on the OpenAI block (Gemini-only).
422thinking_level_unsupportedA thinking_level on a Gemini block whose model takes none (gemini-3.8-live). Omit it.

Tool problems fail hard, not soft. Any malformed or unsupported spec rejects the entire start; no session is created. A kind the schema doesn't know at all comes back as a validation_error; a bad client-tool name or schema, a duplicate, or a known-but-unsupported kind (server, catalog) rejects with invalid_tool_config.

The ready frame's rejected_tools reports something different: tools that were valid but are unavailable on this deployment or workspace, dropped so the rest of the session can run. See Wire protocol.


Example — curl

curl -X POST https://platform.askcosmo.ai/api/v1/external/realtime/session/start \
  -H "Authorization: Bearer cosmo_..." \
  -H "Content-Type: application/json" \
  -d '{
    "agent": {
      "type": "catalog",
      "name": "support-agent",
      "inputs": { "caller_name": "Sam" }
    },
    "session": { "max_session_seconds": 1800 }
  }'

Sibling endpoints

EndpointPurpose
GET /api/v1/external/realtime/verifyCheck the credential without starting a session — no scope required, not metered. See GET /realtime/verify.
POST /api/v1/external/realtime/session/{session_id}/dialDial an E.164 number into a running session as a SIP participant (phone_number, optional caller_number; returns dial_id). Requires an API key — a minted end-user token is rejected with 403 dial_requires_api_key. Other rejections: 403 phone_calls_disabled / minute_limit_exceeded / do_not_call, 400 caller_number_not_available, 404 session_not_found, 409 session_not_live, 409 session_already_dialed, 503 consent_check_unavailable (retry). See Telephony.
POST /api/v1/external/auth/tokenMint an end-user JWT from an API key (external_user_id, optional ttl_seconds → jwt, expires_at, token_id); DELETE /api/v1/external/auth/token/{token_id} revokes one early. See End-user credentials.
GET /api/v1/external/sessionsRecorded-session history — list, read, transcript, delete, capabilities, import. See Sessions REST.

All take the same Bearer auth; session start requires realtime:start, dial realtime:dial, session reads realtime:read, token minting and revocation user_tokens:mint, and verify no scope at all. The deprecated realtime:use umbrella implies all the realtime verbs — see Scopes.

On this page