RealtimeClient (TypeScript)

RealtimeClient owns what outlives any one session: the credential, the resolved base URL, and the agent factories. Each agent.start() creates an independent session — its own transport, state, and event stream — so one client can run any number of concurrent sessions, and two clients coexist without interfering.

Sessions open through the agent surface: build a persona with client.agent({...}) (or client.catalogAgent(name)), then agent.start() returns a RealtimeSession. Everything scoped to one run — events, sends, media controls, lifecycle — lives on that session object. For React apps, RealtimeProvider wraps a session.

import { RealtimeClient } from 'cosmo-ai';

const client = new RealtimeClient({
  token: mintedJwt, // from mintToken() on your backend
});

const agent = client.agent({
  instructions: 'You are a concise support agent.',
  voice: 'Puck',
});
const session = await agent.start();
await session.sendText('Hello');

Constructor

new RealtimeClient(options?: RealtimeClientOptions)

RealtimeClientOptions

Provide at most one of apiKey / token — supplying both throws CredentialsError (conflicting_credentials). Omit both and the client resolves a credential itself on Node: COSMO_API_KEY, then the ~/.cosmo/credentials file cosmo login writes (COSMO_CREDENTIALS_FILE relocates it; COSMO_PROFILE selects the profile). When nothing resolves — including in a browser, which has no environment or credentials file — the first call that needs the credential throws CredentialsError (code: 'no_credential') before any request is sent; construction itself never throws for this, because the chain is async. See API keys.

The API origin is not a constructor option. It resolves at construction from COSMO_BASE_URL in Node, from the cosmo-base-url <meta> tag on a Cosmo-served page (empty value means the page's own origin), and otherwise from the production default. Read the resolved value back off client.baseUrl.

The WebSocket carrier is for the one-process local server. It sends PCM audio and the session protocol on one socket, with no media room, worker or UDP. It supports browser microphone input, caller-owned MediaStream audio and ordinary client tools. A dropped socket ends the session; camera, screen share, background client tools, dial and usage reads remain on the room/managed lane.

Auth

mintToken(externalUserId: string, options?: { ttlSeconds?: number }): Promise<MintedToken>

Mint a short-lived end-user token. Run this on your backend with an apiKey client; hand the returned jwt to the end user's browser, which constructs new RealtimeClient({ token: jwt }). Idempotent per (workspace, externalUserId). options.ttlSeconds (60–86400) shortens the 24-hour default lifetime.

Throws MintTokenError, whose code is missing_api_key (this client has no apiKey — refused before the request goes out), request_failed, invalid_response, or request_rejected; on a rejection serverCode carries the server's own slug, or the http_<status> synthetic when the response carried none.

type MintedToken = { jwt: string; expiresAt: Date; tokenId?: string };

tokenId is the revocation handle (DELETE /api/v1/external/auth/token/{token_id}) — keep it on your server. Cosmo always returns it; the type marks it optional only because MintedToken doubles as the TokenSource fetch shape.

See /production/end-user-credentials.

verify(): Promise<CredentialInfo>

Check the credential without starting a session — free, and works with either credential. Resolving at all means the credential authenticated; canStartSessions reports whether it carries realtime:start, and realtimeVoiceAvailable whether this deployment has the default voice stack configured.

Throws VerifyError when the server rejects the credential — its closed code is request_rejected (with the server's own slug on serverCode), request_failed (the request never completed), or invalid_response (the body did not parse). An under-scoped credential does not throw.

type CredentialInfo = {
  /** 'api_key' or 'user_token' today — open-ended, treat unknown values defensively. */
  credential: CredentialKind;
  /** Null for a minted token — an end user is not told the workspace. */
  workspace: WorkspaceInfo | null; // { name: string; slug: string }
  scopes: string[];
  canStartSessions: boolean;
  realtimeVoiceAvailable: boolean;
  externalUserId: string | null;
};

See /auth/api-keys.

Usage

getSessionUsage(sessionId: string): Promise<SessionUsage>

Fetch a session's usage summary (duration, talk time, token counts) over REST, by explicit id — the client outlives any one session, so the id is passed in. session.usage() is the id-carrying form on the session object. Throws UsageError if the server rejects or the request fails.

Agent builders

agent(config?: AgentConfig): RealtimeAgent

Build a reusable persona — immutable; open runs with agent.start(). Fields left unset fall through to the server-side protocol defaults.

Plugins bundle instructions, skills, tools, and hooks for inline agents. Pass plugins at construction; the built agent exposes the expanded contributions. See Plugins for the bundle fields, merge order, and PluginError codes.

catalogAgent(name: string, options?: CatalogAgentOptions): RealtimeAgent

Build an agent that runs a workspace catalog agent by machine handle; the stored config runs verbatim. Only per-run ride-alongs are accepted — no persona parameters except voice, the one cosmetic override; anything else is a type error.

See RealtimeAgent below for both config shapes.

RealtimeAgent

The reusable persona. Build once via client.agent({...}) or client.catalogAgent(name, {...}) and start any number of sessions from it; the agent is frozen, so build another one to vary a field. Validation (duplicate skill names, malformed hooks) throws when the agent is built, not at start().

config

The resolved persona this agent was built with, frozen (Object.freeze). For a catalog agent it also carries the name and inputs ride-alongs. Read-only and informational — the resolved config type itself is not an exported name, so read fields off it rather than annotating it.

AgentConfig

GeminiModel({ modelId: 'gemini-3.8-live' }), or the plain string 'gemini-3.8-live', runs Gemini 3.8 Live, which takes no thinkingLevel: setting one fails session start with thinking_level_unsupported.

CatalogAgentOptions

Per-run ride-alongs for client.catalogAgent(name, {...}) — the stored config runs verbatim, so there are no persona fields here.

start(options?: SessionStartOptions): Promise<RealtimeSession>

Open one session from this persona. Resolves once the session is ready — the server's handshake has landed, so every session method works immediately. Rejects on any failure to get there — you never receive a session for a run that failed to start. Every failure throws one SessionStartError whose code names it: a server rejection is busy, entitlement, config, version_mismatch, voice_disabled, or rejected; a request that never reached the server is transport; a room that closes before ready is handshake_failed (the server's pre-close error frame rides on detail); a ready handshake that never arrives is torn down after a bounded wait and is ready_timeout. See Errors.

prepareSession(options?: SessionStartOptions): PreparedSession

Prepare one session ahead of its start, so it starts faster. The SDK reserves a room in the background immediately, and the returned PreparedSession joins it while the session request is still in flight when you start it — instead of waiting for a room to be allocated after the request returns. options are the same per-run options start() takes; they are fixed here, and the start takes none.

Prepare as early as the app knows a session is coming — while the rest of its setup runs — and start when the user is ready:

const prepared = agent.prepareSession();
// ... the rest of the app's setup
const session = await prepared.start();

Purely an accelerator: a reservation that failed, lapsed, or is declined by the server leaves the start on the ordinary path, with the same result as start(). Throws when this client's sessions do not run in rooms — the websocket transport, or a custom transportFactory.

PreparedSession

Returned by agent.prepareSession(): one session prepared ahead of its start. The reservation is refreshed in the background until the handle is started or closed, so one held for hours stays warm.

start(): Promise<RealtimeSession>

Start the prepared session. Resolves and rejects exactly as agent.start() does. Single-use: a second call rejects; prepare another session for another start.

close(): void

Drop the reservation and stop refreshing it, for a handle that will never be started. A no-op once started.

RealtimeSession

One live run of an agent, returned by RealtimeAgent.start(). Each session owns its own engine — transport, state, event stream — so concurrent sessions on one client are fully independent, and a session's methods always address that session.

const session = await agent.start();
await session.sendText('Hi');
await session.end();

Properties

type SessionConnectTimings = {
  wsMs: number;              // session-start POST
  roomMs: number;            // LiveKit join
  micMs: number;             // mic publish (0 when none is published)
  totalConnectMs: number;    // the whole connect
  readyMs?: number | null;   // to the ready event
  serverTimings: SessionStartTimings | null;
};

readyMs is measured from the same instant as wsMs, and stays null until the agent reports ready. It is optional on the type — a custom RealtimeTransport that measures only the connect phases may leave it off. Once the agent is live, the client reports its phases to the session so the server can record the whole waterfall against it.

serverTimings is null on a backend that doesn't report it, and carries version_check_ms, project_check_ms, provider_resolve_ms, db_insert_ms, mint_tokens_ms, dispatch_ms, total_ms, and resolve_ms where reported. A server phase the serving flow doesn't have reports 0 rather than a fabricated split, so a zero there is a real measurement, not missing data. Together the two halves attribute startup latency to client, network, or server without leaving the process.

Lifecycle

end(): Promise<void>

Gracefully end the session: sends the end frame and leaves the room. The session stream finishes with reason client ended. Idempotent.

End reasons live on two surfaces with two spellings. The session stream's terminal session_ended item carries a human-readable string (client ended, client closed, …) when the client ended things itself — the server's slug when the server did. The session_ended callback event and state.disconnectReason always carry the DisconnectReason slug (client_ended, client_closed, …). Branch on the slug surfaces, not the stream string.

close(): Promise<void>

Abrupt local teardown without the graceful end frame; the stream finishes with reason client closed. Idempotent.

waitUntilReady(): Promise<void>

Resolves when the session reaches transportState === 'ready'. agent.start() already resolves at ready, so after a resolved start this is instant; it exists for code holding a session from before the start settled (the onSession callback). Rejects if the session ends before becoming ready.

getSnapshot(): RealtimeSnapshot

Shallow clone of the current normalized state:

type RealtimeSnapshot = {
  transportState: TransportState;
  agentState: AgentState;
  mediaState: MediaState;
  error: ErrorEvent | null;
};

Events — callbacks

on<E>(event: E, handler: (payload: RealtimeEventMap[E]) => void): Unsubscribe

Subscribe to a typed event. Returns an unsubscribe function.

const unsub = session.on('transcript', (event) => {
  console.log(event.role, event.text, event.isFinal);
});

Event map

See /concepts/events for payload shapes.

Events — async iteration

for await (const event of session) yields the session's events as one stream:

for await (const event of session) {
  if (event.type === 'transcript') console.log(event.role, event.text);
  if (event.type === 'session_ended') console.log('ended:', event.reason);
}

• Items are the session's own event types, switchable on type. That name is the SDK's own rather than the wire's, and where an event also has a callback it is the same name on() takes (ready, transcript, tool_call, usage, …) carrying the same payload, so ReadyEvent, ToolCallEvent and the rest fit either surface. Nine events are stream-only — the bot_* and user_*_speaking markers and tool_invocation — and have no on() key. Plus { type: 'unknown', rawType, payload } for unrecognized frame types — rawType null and rawText set when the frame could not be decoded at all — and the SDK-local terminal { type: 'session_ended', reason }. An unknown item is never terminal.
• A session_ended item is always the final one, even when the server's notice races later frames.
• Single consumer: a second concurrent next() rejects.
• The queue is bounded at 1024 events; a consumer that stops pulling drops overflow events (logged). Terminal items evict a buffered event rather than being lost.

Messaging

sendText(content: string, options?: { transcript?: boolean }): Promise<void>

Send a text turn — the agent answers it. Requires transportState === 'ready'; throws SessionStateError (not_connected) otherwise. Empty strings are silently dropped.

The sent text lands in session.transcript as its own closed user turn — an in-progress speech transcription is untouched and keeps folding into its own bubble. transcript: false keeps it out (and skips the optimistic transcript event the SDK otherwise emits after delivery), so context notes the user never typed stay out of the UI. The turn still happens — to hand the agent information without provoking a reply, use sendContext.

sendContext(content: string): Promise<void>

Give the agent context without asking it anything. Requires transportState === 'ready'; throws SessionStateError (not_connected) otherwise. Empty strings are silently dropped, and the server rejects a note longer than 4096 characters rather than truncating it.

The note rides the provider's pre-turn channel (turn_complete=false), so the model is never asked for a response: no speech, no assistant turn, no interruption of what it is currently saying — it simply knows this the next time it answers. Nothing is added to the transcript either. Use it for live application state (scroll position, selection, current record, form values); a note that arrives while the agent is speaking is delivered when it stops, and if several arrive in that window only the newest is — the latest state is the true one.

appendThinking(content: string, options?: { delegationId?: string }): Promise<void>

appendCommentary(content: string, options?: { delegationId?: string }): Promise<void>

appendInstructions(content: string, options?: { delegationId?: string }): Promise<void>

Hand text to the voice model on a GPT Live session that delegates work (delegation: 'client' or 'cosmo'). appendThinking is background the model keeps to itself and draws on when relevant; appendCommentary is something it says now, in its own words; appendInstructions changes how it behaves from here on. delegationId names the delegation_created event the text answers; without it the text steers the session as a whole. Each append is one short piece — send several as work progresses rather than one long one at the end. Requires transportState === 'ready'; throws SessionStateError (not_connected) otherwise. Empty strings are dropped; the server rejects content longer than 4096 characters.

sendImage(args: { data: string; mimeType?: string; streamId?: string }): Promise<void>

Send a single image frame as base64 JSON (data is base64-encoded bytes, not raw bytes). mimeType defaults to 'image/jpeg', streamId to 'video.input.default'. Use for one-shot captures; for continuous streams prefer startScreenShare / video tracks. See /multimodal/image-input.

ping(): Promise<void>

Send a keep-alive ping. The server replies with a pong event.

sendActivityEnd(): Promise<void>

Signal end-of-turn for manual-VAD turn-taking: the agent stops waiting for further user audio and responds to what it has. Distinct from ending the session. See /concepts/turn-taking.

Telephony

dial(phoneNumber: string, callerNumber?: string): Promise<DialResult>

Place an outbound phone call into this live session's room. The dialed party joins as a SIP participant and the agent converses with them. phoneNumber (and optional callerNumber) must be E.164 (+ then 8–15 digits). Resolves once the dial is queued (DialResult = { dialId: string }); the call rings asynchronously.

Throws DialError, whose closed code names how far the attempt got: invalid_request for a malformed number, request_failed for a request that never completed, invalid_response for a body that did not parse, and request_rejected for a server refusal — with the server's own slug on serverCode (phone_calls_disabled, minute_limit_exceeded, session_not_found, session_not_live, session_already_dialed, …). Switch on code; log serverCode, which is open-ended. SessionStateError (not_connected) when the session has not started (or already ended). Dialing needs an API key — a minted end-user token is rejected with dial_requires_api_key. See /telephony.

Usage

usage(): Promise<SessionUsage>

Fetch this session's usage summary (duration, talk time, token counts) over REST — during the session or after it ends. usageStatus on the result reports whether the detailed summary is there — 'pending' while it may still land, 'recorded' once the numbers are final, 'unavailable' when none was written and none will be; tokens is null when the provider doesn't report token usage. Throws UsageError, whose closed code is request_rejected (the server's own slug on serverCode), request_failed, invalid_response, or invalid_request when the session never started. client.getSessionUsage(sessionId) is the client-level form.

Microphone and audio output

setMuted(muted: boolean): Promise<void>

Toggle the mic — the local track and the server-side gate. Requires ready. Optimistic: state updates first and rolls back on failure, and the rejected promise carries the reason.

setOutputBlocked(blocked: boolean): void

Mark the remote-audio output as blocked (browser refused autoplay) or unblocked. Called by <RealtimeAudio /> from its play/pause listeners; drives mediaState.output.

resumeAudioPlayback(): Promise<void>

Replay all remote audio from a user gesture (the <StartAudio /> affordance) to clear a browser autoplay block. No-op when the session is not connected.

attachAudioElement(el: HTMLAudioElement | null): void

Hand the session a host-owned <audio> element for the remote bot audio; re-applied on reconnect. Pass null to detach. Idempotent.

Screen share and video

startScreenShare(): Promise<void>

Prompt for display capture, then publish the video track. Sets mediaState.screen through requesting → active. Throws on denial (screen_denied) or capture failure (screen_start_failed). Requires ready.

stopScreenShare(): Promise<void>

Unpublish the screen track and stop capture. Sets mediaState.screen to inactive. Idempotent.

getScreenShareState(): ScreenShareState

The current ScreenShareState discriminated union.

getVisionInputStatus(): { captured: boolean; message: string }

Snapshot whether fresh frames are flowing into the model's vision input, across every published video track (screen-share and camera). captured says whether at least one track is delivering frames; message is model-facing text explaining the status, ready to pass straight through as a tool result.

addVideoStream(stream: MediaStream, options?: VideoStreamOptions): Promise<VideoStreamHandle>

Publish an arbitrary MediaStream as a video track — webcam, canvas, or screen capture. Returns a handle for later removal. Requires ready; throws if the transport doesn't support video. See /multimodal/video.

removeVideoStream(streamId: VideoStreamHandle): Promise<void>

Unpublish a previously added video stream. Idempotent.

startAudioStream(stream: MediaStream): Promise<void>

Publish an arbitrary MediaStream as an audio track — a Web Audio graph (AudioContext.createMediaStreamDestination()), a decoded WAV, an <audio> element's captureStream(), or a non-default input device. Use it for audio the SDK cannot capture itself: synthetic generators, file replay, load tests, or a host with no microphone.

Publishing declares this client the session's voice (bind-input) and clears the server-side mute gate, so the agent listens to the new track. Requires ready; throws if the transport doesn't support audio streams.

A session carries one voice, so the stream takes it from the device microphone for its lifetime and stopAudioStream hands it back. Starting a second stream while one is running throws SessionStateError with code audio_publish_already_active.

stopAudioStream(): Promise<void>

Unpublish the running audio stream and hand the voice back to the microphone it displaced. Idempotent.

Advanced

registerRpcMethod(name: string, handler: (payload: string) => Promise<string>): Unsubscribe

Register a transport-level RPC method the server invokes for client-tool execution. Forwarded to this session's transport. Prefer declared client tools with handlers (/capabilities/tools) — this is the low-level escape hatch.