RealtimeClient

RealtimeClient owns one logical realtime session: the transport (mic + data channel), normalized state, and the typed event stream. Two instances can coexist without interfering — each owns its own transport, analyser nodes, and emitter.

Sessions open through the agent surface: build a persona with client.agent({...}) (or client.catalogAgent(name)), then agent.start() returns a RealtimeSession. For React apps, CosmoRealtimeProvider wraps this class.

import { RealtimeClient } from 'cosmo-ai';

const client = new RealtimeClient({
  baseUrl: 'https://app.askcosmo.ai',
  token: mintedJwt, // from mintToken() on your backend
});

const agent = client.agent({
  instructions: 'You are a concise support agent.',
  voice: 'Puck',
});
const session = await agent.start();
await session.waitUntilReady();
await session.sendText('Hello');

Constructor

new RealtimeClient(options?: RealtimeClientOptions)

RealtimeClientOptions

Provide at most one of apiKey / token — supplying both throws RealtimeCredentialError, as does apiKey without baseUrl.

Auth

mintToken(externalUserId: string): Promise<MintedToken>

Mint a short-lived end-user token. Run this on your backend with an apiKey client; hand the returned jwt to the end user's browser, which constructs new RealtimeClient({ token: jwt }). Idempotent per (workspace, externalUserId).

Throws RealtimeMintTokenError if this client has no apiKey, or the server rejects the mint.

type MintedToken = { jwt: string; expiresAt: Date };

See /production/end-user-credentials.

verify(): Promise<CredentialInfo>

Check the credential without starting a session — free, and works with either credential. Resolving at all means the credential authenticated; canStartSessions reports whether it carries realtime:use, and realtimeVoiceAvailable whether this deployment has the default voice stack configured.

Throws RealtimeVerifyError when the server rejects the credential, or with code: 'no_origin' when there is no baseUrl and no page origin. An under-scoped credential does not throw.

type CredentialInfo = {
  credential: 'api_key' | 'user_token';
  /** Null for a minted token — an end user is not told the workspace. */
  workspace: { name: string; slug: string } | null;
  scopes: string[];
  canStartSessions: boolean;
  realtimeVoiceAvailable: boolean;
  externalUserId: string | null;
};

See /auth/api-keys.

Building agents

agent(config?: AgentConfig): RealtimeAgent

Build a reusable persona — immutable; derive variants with agent.with({...}) and open runs with agent.start(). Fields left unset fall through to the server-side protocol defaults.

catalogAgent(name: string, opts?: CatalogAgentOptions): RealtimeAgent

Build an agent that runs a workspace catalog agent by machine handle; the stored config runs verbatim. Only per-run ride-alongs are accepted — no persona parameters except voice, the one cosmetic override; anything else is a type error.

See RealtimeAgent below for both config shapes.

Lifecycle

isActive(): boolean

true when a transport connection is open.

getSessionId(): string | null

Server-minted session id, available the instant the session-start POST returns (before ready); null when no session started.

getLifecycleState(): SessionLifecycleState

Formal connection lifecycle (idle → connecting → connected ↔ reconnecting → disconnected) with typed end reasons. See /concepts/lifecycle.

getSnapshot(): RealtimeSnapshot

Shallow clone of the current normalized state:

type RealtimeSnapshot = {
  transportState: TransportState;
  agentState: AgentState;
  mediaState: MediaState;
  error: RealtimeError | null;
};

waitUntilReady(): Promise<void>

Resolves when the current session reaches transportState === 'ready'. Resolves immediately if already ready; rejects if the session ends before becoming ready. Use between agent.start() and the first sendText() to avoid the RealtimeNotReadyError race.

disconnect(): Promise<void>

Gracefully end the session: sends the wire end frame and leaves the room. The session stream finishes with reason client ended. Idempotent.

close(): Promise<void>

Abrupt local teardown without the graceful wire end frame; the stream finishes with reason client closed. Idempotent.

on<E>(event: E, handler: (payload: RealtimeEventMap[E]) => void): Unsubscribe

Subscribe to a typed event. Returns an unsubscribe function.

const unsub = client.on('transcript', (event) => {
  console.log(event.role, event.text, event.isFinal);
});

Event map

See /concepts/events for payload shapes.

Messaging

sendText(content: string, options?: { transcript?: boolean }): Promise<void>

Send a text turn — the agent answers it. Requires transportState === 'ready'; throws RealtimeNotReadyError otherwise. Empty strings are silently dropped.

transcript: false skips the optimistic transcript event (role: 'user') the SDK otherwise emits after delivery, so context notes the user never typed stay out of the UI. The turn still happens — to hand the agent information without provoking a reply, use sendContext.

sendContext(content: string): Promise<void>

Give the agent context without asking it anything. Requires transportState === 'ready'; throws RealtimeNotReadyError otherwise. Empty strings are silently dropped, and the server rejects a note longer than 4096 characters rather than truncating it.

The note rides the provider's pre-turn channel (turn_complete=false), so the model is never asked for a response: no speech, no assistant turn, no interruption of what it is currently saying — it simply knows this the next time it answers. Nothing is added to the transcript either. Use it for live application state (scroll position, selection, current record, form values); a note that arrives while the agent is speaking is delivered when it stops, and if several arrive in that window only the newest is — the latest state is the true one.

sendImage(args: { data: string; mimeType?: string; streamId?: string }): Promise<void>

Send a single image frame as base64 JSON (data is base64-encoded bytes, not raw bytes). mimeType defaults to 'image/jpeg', streamId to 'video.input.default'. Use for one-shot captures; for continuous streams prefer startScreenShare / video tracks. See /multimodal/image-input.

sendPing(): Promise<void>

Send a keep-alive ping. The server replies with a pong event.

activityEnd(): Promise<void>

Signal end-of-turn for manual-VAD turn-taking: the agent stops waiting for further user audio and responds to what it has. Distinct from ending the session. See /concepts/turn-taking.

Telephony

dial(phoneNumber: string, callerNumber?: string): Promise<DialResult>

Place an outbound phone call into this live session's room. The dialed party joins as a SIP participant and the agent converses with them. phoneNumber (and optional callerNumber) must be E.164 (+ then 8–15 digits). Resolves once the dial is queued (DialResult = { dialId: string }); the call rings asynchronously.

Throws RealtimeDialError for a malformed number, an unresolvable API origin (not_dialable), or a server rejection (phone calls disabled, over the minute limit, …); RealtimeNotReadyError when no session has started. See /telephony.

Microphone and audio output

setMicMuted(muted: boolean): Promise<void>

Toggle the mic — the local track and the server-side gate. Requires ready. Optimistic: state updates first and rolls back on failure, emitting a server_error.

acquireMic(): Promise<void>

Open the mic and hold it without publishing. Rejects if permission is denied or no device is available, leaving the room untouched. Requires ready.

publishHeldMic(): Promise<void>

Publish the held mic track. Requires ready.

releaseHeldMic(): Promise<void>

Drop a held, unpublished mic track. No-op when not connected.

setOutputBlocked(blocked: boolean): void

Mark the remote-audio output as blocked (browser refused autoplay) or unblocked. Called by <RealtimeAudio /> from its play/pause listeners; drives mediaState.output.

resumeAudioPlayback(): Promise<void>

Replay all remote audio from a user gesture (the <StartAudio /> affordance) to clear a browser autoplay block. No-op when no session is connected.

setListenMuted(muted: boolean): void

Mute/unmute the remote audio the listener hears (supervisor "speaker off"). No-op when not connected.

attachAudioElement(el: HTMLAudioElement | null): void

Hand the client a host-owned <audio> element for the remote bot audio; re-applied on reconnect. Pass null to detach. Idempotent.

Screen share and video

startScreenShare(): Promise<void>

Prompt for display capture, then publish the video track. Sets mediaState.screen through requesting → active. Throws on denial (screen_denied) or capture failure (screen_start_failed). Requires ready.

stopScreenShare(): Promise<void>

Unpublish the screen track and stop capture. Sets mediaState.screen to inactive. Idempotent.

isScreenSharing(): boolean

true when mediaState.screen.kind === 'active'.

getScreenShareState(): ScreenShareState

The current ScreenShareState discriminated union.

getVisionInputStatus(): { captured: boolean; message: string }

Snapshot whether fresh frames are flowing into the model's vision input, across every published video track (screen-share and camera). The returned shape mirrors the desktop get_current_screen tool output.

addVideoStream(stream: MediaStream, options?: VideoStreamOptions): Promise<VideoStreamHandle>

Publish an arbitrary MediaStream as a video track — webcam, canvas, or screen capture. Returns a handle for later removal. Requires ready; throws if the transport doesn't support video. See /multimodal/video.

removeVideoStream(streamId: VideoStreamHandle): Promise<void>

Unpublish a previously added video stream. Idempotent.

Monitoring

connectAsMonitor(opts): Promise<MonitorSession>

connectAsMonitor(opts: {
  livekitUrl: string;
  token: string;
  roomName: string;
  sessionId: string;
}): Promise<MonitorSession>

Join an in-progress session's room as a silent supervisor. Any prior connection on this client is torn down first. The returned MonitorSession accumulates the monitored session's transcript:

Advanced

registerRpcMethod(name: string, handler: (payload: string) => Promise<string>): Unsubscribe

Register a transport-level RPC method the server invokes for client-tool execution. Forwarded to the active transport; throws RealtimeNotReadyError if no session is connected. Prefer declared client tools with handlers (/capabilities/tools) — this is the low-level escape hatch.

RealtimeAgent

The reusable persona. Build once via client.agent({...}) or client.catalogAgent(name, {...}), start any number of sessions from it, derive variants immutably. Validation (duplicate skill names, malformed hooks) throws when the agent is built, not at start().

AgentConfig

CatalogAgentOptions

Per-run ride-alongs for client.catalogAgent(name, {...}) — the stored config runs verbatim, so there are no persona fields here.

with(overrides: AgentConfig): RealtimeAgent

Derive an immutable variant: fields set here override this agent's; undefined keeps the current value. There is deliberately no way to unset a field back to the server default.

const base = client.agent({ instructions: 'You are a support agent.' });
const spanish = base.with({ voice: { speakingStyle: 'Respond in Spanish.' } });

start(opts?: SessionStartOptions): Promise<RealtimeSession>

Open one session from this persona. Resolves once the transport is connected (room joined); wait for ready before the first send. Rejects on handshake or transport failure — you never receive a session for a run that failed to start.

RealtimeSession

One live run of an agent, returned by RealtimeAgent.start(). Wraps the same engine as RealtimeClient, so the browser surface (media state, React hooks) keeps working unchanged.

const session = await agent.start();
await session.waitUntilReady();
await session.sendText('Hi');
await session.end();

Properties

Methods

Events — callbacks

session.on(event, handler) subscribes to the same typed event map as client.on (UI-normalized payloads).

Events — async iteration

for await (const event of session) yields the wire-level event stream:

for await (const event of session) {
  if (event.type === 'transcript') console.log(event.role, event.text);
  if (event.type === 'session-ended') console.log('ended:', event.reason);
}

• Items are external wire frames verbatim (ready, transcript, tool-call, …), { type: 'unknown', rawType, payload } for unrecognized frame types (never terminal), and the SDK-local terminal { type: 'session-ended', reason }.
• A session-ended item is always the final one, even when the server's notice races later frames.
• Single consumer: a second concurrent next() rejects.
• The queue is bounded at 1024 events; a consumer that stops pulling drops overflow events (logged). Terminal items evict a buffered event rather than being lost.