Build a voice React app

End-to-end walkthrough — Vite project, RealtimeClient + agent.start(), RealtimeProvider, mic toggle, transcript pane, tool call display.

Go from an empty directory to a working voice app in the browser. You install cosmo-ai, construct a RealtimeClient, start a session with client.agent({...}).start(), wire the provider and audio element, render a live transcript, and display tool calls as the agent works.

Base code: examples/typescript/realtime-page/ in the examples repo.

If you've already done the TypeScript quickstart, the first few steps here cover the same ground. Skip ahead to Add a client tool.

Prerequisites

Node 18+
A Cosmo credential — a minted end-user token for anything you ship, or a workspace API key (cosmo_…) for local experimentation

A workspace API key is a server-side secret. For a real browser app, mint a short-lived end-user token on your backend (client.mintToken(externalUserId) with an API-key client) and construct the browser client with { token }. See End-user credentials.

1. Create the Vite project

npm create vite@latest my-voice-app -- --template react-ts
cd my-voice-app
npm install

2. Install `cosmo-ai`

npm install cosmo-ai

The package ships both ESM and CJS builds. The React bindings are a separate entry, cosmo-ai/react, so react stays out of a non-React consumer's dependency graph.

3. Construct the client and start a session

RealtimeClient holds the credential. client.agent({...}) builds a reusable persona; await agent.start() opens one live session (REST session-start + room join).

// src/App.tsx
'use client';

import { useCallback, useState } from 'react';
import {
  RealtimeClient,
  type RealtimeClientOptions,
  type RealtimeSession,
} from 'cosmo-ai';
import {
  RealtimeProvider,
  RealtimeAudio,
  MicToggle,
  useRealtimeSessionContext,
  useTranscript,
  useToolCalls,
  useTransportState,
} from 'cosmo-ai/react';

export function App() {
  const [connecting, setConnecting] = useState(false);
  const [session, setSession] = useState<RealtimeSession | null>(null);

  const handleStart = useCallback(async (token: string) => {
    setConnecting(true);
    const options: RealtimeClientOptions = { token };
    const client = new RealtimeClient(options);
    try {
      setSession(
        await client
          .agent({
            instructions: 'You are a concise voice assistant.',
          })
          .start(),
      );
    } finally {
      setConnecting(false);
    }
  }, []);

  if (session == null) {
    return <LoginForm onStart={handleStart} disabled={connecting} />;
  }

  return (
    <RealtimeProvider session={session}>
      <SessionView />
    </RealtimeProvider>
  );
}

Notes:

No project ID to pass — the credential scopes the session server-side.
agent.start() resolves once the session is ready, so every session method works immediately; it rejects on any failure to get there, so you never hold a session that failed to start.
The provider is the read side only: session lifecycle stays yours, and the provider never ends the session on unmount.
This walkthrough wires the lifecycle by hand to show the moving parts. For the common one-session-at-a-time app, useRealtimeSession packages this block — the per-run client, the start/end plumbing, the teardown on every exit path — into one hook, and its session feeds the provider.

// src/main.tsx
import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { App } from './App';

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <App />
  </StrictMode>,
);

4. Render the session view

// inside src/App.tsx (continued)

function SessionView() {
  const session = useRealtimeSessionContext();
  const transport = useTransportState();
  const transcript = useTranscript();
  const toolCalls = useToolCalls();

  const handleEnd = useCallback(() => {
    void session?.end();
  }, [session]);

  return (
    <div style={{ maxWidth: 640, margin: '0 auto', padding: 20, fontFamily: 'system-ui' }}>
      {/* Status bar */}
      <div style={{ display: 'flex', alignItems: 'center', gap: 12, marginBottom: 16 }}>
        <StatusBadge state={transport} />
        <MicToggle />
        <button onClick={handleEnd}>End session</button>
      </div>

      {/* Hidden audio element — agent voice plays through here */}
      <RealtimeAudio />

      {/* Transcript */}
      <TranscriptPane items={transcript} />

      {/* Tool calls */}
      {toolCalls.length > 0 && <ToolCallPane calls={toolCalls} />}
    </div>
  );
}

5. Mount the audio element

<RealtimeAudio /> renders a hidden <audio autoPlay> and wires it to the transport. Without it the agent's voice has nowhere to play. Place it once per provider tree, typically adjacent to your session controls.

import { RealtimeAudio } from 'cosmo-ai/react';

// Inside your session component tree:
<RealtimeAudio onError={(err) => console.error('audio error', err)} />

Browsers (Safari, mobile Chromium) may block autoplay until a user gesture. <StartAudio /> renders a button for that gesture, shown only while playback is blocked:

import { StartAudio } from 'cosmo-ai/react';

<StartAudio />

For your own affordance, pass a render prop — it receives the blocked flag and a start callback to wire to a click handler:

<StartAudio>
  {({ blocked, start }) =>
    blocked ? <button onClick={() => void start()}>Tap to enable audio</button> : null
  }
</StartAudio>

6. Add a mic toggle

<MicToggle /> reads the media state and calls session.setMuted() on click. It renders an unstyled <button> — style it with className or wrap it in your own component.

import { MicToggle } from 'cosmo-ai/react';

<MicToggle
  label={{ muted: 'Unmute', unmuted: 'Mute' }}
  onError={(err) => toast.error('Mic toggle failed')}
/>

The button is automatically disabled when transportState !== 'ready'.

7. Render the transcript pane

useTranscript() returns readonly TranscriptItem[] — the session's own coalesced conversation, one item per turn. Each item has id (a stable render key), role ('user' | 'assistant'), text, and isFinal; while isFinal is false the turn is still in progress. The full conversation is kept; pass useTranscript({ limit: 50 }) when a view only needs the tail.

import { type TranscriptItem } from 'cosmo-ai';
import { useTranscript } from 'cosmo-ai/react';

function TranscriptPane({ items }: { items: readonly TranscriptItem[] }) {
  if (items.length === 0) {
    return <p style={{ color: '#888', fontStyle: 'italic' }}>Waiting for speech…</p>;
  }
  return (
    <div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
      {items.map((item) => (
        <div
          key={item.id}
          style={{
            padding: '6px 12px',
            borderRadius: 10,
            background: item.role === 'user' ? '#dbeafe' : '#f3f4f6',
            alignSelf: item.role === 'user' ? 'flex-end' : 'flex-start',
            maxWidth: '75%',
            opacity: item.isFinal ? 1 : 0.65,
          }}
        >
          <span style={{ fontSize: 11, textTransform: 'uppercase', color: '#555' }}>
            {item.role}
          </span>
          <p style={{ margin: '2px 0 0' }}>{item.text}</p>
        </div>
      ))}
    </div>
  );
}

8. Render the tool call pane

useToolCalls() returns RealtimeToolCallItem[]. Each item has toolCallId, name, status ('in_flight' | 'ok' | 'error'), and summary. The provider updates status and summary when the tool_result event arrives.

import { useToolCalls, type RealtimeToolCallItem } from 'cosmo-ai/react';

function ToolCallPane({ calls }: { calls: RealtimeToolCallItem[] }) {
  return (
    <section style={{ marginTop: 16 }}>
      <h3 style={{ marginBottom: 8 }}>Tool calls</h3>
      <div style={{ display: 'flex', flexDirection: 'column', gap: 4 }}>
        {calls.map((tc) => (
          <div
            key={tc.toolCallId}
            style={{ padding: '6px 10px', background: '#fef9c3', borderRadius: 8, fontSize: 13 }}
          >
            <strong>{tc.name}</strong>
            <span
              style={{
                marginLeft: 8,
                color:
                  tc.status === 'ok'
                    ? '#15803d'
                    : tc.status === 'error'
                    ? '#b91c1c'
                    : '#92400e',
              }}
            >
              [{tc.status}]
            </span>
            {tc.summary && <span style={{ marginLeft: 8 }}>{tc.summary}</span>}
          </div>
        ))}
      </div>
    </section>
  );
}

9. Add a status badge helper

import { type TransportState } from 'cosmo-ai';

const STATE_COLORS: Record<TransportState, string> = {
  disconnected: '#e5e7eb',
  'requesting-permission': '#fef9c3',
  connecting: '#bfdbfe',
  connected: '#bfdbfe',
  ready: '#bbf7d0',
  reconnecting: '#fed7aa',
  disconnecting: '#e5e7eb',
  failed: '#fecaca',
};

function StatusBadge({ state }: { state: TransportState }) {
  return (
    <span
      style={{
        padding: '2px 10px',
        borderRadius: 12,
        fontSize: 13,
        background: STATE_COLORS[state],
      }}
    >
      {state}
    </span>
  );
}

function LoginForm({
  onStart,
  disabled,
}: {
  onStart: (token: string) => void;
  disabled: boolean;
}) {
  const [token, setToken] = useState('');

  const handleSubmit = (e: React.FormEvent) => {
    e.preventDefault();
    if (token) onStart(token);
  };

  return (
    <form onSubmit={handleSubmit} style={{ display: 'flex', flexDirection: 'column', gap: 8, maxWidth: 400 }}>
      <label>End-user token</label>
      <input
        type="password"
        value={token}
        onChange={(e) => setToken(e.target.value)}
        placeholder="minted JWT"
        required
      />
      <button type="submit" disabled={disabled}>
        {disabled ? 'Connecting…' : 'Start session'}
      </button>
    </form>
  );
}

11. Run it

npm run dev

Open http://localhost:5173, paste a token, and select "Start session".

What you should see

The browser asks for microphone permission. Once you grant it, the status badge moves from connecting to ready and the agent greets you out loud. Speak, and your words appear in the transcript pane as transcript events stream in; the agent's reply follows in the same pane. Invoking a tool adds an entry to the tool call pane.

A badge stuck on connecting means ready never arrived — see Debugging.

Add a client tool

The agent config carries the tool set. Declare a client-executed tool with the tool() helper (typed input from zod), and the SDK runs your handler when the agent invokes it. zod is an optional peer dependency — npm install cosmo-ai doesn't pull it in, so install it alongside: npm install zod.

import { clientTool } from 'cosmo-ai/tool';
import { zodInput } from 'cosmo-ai/tool/zod';
import { z } from 'zod/v4';

const getLocalTime = clientTool({
  name: 'get_local_time',
  description: 'Returns the local wall-clock time.',
  input: zodInput(
    z.object({
      locale: z.string().describe('BCP 47 locale tag, e.g. "en-US"').optional(),
    }),
  ),
  handler: async ({ locale }) => ({
    time: new Date().toLocaleTimeString(locale),
  }),
});

await client.agent({ tools: [getLocalTime] }).start();

See Tools for server tools, background tools, and schema rules.

Handle errors

useRealtimeError() returns the most recent error, or null, and the value is the error itself rather than a summary of it. It clears back to null on the next successful session start.

The provider above mounts only once start() has resolved, so this hook reports what goes wrong on a live session: the server's ErrorEvent, carrying its own code and fatal. A start that never produced a session is caught where you called start() — as SessionStartError or AudioUnavailableError. To render both through one banner, pass onSession to start() and mount the provider with the session it hands you, which arrives before the connect begins. Every value carries code and message.

import { useRealtimeError } from 'cosmo-ai/react';

function ErrorBanner() {
  const err = useRealtimeError();
  if (!err) return null;
  return (
    <div style={{ background: '#fecaca', padding: 10, borderRadius: 6 }}>
      {err.code}: {err.message}
    </div>
  );
}

Tell the agent what the user is looking at

An agent that can see your app's state answers better questions. Push that state with sendContext — it's not a turn, so the agent doesn't reply to it, doesn't speak, and isn't interrupted mid-sentence; it just knows this the next time it answers. Nothing lands in useTranscript() either, so these never render as chat bubbles.

const session = useRealtimeSessionContext();

useEffect(() => {
  if (session === null || transportState !== 'ready') return;
  void session.sendContext(`now on ${section.label} (section ${section.n} of ${total}).`);
}, [session, transportState, section, total]);

Anything the user can see is fair game: scroll position, the current record, a selection, unsaved form values. Push on change rather than batching — a stale note is worse than a frequent one, and a note that arrives while the agent is speaking is held until it stops.

Do not use sendText for this. That's a turn: the agent will answer it, out loud, mid-scroll.

To add screen share support, call session.startScreenShare() after transportState === 'ready'. The SDK requests getDisplayMedia, publishes the track, and updates mediaState.screen. Call session.stopScreenShare() to stop.

const session = useRealtimeSessionContext();
const media = useMediaState();
const isSharing = media.screen.kind === 'active';

<button onClick={() => isSharing ? session?.stopScreenShare() : session?.startScreenShare()}>
  {isSharing ? 'Stop screen share' : 'Share screen'}
</button>

See Screen share for details.

Next steps

Handle tool calls — render a tool-call timeline
Tools — client tools, server tools, and schemas
End-user credentials — mint per-user tokens on your backend
Reconnects — surface reconnecting state in UI
Debugging — correlate logs with session IDs

Build a voice React app

On this page