Build a voice React app

End-to-end walkthrough — Vite project, RealtimeClient + agent.start(), CosmoRealtimeProvider, mic toggle, transcript pane, tool call display.

This guide takes you from an empty directory to a working voice app in the browser. You will install cosmo-ai, construct a RealtimeClient, start a session with client.agent({...}).start(), wire the provider and audio element, render a live transcript, and display tool calls as the agent works.

Base code: sdks/cosmo-realtime/typescript/examples/realtime-page/.

Prerequisites

Node 18+
A Cosmo credential — a minted end-user token for anything you ship, or a workspace API key (cosmo_…) for local experimentation

A workspace API key is a server-side secret. For a real browser app, mint a short-lived end-user token on your backend (client.mintToken(externalUserId) with an API-key client) and construct the browser client with { token }. See End-user credentials.

1. Create the Vite project

npm create vite@latest my-voice-app -- --template react-ts
cd my-voice-app
npm install

2. Install `cosmo-ai`

npm install cosmo-ai

The package ships ESM only and re-exports its own React bindings from cosmo-ai.

3. Construct the client and start a session

RealtimeClient holds the credential and base URL. client.agent({...}) builds a reusable persona; await agent.start() opens one live session (REST session-start + room join).

// src/App.tsx
'use client';

import { useCallback, useRef, useState } from 'react';
import {
  CosmoRealtimeProvider,
  RealtimeAudio,
  MicToggle,
  useRealtimeClient,
  useTranscript,
  useToolCalls,
  useTransportState,
  RealtimeClient,
  type RealtimeClientOptions,
} from 'cosmo-ai';

export function App() {
  const [connecting, setConnecting] = useState(false);
  const clientRef = useRef<RealtimeClient | null>(null);

  const handleStart = useCallback(async (token: string) => {
    setConnecting(true);
    const opts: RealtimeClientOptions = {
      baseUrl: 'https://app.askcosmo.ai', // omit to use the page's own origin
      token,
    };
    const c = new RealtimeClient(opts);
    clientRef.current = c;
    try {
      await c
        .agent({
          instructions: 'You are a concise voice assistant.',
        })
        .start();
    } finally {
      setConnecting(false);
    }
  }, []);

  const client = clientRef.current;

  if (client == null) {
    return <LoginForm onStart={handleStart} disabled={connecting} />;
  }

  return (
    <CosmoRealtimeProvider client={client} maxTranscriptLength={50}>
      <SessionView />
    </CosmoRealtimeProvider>
  );
}

Notes:

There is no project ID to pass — the credential scopes the session server-side.
agent.start() resolves once the transport is connected; it rejects on a handshake or transport failure, so you never hold a session that failed to start.
maxTranscriptLength caps how many transcript bubbles the provider keeps in its React state (default 12). Pass Infinity to keep unbounded history.
The provider can also construct and own the client itself (<CosmoRealtimeProvider baseUrl="…" getAuthHeaders={…}>); when you pass client, the lifecycle is yours and the provider does not disconnect on unmount.

// src/main.tsx
import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { App } from './App';

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <App />
  </StrictMode>,
);

4. Render the session view

// inside src/App.tsx (continued)

function SessionView() {
  const client = useRealtimeClient();
  const transport = useTransportState();
  const transcript = useTranscript();
  const toolCalls = useToolCalls();

  const handleEnd = useCallback(() => {
    void client.disconnect();
  }, [client]);

  return (
    <div style={{ maxWidth: 640, margin: '0 auto', padding: 20, fontFamily: 'system-ui' }}>
      {/* Status bar */}
      <div style={{ display: 'flex', alignItems: 'center', gap: 12, marginBottom: 16 }}>
        <StatusBadge state={transport} />
        <MicToggle />
        <button onClick={handleEnd}>End session</button>
      </div>

      {/* Hidden audio element — agent voice plays through here */}
      <RealtimeAudio />

      {/* Transcript */}
      <TranscriptPane items={transcript} />

      {/* Tool calls */}
      {toolCalls.length > 0 && <ToolCallPane calls={toolCalls} />}
    </div>
  );
}

5. Audio element

<RealtimeAudio /> renders a hidden <audio autoPlay> and wires it to the transport. Without it the agent's voice has nowhere to play. Place it once per provider tree, typically adjacent to your session controls.

import { RealtimeAudio } from 'cosmo-ai';

// Inside your session component tree:
<RealtimeAudio onError={(err) => console.error('audio error', err)} />

Browsers (Safari, mobile Chromium) may block autoplay until a user gesture. <StartAudio /> renders a button for that gesture, shown only while playback is blocked:

import { StartAudio } from 'cosmo-ai';

<StartAudio />

For your own affordance, pass a render prop — it receives the blocked flag and a start callback to wire to a click handler:

<StartAudio>
  {({ blocked, start }) =>
    blocked ? <button onClick={() => void start()}>Tap to enable audio</button> : null
  }
</StartAudio>

6. Mic toggle

<MicToggle /> reads the media state and calls client.setMicMuted() on click. It renders an unstyled <button> — style it via className or wrap it in your own component.

import { MicToggle } from 'cosmo-ai';

<MicToggle
  label={{ muted: 'Unmute', unmuted: 'Mute' }}
  onError={(err) => toast.error('Mic toggle failed')}
/>

The button is automatically disabled when transportState !== 'ready'.

7. Transcript pane

useTranscript() returns RealtimeTranscriptItem[]. Each item has id, turnId, role ('user' | 'assistant'), text, and isFinal. Stream updates arrive as append deltas; the provider coalesces them into the existing bubble automatically.

import { useTranscript, type RealtimeTranscriptItem } from 'cosmo-ai';

function TranscriptPane({ items }: { items: RealtimeTranscriptItem[] }) {
  if (items.length === 0) {
    return <p style={{ color: '#888', fontStyle: 'italic' }}>Waiting for speech…</p>;
  }
  return (
    <div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
      {items.map((item) => (
        <div
          key={item.id}
          style={{
            padding: '6px 12px',
            borderRadius: 10,
            background: item.role === 'user' ? '#dbeafe' : '#f3f4f6',
            alignSelf: item.role === 'user' ? 'flex-end' : 'flex-start',
            maxWidth: '75%',
            opacity: item.isFinal ? 1 : 0.65,
          }}
        >
          <span style={{ fontSize: 11, textTransform: 'uppercase', color: '#555' }}>
            {item.role}
          </span>
          <p style={{ margin: '2px 0 0' }}>{item.text}</p>
        </div>
      ))}
    </div>
  );
}

8. Tool call pane

useToolCalls() returns RealtimeToolCallItem[]. Each item has toolCallId, name, status ('in_flight' | 'ok' | 'error'), and summary. The provider updates status and summary when the tool_result event arrives.

import { useToolCalls, type RealtimeToolCallItem } from 'cosmo-ai';

function ToolCallPane({ calls }: { calls: RealtimeToolCallItem[] }) {
  return (
    <section style={{ marginTop: 16 }}>
      <h3 style={{ marginBottom: 8 }}>Tool calls</h3>
      <div style={{ display: 'flex', flexDirection: 'column', gap: 4 }}>
        {calls.map((tc) => (
          <div
            key={tc.toolCallId}
            style={{ padding: '6px 10px', background: '#fef9c3', borderRadius: 8, fontSize: 13 }}
          >
            <strong>{tc.name}</strong>
            <span
              style={{
                marginLeft: 8,
                color:
                  tc.status === 'ok'
                    ? '#15803d'
                    : tc.status === 'error'
                    ? '#b91c1c'
                    : '#92400e',
              }}
            >
              [{tc.status}]
            </span>
            {tc.summary && <span style={{ marginLeft: 8 }}>{tc.summary}</span>}
          </div>
        ))}
      </div>
    </section>
  );
}

9. Status badge helper

import { type TransportState } from 'cosmo-ai';

const STATE_COLORS: Record<TransportState, string> = {
  disconnected: '#e5e7eb',
  'requesting-permission': '#fef9c3',
  connecting: '#bfdbfe',
  connected: '#bfdbfe',
  ready: '#bbf7d0',
  reconnecting: '#fed7aa',
  disconnecting: '#e5e7eb',
  failed: '#fecaca',
};

function StatusBadge({ state }: { state: TransportState }) {
  return (
    <span
      style={{
        padding: '2px 10px',
        borderRadius: 12,
        fontSize: 13,
        background: STATE_COLORS[state],
      }}
    >
      {state}
    </span>
  );
}

function LoginForm({
  onStart,
  disabled,
}: {
  onStart: (token: string) => void;
  disabled: boolean;
}) {
  const [token, setToken] = useState('');

  const handleSubmit = (e: React.FormEvent) => {
    e.preventDefault();
    if (token) onStart(token);
  };

  return (
    <form onSubmit={handleSubmit} style={{ display: 'flex', flexDirection: 'column', gap: 8, maxWidth: 400 }}>
      <label>End-user token</label>
      <input
        type="password"
        value={token}
        onChange={(e) => setToken(e.target.value)}
        placeholder="minted JWT"
        required
      />
      <button type="submit" disabled={disabled}>
        {disabled ? 'Connecting…' : 'Start session'}
      </button>
    </form>
  );
}

11. Run it

npm run dev

Open http://localhost:5173, paste a token, and click "Start session". Your browser will request microphone permission, then the agent will greet you.

Adding a client tool

The agent config carries the tool set. Declare a client-executed tool with the tool() helper (typed input via zod), and the SDK runs your handler when the agent invokes it:

import { tool } from 'cosmo-ai/tool';
import { zodInput } from 'cosmo-ai/tool/zod';
import { z } from 'zod/v4';

const getLocalTime = tool({
  name: 'get_local_time',
  description: 'Returns the local wall-clock time.',
  input: zodInput(
    z.object({
      locale: z.string().describe('BCP 47 locale tag, e.g. "en-US"').optional(),
    }),
  ),
  handler: async ({ locale }) => ({
    time: new Date().toLocaleTimeString(locale),
  }),
});

await client.agent({ tools: [getLocalTime] }).start();

See Tools for server tools, background tools, and schema rules.

Error handling

useRealtimeError() returns the most recent RealtimeError | null. It clears back to null on the next successful session start.

import { useRealtimeError } from 'cosmo-ai';

function ErrorBanner() {
  const err = useRealtimeError();
  if (!err) return null;
  return (
    <div style={{ background: '#fecaca', padding: 10, borderRadius: 6 }}>
      {err.code}: {err.message}
    </div>
  );
}

Telling the agent what the user is looking at

An agent that can see your app's state answers better questions. Push that state with sendContext — it is not a turn, so the agent doesn't reply to it, doesn't speak, and isn't interrupted mid-sentence; it just knows this the next time it answers. Nothing lands in useTranscript() either, so these never render as chat bubbles.

const client = useRealtimeClient();

useEffect(() => {
  if (transportState !== 'ready') return;
  void client.sendContext(`now on ${section.label} (section ${section.n} of ${total}).`);
}, [client, transportState, section, total]);

Anything the user can see is fair game: scroll position, the current record, a selection, unsaved form values. Push on change rather than batching — a stale note is worse than a frequent one, and a note that arrives while the agent is speaking is held until it stops.

Do not use sendText for this. That is a turn: the agent will answer it, out loud, mid-scroll.

To add screen share support, call client.startScreenShare() after transportState === 'ready'. The SDK requests getDisplayMedia, publishes the track, and updates mediaState.screen. Call client.stopScreenShare() to stop.

const client = useRealtimeClient();
const media = useMediaState();
const isSharing = media.screen.kind === 'active';

<button onClick={() => isSharing ? client.stopScreenShare() : client.startScreenShare()}>
  {isSharing ? 'Stop screen share' : 'Share screen'}
</button>

See Screen share for details.

Next steps

Handle tool calls — render a tool-call timeline
Tools — client tools, server tools, and schemas
End-user credentials — mint per-user tokens on your backend
Reconnects — surface reconnecting state in UI
Debugging — correlate logs with session IDs

Build a voice React app

On this page