Coach

A real-time presentation coach that watches the user via webcam and gives spoken feedback. Shows how to stream camera frames via addVideoStream and the Python equivalent.

The Coach app watches a user through their webcam and gives live spoken feedback on posture, eye contact, and verbal delivery. The scoring rubric ships with the agent as a skill so scoring criteria load on demand, and the examine_image server tool lets the agent take a full-resolution look at the camera when it needs fine detail.

What this app does

Captures the camera at low frame rate (1–2 fps) and publishes it as a video track the model watches.
Speaks feedback in real time using the LiveKit audio track.
Attaches the coaching rubric as a skill — loaded just-in-time when a practice run ends.
Opts into examine_image so the agent can re-read the latest camera frame at full capture resolution.

System prompt

You are a professional presentation coach. You can see the user through their camera.

Observe posture, eye contact, and vocal delivery. Give concise spoken feedback every
20–30 seconds. When the user finishes a practice run, load the scoring rubric and
score their performance.

Keep feedback positive and specific. Never comment on appearance.

TypeScript (React)

Install and setup

npm install cosmo-ai

Full component

'use client';

import { useCallback, useEffect, useRef } from 'react';
import {
  CosmoRealtimeProvider,
  RealtimeAudio,
  MicToggle,
  RealtimeClient,
  parseSkillMd,
  useTranscript,
  useToolCalls,
  useTransportState,
  type RealtimeSession,
} from 'cosmo-ai';
import {
} from 'cosmo-ai/cosmo';

const COACH_INSTRUCTIONS = `You are a professional presentation coach. …`;

export function CoachApp({ token }: { token: string }) {
  // token is a short-lived end-user JWT minted on your backend.
  const clientRef = useRef<RealtimeClient | null>(null);
  if (clientRef.current === null) {
    clientRef.current = new RealtimeClient({
      baseUrl: 'https://app.askcosmo.ai',
      token,
    });
  }
  const client = clientRef.current;
  const sessionRef = useRef<RealtimeSession | null>(null);

  useEffect(() => {
    let cancelled = false;
    void (async () => {
      const rubricMd = await fetch('/skills/scoring-rubric.md').then((r) => r.text());
      const agent = client.agent({
        instructions: COACH_INSTRUCTIONS,
        skills: [parseSkillMd(rubricMd, { defaultName: 'scoring-rubric' })],
        tools: [
          { kind: 'examine_image' },
          { kind: 'web_search' },
        ],
      });
      const session = await agent.start();
      if (cancelled) {
        void session.end();
        return;
      }
      sessionRef.current = session;
    })();
    return () => {
      cancelled = true;
      void sessionRef.current?.end();
      sessionRef.current = null;
    };
  }, [client]);

  return (
    <CosmoRealtimeProvider client={client}>
      <CoachSession sessionRef={sessionRef} />
    </CosmoRealtimeProvider>
  );
}

function CoachSession({
  sessionRef,
}: {
  sessionRef: React.RefObject<RealtimeSession | null>;
}) {
  const transport = useTransportState();
  const transcript = useTranscript({ limit: 20 });
  const toolCalls = useToolCalls();
  const cameraHandleRef = useRef<string | null>(null);
  const videoRef = useRef<HTMLVideoElement | null>(null);

  const startCamera = useCallback(async () => {
    const session = sessionRef.current;
    if (session === null || transport !== 'ready') return;

    const stream = await navigator.mediaDevices.getUserMedia({ video: true });
    if (videoRef.current) {
      videoRef.current.srcObject = stream;
    }
    // addVideoStream publishes the track to the LiveKit room; the
    // agent receives frames at the fps you specify.
    cameraHandleRef.current = await session.addVideoStream(stream, { fps: 2 });
  }, [sessionRef, transport]);

  const stopCamera = useCallback(async () => {
    const session = sessionRef.current;
    const handle = cameraHandleRef.current;
    if (session !== null && handle !== null) {
      await session.removeVideoStream(handle);
      cameraHandleRef.current = null;
    }
    if (videoRef.current?.srcObject) {
      const stream = videoRef.current.srcObject as MediaStream;
      for (const track of stream.getTracks()) track.stop();
      videoRef.current.srcObject = null;
    }
  }, [sessionRef]);

  const isReady = transport === 'ready';

  return (
    <div style={{ maxWidth: 720, margin: '0 auto', padding: 20, fontFamily: 'system-ui' }}>
      <h1>Presentation Coach</h1>

      {/* Camera preview */}
      <video
        ref={videoRef}
        autoPlay
        muted
        playsInline
        style={{ width: '100%', maxWidth: 480, borderRadius: 8, background: '#000' }}
      />

      {/* Controls */}
      <div style={{ display: 'flex', gap: 12, marginTop: 12 }}>
        <MicToggle />
        <button onClick={startCamera} disabled={!isReady}>
          Start camera
        </button>
        <button onClick={stopCamera}>Stop camera</button>
      </div>

      {/* Bot audio */}
      <RealtimeAudio />

      {/* Transcript */}
      <section style={{ marginTop: 20 }}>
        <h2>Session</h2>
        <div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
          {transcript.map((item) => (
            <div
              key={item.id}
              style={{
                padding: '6px 12px',
                borderRadius: 8,
                background: item.role === 'user' ? '#dbeafe' : '#f3f4f6',
                alignSelf: item.role === 'user' ? 'flex-end' : 'flex-start',
                maxWidth: '80%',
                opacity: item.isFinal ? 1 : 0.6,
              }}
            >
              {item.text}
            </div>
          ))}
        </div>
      </section>

      {/* Tool call timeline */}
      {toolCalls.length > 0 && (
        <section style={{ marginTop: 16 }}>
          <h2>Agent activity</h2>
          <ol style={{ listStyle: 'none', padding: 0 }}>
            {toolCalls.map((tc) => (
              <li key={tc.toolCallId} style={{ padding: '4px 0', fontSize: 13 }}>
                <strong>{tc.name}</strong>{' '}
                <span
                  style={{
                    color: tc.status === 'ok' ? '#15803d' : tc.status === 'error' ? '#b91c1c' : '#92400e',
                  }}
                >
                  [{tc.status}]
                </span>
                {tc.summary && <span style={{ marginLeft: 8, color: '#555' }}>{tc.summary}</span>}
              </li>
            ))}
          </ol>
        </section>
      )}
    </div>
  );
}

Never ship an apiKey to the browser. Mint a short-lived end-user token on your backend with an apiKey client (client.mintToken(...)) and construct the browser client with { token } — see End-user credentials.

Key points

session.addVideoStream(stream, { fps: 2 }) publishes the camera track via LiveKit (kind: 'camera' is the default). The transport delivers frames at the requested rate.
session.removeVideoStream(handle) unpublishes the track. Always stop the underlying MediaStream tracks too to release the camera indicator in the browser.
fps: 2 is enough for posture and eye contact feedback while keeping bandwidth low. Increase to fps: 5 for motion-heavy activities.
examine_image only has something to look at while a video track is published — without a frame it reports there is nothing to examine.

Python

The Python SDK can capture a camera with OpenCV and send JPEG frames with the typed send_image() helper:

import asyncio
import base64
import os

import cv2

from cosmo_ai import (
    CosmoRealtime,
    RealtimeReady,
    RealtimeSession,
    RealtimeTranscriptDelta,
    WebSearchTool,
    ExamineImageTool,
)

FRAME_INTERVAL = 0.5  # 2 fps

COACH_INSTRUCTIONS = "You are a professional presentation coach. …"


async def stream_camera(session: RealtimeSession, stop_event: asyncio.Event) -> None:
    cap = cv2.VideoCapture(0)
    try:
        while not stop_event.is_set():
            ret, frame = cap.read()
            if not ret:
                break

            # Resize to 640×480 and encode as JPEG (~30–50 KiB).
            frame = cv2.resize(frame, (640, 480))
            _, buf = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, 60])
            await session.send_image(
                data=base64.b64encode(buf.tobytes()).decode(),
                mime_type="image/jpeg",
                stream_id="camera.coach",
            )
            await asyncio.sleep(FRAME_INTERVAL)
    finally:
        cap.release()


async def run(api_key: str) -> None:
    async with CosmoRealtime(api_key=api_key) as client:
        agent = client.agent(
            instructions=COACH_INSTRUCTIONS,
            tools=[WebSearchTool(), ExamineImageTool()],
        )
        async with agent.start() as session:

            async def print_events() -> None:
                async for event in session:
                    if isinstance(event, RealtimeReady):
                        print(f"Session ready: {event.session_id}")
                    elif isinstance(event, RealtimeTranscriptDelta) and event.is_final:
                        print(f"[{event.role.value}] {event.text}", flush=True)

            printer = asyncio.create_task(print_events())

            await session.set_microphone_enabled(True)
            await session.set_speaker_enabled(True)

            stop_event = asyncio.Event()
            camera_task = asyncio.create_task(stream_camera(session, stop_event))

            print("Coaching session started. Press Enter to stop.")
            loop = asyncio.get_event_loop()
            await loop.run_in_executor(None, input)

            stop_event.set()
            await camera_task
            await session.end()
            await printer

send_image() sends discrete JPEG frames over the control channel — a different path from the browser's published video track. examine_image re-reads retained frames from published video tracks, so it pairs with addVideoStream / screen share rather than with send_image. See Image input for the trade-offs.

Server-side tools used

Tool	Role
`examine_image`	Full-resolution re-read of the latest camera frame for fine detail — is the user actually making eye contact with the lens?
`web_search`	Looks up presentation-coaching guidance when the user asks for references.

The scoring rubric itself is not a tool — it is a skill, so its full text loads only when a practice run ends.

Next steps

Build a voice React app — full React app walkthrough
Video — video tracks vs one-shot image frames
Skills — packaging the rubric as a SKILL.md
Envelope chunking — how JPEG frames are chunked and reassembled

Coach

On this page