Squat coach

Upload a video of a squat set, then talk to a coach about it. The coach can replay the exact moment it's describing: ask "show me the worst rep" and the player seeks to that rep while the agent talks you through it.

The realtime model never watches the video. Streaming footage into a live session is slow to first useful word and gives poor temporal grounding, so this example precomputes structure — a deterministic pose pipeline measures joint angles and segments reps, and an optional two-pass visual review audits and synthesizes findings — then hands the realtime session a compact index in its instructions and one play_video client tool for pulling pixels on demand. It's the pattern to copy whenever a conversation should be grounded in analysis that happened before the call.

Full source: examples/typescript/squat-coach in the examples repo.

Prerequisites

• Node 18+ and Python 3.11
• ffmpeg on PATH (overlay videos are re-encoded to H.264 so browsers can play them)
• A workspace API key with the realtime:start scope (API keys)
• Optional: a Gemini API key — it powers the visual-review stage; without it the coach works from joint-angle measurements alone
• A video of a squat set: one continuous take, full body in frame (side view works best)

Run it

1. Clone the examples repo and install both halves:

   git clone https://github.com/socratic-ai/cosmo-ai
   cd cosmo-ai/examples/typescript/squat-coach
   npm install
   
   python3.11 -m venv pipeline/venv
   pipeline/venv/bin/pip install -r pipeline/requirements.txt
   
   curl -sL -o pipeline/models/pose_landmarker_lite.task --create-dirs \
     https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/latest/pose_landmarker_lite.task

2. Copy .env.example to .env and fill in keys, or plan to paste the Cosmo key into the form at runtime.

3. Start both processes:

   npm run server   # pipeline backend on :7861
   npm run dev      # UI on :7860

4. Open http://localhost:7860, drop in a video, wait for processing (about a minute for a one-minute clip), then start the session and allow microphone access.

How the session is grounded

The processed measurements and findings are folded into the agent's instructions — per-rep numbers with their provenance, and explicit rules about what may be claimed from which source. Findings marked visual were seen but not measured, and the prompt tells the agent to speak them as observations ("your back looked like it rounded"), never as numbers. Reps the visual audit disputed are removed entirely.

The one tool is where the example earns study. play_video seeks a shared <video> element, plays up to four labelled segments in order, and — because playback outlives the tool call — reports what actually happened during the previous call rather than pretending to know about its own:

const playVideo = clientTool({
  name: 'play_video',
  description:
    'Plays one or more clips of the squat set so the user can see the moment being discussed; ' +
    "segments play in order … the result's previous_playback reports what actually played " +
    'during the PREVIOUS call, including clips the user paused or skipped.',
  input: zodInput(
    z.object({
      segments: z
        .array(
          z.object({
            t_start: z.number().min(0).finite().describe('segment start time in seconds'),
            t_end: z.number().min(0).finite().describe('segment end time in seconds'),
            label: z.string().optional().describe("short name for this segment, e.g. 'rep 5'"),
          }),
        )
        .describe('1-4 segments, played in order'),
    }),
  ),
  handler: ({ segments }) => {
    /* seek, play, and record a ledger of outcomes */
  },
});

Three decisions in the full implementation are worth copying:

• The result returns when the first clip starts, not when playback ends — a tool call that blocked for eight seconds of video would stall the conversation. What it carries instead is previous_playback: a ledger of the prior call's segments, each marked completed, interrupted (with where the user paused or scrubbed to), or error. The prompt tells the agent to read it before assuming you saw what it showed you.
• An empty segments list is a status query — it plays nothing and returns the ledger, so when a user says "I didn't see that", the agent can check what actually happened and answer with evidence instead of insisting.
• The zod schema avoids .min()/.max() on the array — those compile to minItems/maxItems, which the restricted tool-schema dialect rejects at session start. Bounds are enforced in the handler instead. See Tools for the dialect limits.

What you should see

After processing, a findings list appears with each item tagged measured or seen, and the overlay video (pose skeleton drawn on your footage) sits beside the conversation panel. Start the session and the coach opens with the highest-priority finding. Say "show me my worst rep" — the player seeks and plays a two-to-three-second window while the coach narrates it. Say "play that again" and the same window replays. If Gemini wasn't configured, the UI says the visual review was skipped and the coach talks only about what joint angles can measure.

Troubleshooting

Issue: the session says "Not connected" and there's no audio. Cause: almost always the key and the server named in the form come from different environments — the browser console shows a 401. Solution: use a key from the same backend the Server field points at.

Issue: processing succeeds but the video won't play. Cause: ffmpeg wasn't on PATH, so the OpenCV-written overlay (MPEG-4 Part 2) was never re-encoded to H.264. Solution: install ffmpeg and re-upload.

Next steps

• Docs agent — grounding through live client tools instead of precomputed instructions
• Tools — the tool-schema dialect and its limits
• Share your app — what to change before anyone else touches this (the repo README's production-hardening list is a good checklist)