Garden doctor

Run the garden-doctor example — point a phone camera at a plant and talk to a doctor that locates what you ask about and draws a labeled box or a point over the live preview.

Point your phone's rear camera at a plant and talk to a doctor about it. Ask "which leaves are yellowing?" and the agent looks through the camera, answers out loud, and a labeled box lands on your screen around the leaves while it is still talking. Ask "where should I prune?" and a single point marks the spot.

It is the live-camera vision loop end to end, and the pattern to copy whenever an agent has to show the user where something is rather than describe it. The page publishes the camera to the session; the agent declares the server-side locators (detect_objects, point_at_object), which return normalized coordinates to the model; the model picks the match it is looking at and calls the SDK's renderers (cosmo_sdk_draw_box, cosmo_sdk_draw_point); and the page maps those coordinates onto its cropped, possibly mirrored preview with boxRect and pointPosition. Nothing in the app estimates a coordinate itself.

Full source: examples/typescript/garden-doctor in the examples repo.

Prerequisites

Node 18+
A credential that can mint end-user tokens: the one cosmo init stores, or a workspace API key with the user_tokens:mint scope in COSMO_API_KEY (API keys). The dev server's /token route mints with it; a key that can only start sessions cannot.
A deployment with the vision locators configured; if they are not, the session still starts and the ready event lists them under rejectedTools
For the phone: cloudflared, since mobile browsers allow camera and microphone on HTTPS only

Run it

Clone the examples repo, install, and sign in:

git clone https://github.com/socratic-ai/cosmo-ai
cd cosmo-ai/examples/typescript/garden-doctor
npm install
cosmo init

Start the dev server. Its /token route mints short-lived tokens from the stored credential, so the page never holds a key:
```
npm run dev
```
Open the printed URL on your laptop and click Start the visit. The browser asks for camera and microphone, then the session is live.
To run it on a phone, open a tunnel in a second terminal and load the printed https://….trycloudflare.com URL on the phone:
```
cloudflared tunnel --url http://localhost:7880
```
Stop the tunnel when you are done: while it is up, anyone with the URL can mint tokens against your workspace. With the credential cosmo init stores, each token lives an hour and mints are budgeted; with a workspace API key, tokens last a day and nothing throttles the route.

Declare the locator and the renderer together

The whole agent is a persona plus four tools. The two locators are zero-config server opt-ins; the two renderers are the SDK's, built once at module scope with a swappable drawing surface:

import type { AgentConfig } from 'cosmo-ai';
import { detectObjectsTool, pointAtObjectTool } from 'cosmo-ai';

import { DRAW_TOOLS } from './draw/draw_tools';
import { INSTRUCTIONS, VOICE } from './persona';

export function gardenDoctorAgent(): AgentConfig {
  return {
    instructions: INSTRUCTIONS,
    voice: VOICE,
    model: { provider: 'gemini', includeThoughts: false },
    tools: [detectObjectsTool(), pointAtObjectTool(), ...DRAW_TOOLS],
  };
}

There is no greeting: it would start the moment the session connects, before the phone's audio output is playing, and the user would hear a fragment. The doctor waits to be asked.

The renderers are declared where the session starts, which is above the component that owns the camera view. So the view registers itself as the surface while it is mounted, and until it does the model is told, in words it can say, that there is nothing to draw on:

import {
  drawBoxTool,
  drawPointTool,
  notShown,
  shown,
  type DrawBoxRequest,
  type DrawOutcome,
  type DrawPointRequest,
} from 'cosmo-ai/tool/draw';

type DrawSurface = {
  showBox: (request: DrawBoxRequest) => void;
  showPoint: (request: DrawPointRequest) => void;
};

let surface: DrawSurface | null = null;

export function setDrawSurface(next: DrawSurface | null): void {
  surface = next;
}

function draw(render: (surface: DrawSurface) => void): DrawOutcome {
  if (surface === null) {
    return notShown('the camera view is closed — ask the user to point the camera first');
  }
  render(surface);
  return shown;
}

export const DRAW_TOOLS = [
  drawBoxTool((request) => draw((s) => s.showBox(request))),
  drawPointTool((request) => draw((s) => s.showPoint(request))),
];

The locators run in the background, so the doctor keeps talking while the box lands a beat later. The persona says so, and tells the agent to find and draw straight away when asked where something is, never to name a tool out loud, and never to add a disclaimer.

Map the coordinates onto the preview

A locator's coordinates are normalized to the frame the model was shown. The preview fills the screen and crops the frame, and a front lens is mirrored in CSS while the published frame is not. The stage measures the video element and lets the SDK undo both:

const placement = useVideoPlacement(videoRef, 'fill', mirrored);

useVideoPlacement watches loadedmetadata, the element's resize event, and a ResizeObserver, and returns a VideoPlacement once the frame size and the rendered size are both known. Each overlay then draws at boxRect(request.box, placement) or pointPosition(request.point, placement). Render locator results has the full mapping story.

Marks accumulate to six and expire after twenty seconds, and the session ending wipes them, so a question answered a minute ago is not still boxed on screen.

What you should see

A full-bleed camera view with a status pill and a live caption of the doctor's latest words. Say "what's wrong with this one?" and the doctor answers in one to three sentences with a single next action. Say "show me the yellow leaves" and, while it is still speaking, labeled boxes appear around them, up to three. Say "where would you prune?" and one point appears. Point the camera at something that is not a plant and it identifies it with good humor and waits.

Troubleshooting

Issue: the doctor describes where something is but nothing is drawn. Cause: the deployment does not have the vision locators configured, so the session started without them. Solution: read rejectedTools on the ready event; each entry names the kind that was dropped and why. See Server tools.

Issue: boxes land beside the plant rather than on it. Cause: the preview's crop or mirroring is not what the placement says. Solution: pass the content mode your preview renders with ('fill' for object-fit: cover) and mirrored: true only for a front-camera preview.

Issue: starting a second visit right after ending one says the line is busy. Cause: a session cannot start twice in a short window after an unclean exit; the server answers 429. Solution: wait a minute and start again.

Issue: the phone shows the page but the camera never turns on. Cause: mobile browsers grant camera and microphone on HTTPS only, and the dev server is plain HTTP. Solution: load the app through the tunnel URL, not the LAN address.

Next steps

Render locator results — content modes, letterboxing, and selfie mirroring
Tools — the locate-then-draw pair and the honest-refusal contract
Screen tools — the same shape for a shared screen, where the server locates UI elements and your handlers highlight or click them
Share your app — the example's functions/token.ts is a deployable mint route; the README covers the Cloudflare Pages deploy