Docs agent

Run the docs-agent example — open a PDF or a link and talk to an agent that reads it with you, built on client tools that see your live scroll position and selection.

The docs agent opens a PDF or a web page and puts a voice agent next to it that is reading along with you. Ask "what's this table saying?" about the page you're on, select a paragraph and ask for an explanation, or ask about something twenty pages away — the agent looks it up rather than guessing.

It is the clearest demonstration of client tools as a live view of your app rather than a snapshot: the reader's scroll position and text selection are read at call time, so scrolling never means restarting the session. It also shows sendContext for pushing ambient state, and a deployable token flow for shipping the page without a credential in it.

Full source: examples/typescript/docs-agent in the examples repo.

Prerequisites

Node 18+
A workspace API key with the realtime:start scope (API keys)
A microphone

Run it

Clone the examples repo and install:

git clone https://github.com/socratic-ai/cosmo-ai
cd cosmo-ai/examples/typescript/docs-agent
npm install

Start both processes — the Vite dev server, and a small backend whose only job is fetching URLs (a browser can't fetch a cross-origin page):
```
npm run server
```
```
npm run dev
```
Open the Vite URL, open a PDF or paste a link, then click Start talking and paste your API key. cp .env.example .env saves the pasting on every run.

How the agent sees the document

Both sources reduce to one model — a title plus ordered sections of text. A PDF section is a page, extracted client-side with pdfjs-dist (the file never leaves the browser); a web-page section is a heading block, fetched and sanitized server-side.

A document short enough to fit rides in the prompt whole. Anything longer, and the prompt carries only the outline — labels and counts — so a 400-page PDF costs the same at session start as a one-pager. Body text arrives through five client tools:

Tool	What it answers
`get_current_view`	"what am I looking at?" — the section on screen, plus any selection
`read_document`	the whole thing, for "what is this?" and anything document-wide
`get_section`	one section in full, by index
`search_document`	where a phrase appears, as snippets with section indices
`get_outline`	the section list, no body text

The tools close over a mutable ref instead of captured data, which is the whole trick — re-declaring tools would mean restarting the session:

/** Live state, not a snapshot: the reader scrolls and re-selects constantly,
 *  and re-declaring tools would mean restarting the session. */
export type StateRef = { current: DocumentState | null };

const getCurrentView = clientTool({
  name: 'get_current_view',
  description: 'What the reader is looking at right now …',
  input: zodInput(z.object({})),
  handler: async () => {
    const { doc, view } = requireState(ref);   // read at CALL time
    const section = doc.sections[view.sectionIndex];
    return { sectionIndex: view.sectionIndex, selection: view.selection, /* … */ };
  },
});

On top of the tools, the app pushes [reading] … notes with session.sendContext(...) whenever you scroll to a new section or select text. sendContext is the primitive for exactly this: the note lands in the model's context without becoming a turn, so the agent never speaks up about it and nothing reaches the transcript panel.

Deploy it without shipping a key

A deployed docs agent holds no Cosmo credential at all. The workspace key lives server-side in one function that trades the deployment's access password for short-lived end-user tokens, and the page consumes them through TokenSource:

const options: RealtimeClientOptions = HOSTED
  ? { token: TokenSource.endpoint('/token', { headers: () => mintHeaders(password) }) }
  : { apiKey };
const client = new RealtimeClient(options);

TokenSource.endpoint keeps a fresh token on hand and re-fetches as expiry nears; each visitor mints under a stable per-browser id, so usage meters per visitor instead of pooling under one identity. The example's build step also blanks VITE_* credentials and fails the build if anything key-shaped survives into the bundle — Vite inlines env vars, so a build made from a dev .env would otherwise ship a live key. The repo README covers the Cloudflare Pages deployment end to end; Share your app covers the same trade as a general pattern.

What you should see

Open a PDF and the agent greets you by naming the document. Ask "what is this?" and a read_document call flashes in the panel before the agent summarizes. Scroll a few pages, ask "what's this section about?", and the answer tracks where you actually are — that's get_current_view reading the ref at call time. Select a sentence and ask "explain this" and the selection text comes back in the answer.

Troubleshooting

Issue: pasted links fail to load while PDFs work. Cause: the URL-fetching backend (npm run server) isn't running — the browser can't fetch cross-origin pages itself. Solution: start npm run server alongside npm run dev.

Next steps

Handle tool calls — the client-tool contract in depth
End-user credentials — the minted-token model this example deploys with
Squat coach — a different grounding pattern: analysis computed before the call