Render locator results

The vision locators (detect_objects, point_at_object) and the SDK's renderer tools (cosmo_sdk_draw_box, cosmo_sdk_draw_point) speak one coordinate space: normalized to the frame the model was shown — [0,1] on each axis, top-left origin, y increasing downward. Your preview almost never shows that frame pixel-for-pixel: the view crops it (object-fit: cover / .fill) or letterboxes it (contain / .fit), and a front-camera preview is mirrored while the published frame never is. This page covers turning a normalized box or point into something drawn where the user actually sees the thing. Declaring the renderer tools themselves is covered on Tools.

Receive a normalized box or point

A renderer handler receives coordinates already decoded and clamped to [0,1] — a DrawBoxRequest (box: {x, y, width, height}, optional label) or a DrawPointRequest (point: {x, y}, optional label). Malformed model arguments never reach your handler; out-of-range values are clamped, so a model that overshoots the frame edge still yields a drawable annotation.

import { drawBoxTool, notShown, shown } from 'cosmo-ai';
import type { DrawBoxRequest } from 'cosmo-ai';

const renderer = drawBoxTool((request: DrawBoxRequest) => {
  if (!preview.visible) return notShown('the preview is not on screen');
  preview.showBox(request.box, request.label);
  return shown;
});

from cosmo_ai.tools import DrawBoxRequest, DrawOutcome, draw_box_tool

def on_draw(request: DrawBoxRequest) -> DrawOutcome:
    if not preview.visible:
        return DrawOutcome(shown=False, reason="the preview is not on screen")
    preview.show_box(request.box, label=request.label)
    return DrawOutcome(shown=True)

renderer = draw_box_tool(on_draw)

let renderer = AgentTool.drawBoxTool { request in
    guard isPreviewOnScreen() else {
        return .notShown("the preview is not on screen")
    }
    showBox(request.box, label: request.label)
    return .shown
}

Map coordinates onto your preview

Every SDK ships the same mapping arithmetic, so you never hand-roll it — hand-rolled versions of this math are how a box ends up somewhere the model never pointed. You supply four facts: the element's size, the frame's own size, the content mode, and whether the preview is mirrored. The helper returns pixel coordinates in your preview's space.

import { boxRect, pointPosition } from 'cosmo-ai';
import type { VideoPlacement } from 'cosmo-ai';

const placement: VideoPlacement = {
  container: { width: video.clientWidth, height: video.clientHeight },
  frameSize: { width: video.videoWidth, height: video.videoHeight },
  contentMode: 'fill',
  mirrored: isSelfiePreview,
};

const rect = boxRect(request.box, placement);       // { x, y, width, height } in CSS px
const spot = pointPosition(point, placement);       // { x, y } in CSS px

from cosmo_ai.tools import Size, box_rect, point_position

rect = box_rect(
    request.box,
    container=Size(width=view_width, height=view_height),
    frame_size=Size(width=frame_width, height=frame_height),
    content_mode="fill",
    mirrored=is_selfie_preview,
)
spot = point_position(
    point,
    container=Size(width=view_width, height=view_height),
    frame_size=Size(width=frame_width, height=frame_height),
)

let rect = request.box.rect(
    in: containerSize,        // the preview view's size
    frameSize: frameSize,     // the frame's own pixel size
    contentMode: .fill,
    mirrored: isSelfiePreview
)

A degenerate frame or element (zero or negative dimensions) maps to a zero rect or point rather than NaN, so a preview that hasn't produced its first frame yet draws nothing instead of crashing your overlay math.

Handle cropping and letterboxing

contentMode mirrors the two choices object-fit and AVLayerVideoGravity both offer:

• fill (the default, the common preview case) — the frame covers the element and is cropped symmetrically on the long axis. Part of the frame is off-screen, so a mapped rect can land partly (or wholly) outside the element; clip your overlay to the element's bounds.
• fit — the whole frame is visible, letterboxed where the aspect ratios differ. The helpers offset coordinates past the letterbox bars for you, so a box never lands on the bars.

Pass the mode your preview actually renders with. Mapping with the wrong mode is off by exactly the crop or letterbox amount — a bug that looks like "the model is slightly wrong" and is really your layout description being wrong.

Handle mirrored selfie previews

A front-camera preview is conventionally mirrored so users see themselves as in a mirror — but the published frame is never mirrored, and the model measured against the published frame. Pass mirrored: true for a mirrored preview and the helpers reflect the x axis so the mark lands where the user sees the object. Skip it and every annotation lands on the wrong side of the screen. Screen-share previews and rear-camera views are not mirrored; leave the flag off there.

Parse raw requests in a custom pipeline

If you route tool traffic yourself instead of using the drawBoxTool / drawPointTool factories, the TypeScript SDK exports the decoders those factories use — parseDrawBoxRequest(args) and parseDrawPointRequest(args) return a clamped request, or null for absent or malformed coordinates. The wire tool names are exported as DRAW_BOX_TOOL_NAME (cosmo_sdk_draw_box) and DRAW_POINT_TOOL_NAME (cosmo_sdk_draw_point); Python exposes the same constants from cosmo_ai.tools, and Swift exposes DrawBoxTool.request(from:) / DrawPointTool.request(from:) alongside DrawBoxTool.name / DrawPointTool.name.

See also

• Tools — declaring the locate-then-draw pair and the honest-refusal contract.
• Screen tools — the screen-share siblings, where the server locates UI elements and your handlers highlight or click them.
• Video — how frames reach the model in the first place.