RealtimeClient (Swift)

The Swift SDK is three objects, one per concern:

• RealtimeClient — how to reach Cosmo: credential, endpoint, timeouts. A Sendable struct; construct once and reuse across calls and sessions.
• RealtimeAgent — the persona/configuration of the model on the other end: instructions, model, voice, tools, skills. Independent of any one run and reusable across them.
• RealtimeSession — one live run: a Swift actor exposing a single typed event stream plus every mid-call send. Single-attempt — once it ends (by end(), by the server, or by a transport failure) it's terminal; start another from the same agent to reconnect.

import CosmoRealtime

let client = RealtimeClient(apiKey: "cosmo_your_api_key")
let agent = try client.agent(instructions: "You are a terse assistant.")
let session = try await agent.start()

for try await event in session.events {
    switch event {
    case .ready(let ready): print("live, session:", ready.sessionId)
    case .transcript(let delta) where delta.isFinal && !delta.text.isEmpty:
        print("[\(delta.role)] \(delta.text)")  // completed turns, once each
    case .sessionEnded(let ended): print("over:", ended.reason ?? "")
    default: break
    }
}

RealtimeClient

Four initializers, one per credential form. Each takes the same trailing parameters and defaults:

public struct RealtimeClient: Sendable {
    public init(apiKey: String, baseURL: URL? = nil, connectTimeout: TimeInterval = 30,
                requestTimeout: TimeInterval = 45, verifyTLS: VerifyTLS = .auto,
                transport: RealtimeClient.Transport = .webrtc)
    public init(token: String, ...)
    public init(tokenSource: TokenSource, ...)
    public init(connectTimeout: TimeInterval = 30, requestTimeout: TimeInterval = 45,
                verifyTLS: VerifyTLS = .auto) throws
}

Which one you reach for is a deployment decision. apiKey is workspace-scoped and server-side only — it opens sessions and can mint end-user tokens, so never embed it in a distributed app. token is a minted per-user JWT, safe to ship in a device or browser: it opens sessions but cannot mint (see End-user credentials); a cosmo_… API key passed here traps at construction — pass it as apiKey:, or mint a token. tokenSource hands the SDK your minting endpoint so it fetches and refreshes that JWT itself.

The zero-credential form — try RealtimeClient() — reads COSMO_API_KEY from the environment, else the cosmo login credentials file (COSMO_CREDENTIALS_FILE or ~/.cosmo/credentials, profile from COSMO_PROFILE). A file credential brings its own base_url along, since a stored key is only valid against the backend that issued it. Throws CredentialsError when nothing resolves, the file is unusable, the stored key expired, or the credential was supplied in a way the SDK refuses to send. code is a closed CredentialsErrorCode: noCredential, profileNotFound, fileInvalid, expired, baseURLMismatch, conflictingCredentials, apiKeyInTokenSlot, insecureBaseURL. conflictingCredentials and apiKeyInTokenSlot are declared but unreachable in Swift — the initializers make passing both credentials inexpressible, and the token-slot guard is a fatalError — so a switch written against another SDK ports unchanged. See API keys.

The WebSocket carrier sends PCM audio and the session protocol over one socket, with no media room, worker or UDP. It runs on macOS and iOS, reusing AVAudioPCMBuffer, the typed event stream and ordinary client-tool handlers. The microphone is echo-cancelled through Apple's platform voice processing, with gain control disabled and — on macOS 14 and iOS 17 or later — other-app ducking held at its minimum; expect some residual echo on speakers, and see the platform guide for what voice processing does to other apps' audio. A dropped socket ends the session; camera, screen share, byte streams, background client tools, dial and usage reads remain unavailable on this local lane.

On iOS the SDK configures and activates the shared AVAudioSession for the call, unless the app took that over with setAutomaticAudioSessionManagement(enabled: false) — see Swift platform behavior. A plain ws:// address is accepted for loopback only; a remote server needs wss://.

TokenSource

A credential that fetches — and keeps fresh — a minted end-user token, so a client built with init(tokenSource:) stays valid for the life of the process with no refresh code in the app. The cached token re-fetches when under 60 seconds of life remain, and a session start rejected with HTTP 401 drops the cache. Two constructors; there is no public initializer:

public static func endpoint(_ url: URL, headers: [String: String] = [:]) throws -> TokenSource
public static func endpoint(
    _ url: URL,
    headers: @escaping @Sendable () async throws -> [String: String]
) throws -> TokenSource
public static func custom(
    _ fetchToken: @escaping @Sendable () async throws -> MintedToken
) -> TokenSource

endpoint POSTs url (empty JSON body) and reads { jwt, expires_at } — the wire shape of POST /api/v1/external/auth/token, so a backend that forwards a mint response qualifies (the serialized expiresAt spelling is accepted too). headers carry the app's own auth — a static dictionary, or a closure resolved per fetch for a rotating credential (the callback form Python and TypeScript also take). Failures throw TokenSourceError, whose code is requestFailed, requestRejected, invalidResponse, or fetcherFailed; on a rejection serverCode carries the server's own slug when the body parses, else a synthetic http_<status>. A plain-http URL to a non-loopback host is refused at construction. Redirects are refused. custom wraps any async function returning a MintedToken.

agent(...)

public func agent(
    instructions: String? = nil,
    model: RealtimeModel? = nil,
    voice: VoiceConfig? = nil,
    audio: AudioConfig? = nil,
    tools: [AgentTool] = [],
    interruptionSensitivity: InterruptionSensitivity? = nil,
    greeting: String? = nil,
    skills: [Skill] = [],
    mcp: [McpStdioServer]? = nil,
    hooks: [Hook]? = nil,
    plugins: [Plugin] = []
) throws -> RealtimeAgent

Build an inline RealtimeAgent. Fields left nil fall back to the protocol's server-side defaults. It throws on duplicate skill names — when the agent is built, not mid-call.

Plugins bundle instructions, skills, tools, and hooks for inline agents. Pass plugins at construction; the built agent exposes the expanded contributions. See Plugins for the bundle fields, merge order, and PluginError codes.

catalogAgent(...)

public func catalogAgent(
    _ name: String,
    inputs: [String: String]? = nil,
    voice: VoiceConfig? = nil,
    tools: [AgentTool] = [],
    mcp: [McpStdioServer]? = nil,
    hooks: [Hook]? = nil
) -> RealtimeAgent

Run a workspace catalog agent by its machine handle (lowercase [a-z0-9-]) — the server resolves name at session start and runs the stored config verbatim. Only per-run ride-alongs are accepted: inputs (values for the agent's declared template placeholders), voice (cosmetic override), tools / mcp (client-executed declarations), and hooks (client hooks only — server hooks aren't accepted here). Stored-config fields like instructions have no parameter, so the illegal combination is a compile error.

verify()

public func verify() async throws -> CredentialInfo

Check this client's credential without starting a session (GET realtime/verify) — a free preflight for a launch-time check or a CI smoke test. It confirms the credential authenticates against the backend the client was built for, and separates the failure modes a first session would conflate: under-scoped (CredentialInfo.canStartSessions is false) versus a deployment with no default voice stack (CredentialInfo.realtimeVoiceAvailable is false). Throws VerifyError — .rejected(code:detail:), .transport(message:), or .invalidResponse(message:); an under-scoped credential is not an error. CredentialInfo, WorkspaceInfo, and CredentialKind are declared in CosmoRealtime — see Wire types.

sessionUsage(sessionId:)

public func sessionUsage(sessionId: String) async throws -> SessionUsage

Fetch a session's usage summary (GET sessions/{id}/usage) — duration, talk time, and token counts in provider-reported units. RealtimeSession.usage() is the session-level form: it uses the session's own id and works during the session or after it ends. usageStatus on the result reports whether the detailed summary is there — .pending while it may still land, .recorded once the numbers are final, .unavailable when none was written and none will be; tokens is nil when the provider doesn't report token usage. Throws UsageError — .rejected(code:detail:), .transport(message:), or .invalidResponse(message:). SessionUsage, SessionTokenUsage, SessionStatus, and UsageStatus are declared in CosmoRealtime — see Wire types.

mintToken(_:ttlSeconds:)

Run this server-side: minting needs a workspace API key, which never belongs in a shipped app.

import CosmoRealtime

let minted = try await client.mintToken("user-42")
// minted.jwt, minted.expiresAt, minted.tokenId

mintToken(_:ttlSeconds:) POSTs auth/token with an .apiKey credential; idempotent per (workspace, externalUserId). ttlSeconds (60–86400) shortens the 24-hour default lifetime. MintedToken carries jwt, expiresAt, and tokenId — the server-side revocation handle (DELETE auth/token/{token_id}); keep it on your server, the device only needs jwt. Throws MintTokenError, whose code is missingApiKey (this client holds no api key — refused before the request goes out), requestFailed, invalidResponse, or requestRejected; on a rejection serverCode carries the server's own slug, or the http_<status> synthetic when the response carried none.

Connection tracing

public static func installConnectTracing()

Routes connect-latency spans into the socratic.cosmo-realtime os_log subsystem. Call once at app launch, before any session starts. Setting COSMO_REALTIME_LIVEKIT_LOG (debug / info / warning / error) additionally enables the transport's verbose internal logging. See Debugging.

Past sessions

Session history and provider capabilities live on the REST API, not this client: list, fetch, and delete recorded sessions, and discover valid model values at runtime via the capabilities route. Call them with your HTTP client and workspace credential — see Sessions REST.

RealtimeAgent

The persona, built by client.agent(...) or client.catalogAgent(...) and reused unchanged across runs. Every parameter of those factories is readable and settable on the value.

public struct RealtimeAgent: Sendable {
    public var name: String?
    public var inputs: [String: String]?
    public var instructions: String?
    public var model: RealtimeModel?
    public var voice: VoiceConfig?
    public var audio: AudioConfig?
    public var tools: [AgentTool]
    public var interruptionSensitivity: InterruptionSensitivity?
    public var greeting: String?
    public var skills: [Skill]
    public let mcp: [McpStdioServer]?
    public var hooks: [Hook]?
}

name and inputs are set only on a catalog agent; the rest describe an inline one. Skill is { name, description, body } with a public initializer; McpStdioServer is { name, command, args, env, cwd }, and [McpStdioServer].configFile(_:) reads one from a .mcp.json.

start(...)

public func start(
    resumeSessionId: String? = nil,
    maxSessionSeconds: Int? = nil,
    storeRecording: Bool? = nil,
    storeAudio: Bool? = nil,
    storeTranscript: Bool? = nil,
    storeVideo: Bool? = nil,
    micMuted: Bool = false,
    rpcHandlers: [String: ClientToolHandler] = [:]
) async throws -> RealtimeSession

Open one run of this agent: one REST session-start plus the media-transport join. Returns once the session is ready — the server's handshake has landed, so every method on the returned session works immediately. Throws on any failure to get there: a room that closes before ready throws .handshakeFailed with the synthetic status: 0 (the server's pre-close error frame supplies code and detail when one arrived, else handshake_disconnect); a handshake that never arrives is torn down after a bounded wait and throws .readyTimeout; cancelling the calling task tears the session down and throws CancellationError. Every parameter is per-run — the values that differ between two runs of the same persona.

Throws SessionStartError on rejection or transport failure, or AudioUnavailableError when the microphone it publishes at join will not open. Both conform to RealtimeError, and neither is the other — catch RealtimeError to take both, or the two types separately.

At start, tool names merge first-occurrence-wins in order: agent tools → the skill-loader tool → MCP. The session owns any MCP connections the agent opened: end(), or an end from any other cause, tears them down with it.

A started session exposes session.sessionId (String?, set once the start succeeds), session.connectTimings (SessionConnectTimings), and session.transcript ([TranscriptItem], an actor read — await session.transcript) — the coalesced conversation so far, one item per turn (id, role, text, isFinal), folded by the session from its own transcript stream; it survives end(). See /concepts/transcripts.

connectTimings carries the client-measured phases — wsMs (REST session-start), roomMs (LiveKit join), micMs (mic publish, 0 for a muted join), totalConnectMs, readyMs (to the ready event) — plus serverTimings, the server's own breakdown: versionCheckMs, projectCheckMs, providerResolveMs, dbInsertMs, mintTokensMs, dispatchMs, totalMs, and resolveMs where reported. Every field is nil before the corresponding phase completes; serverTimings is nil on a backend that doesn't report it. A server phase the serving flow doesn't have reports 0 rather than a fabricated split, so a zero there is a real measurement, not missing data.

Together the two halves attribute startup latency to client, network, or server. On the prepared-room fast path wsMs and roomMs overlap, so they do not sum to totalConnectMs. readyMs is measured from the same instant as wsMs. Once the agent is live, the client reports its phases to the session so the server can record the whole waterfall against it.

Agent configuration types

The values the agent factories take. Each is a top-level type in CosmoRealtime.

VoiceConfig

public struct VoiceConfig: Sendable, Equatable {
    public init(name: String? = nil, speakingStyle: String? = nil)
}

An empty VoiceConfig stays off the wire entirely.

AudioConfig

public struct AudioConfig: Sendable, Equatable {
    public init(output: Bool? = nil, noiseCancellation: NoiseCancellation? = nil)
}

let agent = try client.agent(
    instructions: "You are a terse voice assistant.",
    voice: VoiceConfig(name: "Puck", speakingStyle: "Warm and unhurried."),
    audio: AudioConfig(noiseCancellation: .denoise)
)

RealtimeModel

Each knob is honored only by its provider: a provider case carries that provider's block — the same GeminiModel / OpenAIModel / OpenAIMiniModel / OpenAILiveModel / GrokModel types the other SDKs publish — so a model that disagrees with its knobs is unrepresentable. modelId: pins the concrete model within the provider; leave it unset to run that provider's default, and a modelId: belonging to another provider is rejected at session start. .gemini(GeminiModel(modelId: "gemini-3.8-live")) runs Gemini 3.8 Live, which takes no thinkingLevel: setting one fails session start with thinking_level_unsupported. .id is the string form — a provider family alias or a concrete model id, resolved server-side, carrying no knobs. It takes no string literal shorthand: spell it .id("gemini-live").

public enum RealtimeModel: Sendable, Equatable {
    case id(String)
    case gemini(GeminiModel)
    case openai(OpenAIModel)
    case openaiMini(OpenAIMiniModel)
    case openaiLive(OpenAILiveModel)
    case grok(GrokModel)
}

public struct GeminiModel: Sendable, Equatable {
    public enum TurnDetection: Sendable, Equatable { case cosmoVad, serverVad }
    public init(modelId: String? = nil,
                temperature: Double? = nil, maxOutputTokens: Int? = nil,
                thinkingLevel: ThinkingLevel? = nil, includeThoughts: Bool? = nil,
                turnDetection: TurnDetection? = nil,
                endOfSpeechSensitivity: EndOfSpeechSensitivity? = nil,
                silenceDurationMs: Int? = nil, prefixPaddingMs: Int? = nil,
                cosmoVad: CosmoVadConfig? = nil,
                toolResponsePolicy: GeminiToolResponsePolicy? = nil,
                toolResponseOverrides: [String: GeminiToolResponsePolicy]? = nil)
}

public struct OpenAIModel: Sendable, Equatable {
    public enum TurnDetection: Sendable, Equatable { case semanticVad, serverVad }
    public init(modelId: String? = nil, turnDetection: TurnDetection? = nil,
                eagerness: SemanticEagerness? = nil,
                silenceDurationMs: Int? = nil, prefixPaddingMs: Int? = nil)
}

public struct OpenAIMiniModel: Sendable, Equatable {
    public init(modelId: String? = nil)
}

public struct OpenAILiveModel: Sendable, Equatable {
    public init(modelId: String? = nil, responsesModel: String? = nil,
                responsesInstructions: String? = nil,
                reasoningEffort: OpenAILiveReasoningEffort? = nil,
                verbosity: OpenAILiveVerbosity? = nil,
                toolChoice: OpenAILiveToolChoice? = nil,
                parallelToolCalls: Bool? = nil, maxOutputTokens: Int? = nil,
                serviceTier: OpenAILiveServiceTier? = nil,
                delegation: OpenAILiveDelegation? = nil)
}

public struct GrokModel: Sendable, Equatable {
    public enum TurnDetection: Sendable, Equatable { case serverVad }
    public init(modelId: String? = nil, turnDetection: TurnDetection? = nil,
                silenceDurationMs: Int? = nil, prefixPaddingMs: Int? = nil)
}

public struct CosmoVadConfig: Sendable, Equatable {
    public init(pauseMs: Int? = nil, prefixMs: Int? = nil, maxHoldMs: Int? = nil)
}

ThinkingLevel wire values: minimal, low, medium, high (Gemini only). EndOfSpeechSensitivity: low, high (Gemini only). SemanticEagerness: low, medium, high, auto (OpenAI semanticVad only). OpenAILiveReasoningEffort: minimal, low, medium, high; OpenAILiveVerbosity: low, medium, high; OpenAILiveToolChoice: auto, required, none; OpenAILiveServiceTier: auto, default, flex, priority (GPT Live only — every OpenAILiveModel knob configures the backend Responses model tool calls are delegated to; GPT Live itself owns its turn-taking and has no detector knobs). Each block's TurnDetection lists exactly the detectors its provider offers, so an illegal pairing does not typecheck.

Gemini's turnDetection selects the end-of-turn detector: nil and .cosmoVad are Cosmo's semantic detector, which classifies whether the utterance reads as finished — tuned by the block's cosmoVad field (pauseMs is the silence that triggers the inference, prefixMs the audio kept from before speech was detected, maxHoldMs the total silence after which the turn ends regardless). .serverVad opts the session into the provider's fixed silence window, which is what endOfSpeechSensitivity, silenceDurationMs and prefixPaddingMs tune — they are unread under the semantic detector. The OpenAI turn detectors bind the same decision on OpenAI; Grok runs the fixed silence window only, so its two knobs sit alone on the block — see Turn-taking.

AgentTool

public enum AgentTool: Sendable, Equatable {
    case client(name: String, description: String,
                parameters: [String: JSONValue], handler: ClientToolHandler? = nil)
    case backgroundClient(name: String, description: String,
                          parameters: [String: JSONValue], handler: BackgroundClientToolHandler)
    case webSearch
    case examineImage
    case detectObjects
    case pointAtObject
    case endCall
    case speakerLog
    case sdkClient(SDKClientTool)
    case screenLocate(capture: ScreenCaptureHandler)
}

public typealias ClientToolHandler =
    @Sendable ([String: JSONValue]) async throws -> [String: JSONValue]

public typealias ScreenCaptureHandler =
    @Sendable (ScreenCaptureRequest) async throws -> ScreenCapture

• .client — executed by this client over LiveKit RPC. parameters is the JSON Schema for the arguments (restricted dialect, top-level type: "object"). A tool carries the handler that runs it, so a declared tool that cannot run is unrepresentable; a server-orchestrated RPC method the agent never sees in its tool list registers through start(rpcHandlers:) instead.
• .backgroundClient — same wire shape; the handler receives a ClientToolJob, acks immediately (job.ack) and delivers later (job.complete / job.fail) so long-running work doesn't block the voice turn.
• .webSearchTool() / .examineImageTool() / .detectObjectsTool() / .pointAtObjectTool() / .endCallTool() / .speakerLogTool() — zero-config opt-ins to server-executed tools; the server owns the model-facing declaration. .endCallTool() lets the agent hang up the call itself (in-call tools).
• .speakerLogTool() — enable the room's speaker-labelled transcript and the model's cosmo_who_said_what lookup. See Speaker diarization for setup.
• .sdkClient — a client tool the SDK ships itself (SDKClientTool has no public initializer; the case is produced by SDK factories such as AgentTool.drawBoxTool(onDraw:), .drawPointTool(onDraw:), .screenClickElementTool, .screenHighlightElementTool, .screenHighlightBoxTool).
• .screenLocate — the one server-tool opt-in that takes configuration: a capture handler producing the screenshot and element list the locator grounds against. Built with AgentTool.screenLocateTool(capture:); the handler receives a ScreenCaptureRequest (carrying no options today) and may throw ScreenCaptureUnavailable to decline benignly.

The cosmo_sdk_ name prefix is reserved for the client tools the SDK ships: a caller tool declared under it throws .invalidPayload at start.

Typed tool definitions — clientTool

Build a .client tool from a Decodable argument type and a ToolSchema, validated at definition time (name pattern ^[a-z][a-z0-9_]{2,63}$, description ≤ 2048 chars):

public static func define<Args: Decodable & Sendable>(
    name: String,
    description: String,
    input: ToolSchema,
    handler: @escaping @Sendable (Args) async throws -> [String: JSONValue]
) throws -> AgentTool

public static func defineBackground<Args: Decodable & Sendable>(
    name: String,
    description: String,
    input: ToolSchema,
    handler: @escaping @Sendable (Args, ClientToolJob) async throws -> Void
) throws -> AgentTool

An invalid name or description, or a schema the restricted dialect can't express, throws ToolDefinitionError; code is a closed ToolDefinitionErrorCode naming which rule was broken. The model's arguments failing validation at call time surfaces ToolInputValidationError, whose issues are ToolInputIssue values — path, code, constraint — built from schema-derived fields only, so submitted values never appear. ToolSchemaConsistencyCheck.verify(input:decodesInto:) asserts in tests that a schema and its Args type agree, throwing ToolDefinitionError with code schemaTypeMismatch when they don't.

ClientToolJob

The handle a background tool's handler receives, one per invocation. It is not a kind of tool: ack releases the RPC reply so the session keeps moving, and the terminal call delivers the outcome whenever the work lands.

public actor ClientToolJob {
    public let jobId: String
    public let toolName: String
    public var acked: Bool { get }

    public func ack(_ note: String = "")
    public func complete(result: [String: JSONValue]? = nil, summary: String? = nil) async throws
    public func fail(error: String) async throws
}

public typealias BackgroundClientToolHandler =
    @Sendable ([String: JSONValue], ClientToolJob) async throws -> Void

Both terminal calls are idempotent once delivered, and both throw if the publish fails — a throw leaves the job unsettled so you can retry rather than silently losing the result. A terminal call after the session has closed is dropped and logged. Completing or failing without having acked acks first, so the worker can still resolve the call.

A handler that returns without acking is answered inline as an error; one that acks and returns without a terminal call is failed for you. A throw before the ack becomes the call's error reply, and a throw after it becomes fail(error:).

PostToolUse fires at the terminal signal, not at the ack, so a hook observes the real outcome.

RealtimeSession

One live run, returned by agent.start(...). The sections below cover its whole surface: the event and state streams, the mid-call sends, the media publishes, and the outbound dial.

public static let maxImageBase64Length = 12_000_000

SDK identity constants

public let sdkName = "cosmo-swift-sdk"
public let sdkVersion = "0.8.1"

Module-scope constants, reached as sdkName and sdkVersion after import CosmoRealtime. They are the package identity, stamped on the session-config start payload and sent as an X-Cosmo-SDK: cosmo-swift-sdk/<version> header on every Cosmo REST call.

RealtimeSession.sdkName and RealtimeSession.sdkVersion still resolve to the same values and are deprecated; move to the module-scope constants.

Event stream

public nonisolated let events: AsyncThrowingStream<RealtimeSessionEvent, Error>

Typed server events in arrival order. Every terminal path of a live session ends with a locally synthesized .sessionEnded as the final element, after which the sequence finishes — the stream doesn't throw; start failures throw from start instead.

Payload types (ReadyEvent, TranscriptDeltaEvent, ToolCallEvent, …) are declared in CosmoRealtime as top-level types — the same symbol names Python and TypeScript publish.

state and onStateChange

public var state: SessionState   // current value; terminal state stays readable after the end

let onStateChange: @Sendable (SessionState) -> Void = { state in … }
let session = try await agent.start(onStateChange: onStateChange)

The session state machine, shared across the Cosmo SDKs and distinct from the application-level .ready event. The onStateChange handler passed to agent.start receives every transition from .idle on — including the ones that fire before start() returns; a transient recovery re-enters .connected.

public enum SessionState: Sendable, Equatable {
    case idle
    case connecting
    case connected
    case reconnecting          // transient drop; session survives if recovery succeeds
    case disconnected(reason: DisconnectReason, detail: String?)   // terminal
}

public enum DisconnectReason: String, Sendable, Equatable {
    case clientEnded = "client_ended"           // this client called end()
    case clientClosed = "client_closed"         // close() without the wire end frame
    case handshakeFailed = "handshake_failed"   // server refused the start
    case serverEnded = "server_ended"           // graceful server teardown; detail carries the slug
    case transportError = "transport_error"
}

DisconnectReason is the same five-slug vocabulary the SessionEnd hook context and the sibling SDKs use; detail carries the server's end slug or a transport message when one exists.

Sends

All sends throw SessionStateError with code notConnected outside an active session.

send(text:transcript:)

public func send(text: String, transcript: Bool = true) async throws

Send a text turn. For a session that never speaks, configure the agent with AudioConfig(output: false). The sent text lands in session.transcript as its own closed user turn — an in-progress speech transcription is untouched. Pass transcript: false to keep it out.

send(context:)

public func send(context: String) async throws

Give the agent context without asking it anything. The note lands in the model's context for its next reply and never becomes a turn of its own: no spoken response, no assistant message, no interruption of what the agent is saying. For live application state; send(text:) is the opposite — it asks.

setMuted(_:)

public func setMuted(_ muted: Bool) async throws

Sends the wire mute frame (so the agent can update VAD state) and toggles local capture. Throws if the capture toggle fails — notably a denied-permission first publish when unmuting a session that joined with micMuted: true. Re-asserted automatically after a transport reconnect.

ping()

public func ping() async throws

Keep-alive; the server replies with .pong.

send(image:maxLongEdge:quality:streamId:)

public func send(
    image: CGImage,
    maxLongEdge: Int = ImageDownscale.recommendedMaxLongEdge,
    quality: Double = ImageDownscale.recommendedQuality,
    streamId: String = "video.input.default"
) async throws

Send a single image frame, downscaled to maxLongEdge (1280 by default) and JPEG-encoded. Preferred over the base64 overload: the frame is bounded before it is encoded or base64-inflated. See Image input for why 1280.

send(image:mimeType:streamId:)

public func send(
    image data: String,
    mimeType: String = "image/jpeg",
    streamId: String = "video.input.default"
) async throws

Send a single image frame as base64 JSON (data is the base64-encoded bytes, not raw bytes). For one-shot captures where a continuous video track would be overkill. Oversized frames are split into envelope chunks transparently.

A payload past RealtimeSession.maxImageBase64Length (12,000,000 chars, mirroring the server's ingress bound) is rejected with ImageDownscale.Error.payloadTooLarge. Between the inspection threshold and that limit, an over-resolution frame is decoded and re-encoded at ImageDownscale.recommendedMaxLongEdge, and the re-encode is logged — prefer the CGImage overload to avoid the lossy round trip.

sendActivityEnd()

public func sendActivityEnd() async throws

Manual-VAD end-of-turn: the user's "I'm done speaking" hint for turn-taking modes with silence detection off. No microphone side effect, unlike setMuted.

end() / close()

public func end() async     // graceful: best-effort wire end frame, reason .clientEnded
public func close() async   // abrupt local teardown, reason .clientClosed

Both are idempotent and terminal; the events stream finishes with a final .sessionEnded.

Liveness and waiting

public nonisolated let agentLive: AsyncStream<Void>
public func waitUntilAgentLive() async
public func waitUntilEnded() async

Audio levels and playback

public nonisolated var inputLevels: AsyncStream<Float>    // mic RMS, 0…1
public nonisolated var outputLevels: AsyncStream<Float>   // agent audio RMS, 0…1
public nonisolated func setAgentPlaybackVolume(_ volume: Double)

Level streams are latest-value (a slow consumer drops intermediate samples), render-callback driven, and finish when the session ends. setAgentPlaybackVolume is a software gain: 0 mutes, 1 is unity, values outside 0…1 are clamped; it re-applies to any agent track that attaches later. A read-only connectTimings: SessionConnectTimings exposes the connect-phase breakdown.

Screen share

public func startScreenShare() async throws
public nonisolated func pushScreenShareFrame(_ sampleBuffer: CMSampleBuffer)
public func stopScreenShare() async
public nonisolated func setScreenShareFrameProcessor(_ processor: ScreenShareFrameProcessor?)
public nonisolated func onScreenShareFailed(_ handler: @escaping @Sendable (Error) -> Void) -> Cancellable

startScreenShare creates the video track immediately but defers the SFU publish until the first pushScreenShareFrame (the capturer needs a frame to resolve dimensions); it's idempotent and throws .notConnected when the session isn't live, or .screenShareUnavailable if the capturer can't be created. pushScreenShareFrame is safe from a capture thread and no-ops outside an active share. ScreenShareFrameProcessor is @Sendable (CMSampleBuffer) -> CMSampleBuffer?, run before each frame reaches the capturer. onScreenShareFailed fires when the deferred publish fails; share state is cleared first, so the handler may call startScreenShare() again.

Video streams

public func addVideoStream() async throws -> VideoStreamHandle
public func removeVideoStream(_ handle: VideoStreamHandle) async

The non-screen counterpart of screen share, for any pixels-only video — a camera, a file, anything that's not the user's screen. The backend narrates it to the model as a camera feed, so screen questions and screen tools stay anchored to actual screen shares. Same deferred-publish contract: the returned VideoStreamHandle is the frame sink — call handle.push(_ sampleBuffer:) from the capture callback (safe from a capture thread; the first push kicks off the publish). One video publish at a time: addVideoStream throws .videoPublishAlreadyActive while another video publish (stream or share) is live. removeVideoStream is identity-keyed and idempotent — a stale handle is a no-op, and pushes into a removed handle are safely inert. Publish failures surface on onScreenShareFailed.

Audio streams

public func startAudioStream() async throws
public nonisolated func pushAudioBuffer(_ buffer: AVAudioPCMBuffer)
public func stopAudioStream() async

For audio the SDK cannot capture itself — a synthetic generator, file replay, a load test, or a host with no usable microphone. Feed it with pushAudioBuffer(_:) from your render callback (safe from an audio thread; buffers are resampled to the engine's format). For the device microphone use setMuted(_:).

startAudioStream publishes the local audio track if it isn't already publishing and clears the server-side mute gate, so the agent listens to the pushed audio. A session carries one voice: while the stream is running the device microphone is silenced — the agent hears exactly the buffers you push — and stopAudioStream gives it back, as does ending the session. Starting a second stream throws .audioPublishAlreadyActive, and starting one outside a live session throws .notConnected. stopAudioStream is idempotent, and a push after it is inert.

Telephony: dial

public func dial(phoneNumber: String, callerNumber: String? = nil) async throws -> DialResult

public struct DialResult: Codable, Hashable, Sendable {
    public var dialId: UUID
}

Place an outbound phone call into this running session. Returns once the dial is queued — the call rings asynchronously, and the conversation arrives on the ordinary event stream. DialResult.dialId is the handle to correlate the call with server-side dial status. Both numbers are validated as E.164 (+ followed by 8–15 digits) before the request — a malformed number throws DialError with code .invalidRequest. Server rejections throw it with code .requestRejected, carrying the server's own slug on serverCode (for example, caller_number_not_available, minute_limit_exceeded) and its human-readable reason on message. To grant the model hang-up, add .endCallTool() to the agent's tools (in-call tools); to end from your own code, call end(). See Telephony.

Audio session management (static, iOS)

public static func setAutomaticAudioSessionManagement(enabled: Bool) // iOS only

Toggle LiveKit's automatic AVAudioSession management. When disabled, the host app is the sole owner of the audio session — LiveKit no longer configures the category/mode/route or activates it around a connect. Set once before the first start. iOS only (compiled out on macOS, which has no AVAudioSession).

To keep the OS audio engine hot so the next start's mic publish skips a cold start, drive MicPrewarmCoordinator — set(_:) requests a warm state and settle() awaits the last request. While the engine is warm the OS shows the mic-active indicator, so scope it to a "ready to start" window rather than leaving it on.

SessionStartError

public struct SessionStartError: RealtimeError, LocalizedError, Equatable {
    public let code: SessionStartErrorCode
    public let message: String
    public let status: Int?
    public let serverCode: String?
    public let retryAfterSeconds: Int?
}

Thrown by agent.start(...) for every way a start can fail to produce a live session. code is closed — this SDK throws each one, so a switch over it stays exhaustive. serverCode is the server's own slug for why it refused, an open set: log it, don't branch on it. status is the HTTP status of a server rejection, nil when nothing answered. retryAfterSeconds is set only for .busy, and only when the server sent a Retry-After.

A call the session cannot serve in its current state throws SessionStateError instead — a send outside an active session, a second start, a malformed payload. See Errors.

Gemini tool response policies

GeminiToolResponsePolicy(behavior:scheduling:) chooses .blocking or .nonBlocking and optional .whenIdle, .silent or .interrupt scheduling. GeminiModel.toolResponsePolicy supplies the default; toolResponseOverrides maps declared tool names to replacement policies. Omitted policies keep standard Gemini Live tools blocking. Non-blocking results default to .whenIdle; .silent absorbs without speaking and .interrupt interrupts speech to answer.

let model = GeminiModel(
    modelId: "gemini-3.8-live",
    toolResponseOverrides: [
        "lookup": .init(behavior: .nonBlocking, scheduling: .whenIdle)
    ]
)

gemini-3.8-live-extended-thinking defaults to non-blocking tools, accepts .low, .medium or .high thinking, and rejects .minimal, blocking policies and explicit scheduling. An utterance can finish while the model is still thinking. Select independent lookups for non-blocking behavior; keep dependent actions blocking on standard Gemini Live. Background job tools retain their acknowledgement and completion contract.