Cosmo Realtime SDK
ReferenceSwift

RealtimeSession (Swift)

Swift actor for a single Cosmo Realtime voice session.

RealtimeSession

RealtimeSession is a Swift actor — one live realtime voice session speaking the published developer protocol. One call starts it; consumption is a single typed event stream. A session is single-attempt: once it ends (by end(), by the server, or by a transport failure) it is terminal — start a new one to reconnect.

import CosmoRealtime

let session = try await RealtimeSession.start(
    .init(apiKey: "key", baseURL: URL(string: "https://app.askcosmo.ai")!),
    config: SessionConfig(instructions: "You are a terse assistant.")
)
for try await event in session.events {
    switch event {
    case .ready(let ready): print("live, session:", ready.sessionId)
    case .transcript(let delta): print(delta.text)
    case .sessionEnded(let ended): print("over:", ended.reason ?? "")
    default: break
    }
}

session.events is single-consumer: iterate it from exactly one task.


RealtimeSession.Options

Client-level settings: credential, endpoint, timeouts.

public struct Options: Sendable {
    public var credential: Credential   // .apiKey(String) or .token(String)
    public var baseURL: URL
    public var connectTimeout: TimeInterval
    public var requestTimeout: TimeInterval
    public var verifyTLS: VerifyTLS
    public var clientIdentity: ClientIdentity?

    public init(
        credential: Credential,
        baseURL: URL,
        connectTimeout: TimeInterval = 30,
        requestTimeout: TimeInterval = 45,
        verifyTLS: VerifyTLS = .auto,
        clientIdentity: ClientIdentity? = nil
    )
}

Two convenience initializers pick the credential form directly (same remaining parameters and defaults):

public init(apiKey: String, baseURL: URL, ...)   // Options.Credential.apiKey
public init(token: String, baseURL: URL, ...)    // Options.Credential.token
PropertyTypeDefaultDescription
credentialCredentialrequiredExactly one of .apiKey(String) (workspace-scoped, server-side only; can mint end-user tokens) or .token(String) (minted per-user JWT, safe to embed in a client — see End-user credentials).
baseURLURLrequiredBackend origin. Must be https; plain http is allowed only for loopback hosts, otherwise start throws .insecureBaseURL.
connectTimeoutTimeInterval30Media-transport join (signaling + ICE).
requestTimeoutTimeInterval45REST session-start request.
verifyTLSVerifyTLS.auto.auto verifies remote hosts and skips verification only for loopback (self-signed local dev); .enabled always verifies; .disabled never verifies.
clientIdentityClientIdentity?nilIdentifies the calling app via X-Cosmo-Client / -Version / -Build headers. nil sends no client headers. Telemetry only, never an auth signal.

options.canMint is true only for an .apiKey credential.

public struct ClientIdentity: Sendable, Equatable {
    public init(client: String, marketingVersion: String, build: String)
}

Starting a session

public static func start(
    _ options: Options,
    config: SessionConfig = SessionConfig(),
    micMuted: Bool = false,
    rpcHandlers: [String: ClientToolHandler] = [:]
) async throws -> RealtimeSession

One REST session-start plus the media-transport join. Returns once the transport is live; await .ready on events for the agent-ready signal.

ParameterDescription
optionsCredential, base URL, timeouts.
configPer-session SessionConfig. Empty config runs the server's neutral defaults.
micMutedWhen true, the session joins without publishing the microphone — nothing is captured or sent until the first setMuted(false).
rpcHandlersClient-tool handlers registered by method name but not advertised to the agent — for server-orchestrated tools invoked over RPC directly. Advertised-and-handled tools belong in SessionConfig.tools; on a name collision the rpcHandlers entry wins.

Throws RealtimeSessionError on rejection or transport failure. A started session exposes session.sessionId (String?, set once the start succeeds).


SessionConfig

Every field is optional: the server applies neutral defaults for anything left unset, and unset fields stay off the wire.

public struct SessionConfig: Sendable, Equatable {
    public init(
        agentName: String? = nil,
        agentInputs: [String: String]? = nil,
        model: String? = nil,
        modelOptions: ModelOptions? = nil,
        voice: Voice? = nil,
        audio: Audio? = nil,
        instructions: String? = nil,
        tools: [Tool]? = nil,
        interruptionSensitivity: InterruptionSensitivity? = nil,
        greeting: String? = nil,
        resumeSessionId: String? = nil,
        maxSessionSeconds: Int? = nil,
        storeRecording: Bool? = nil,
        hooks: [Hook]? = nil
    )
}
FieldTypeDescription
agentNameString?Machine handle of a workspace catalog agent (lowercase [a-z0-9-]). The stored config runs verbatim, so setting model, modelOptions, audio, instructions, interruptionSensitivity, greeting, or a server hook alongside it throws .invalidPayload at start. Per-run fields still apply: agentInputs, tools, voice, resumeSessionId, maxSessionSeconds, storeRecording, and client hooks.
agentInputs[String: String]?String inputs for the referenced agent (template placeholders). Valid only alongside agentName.
modelString?Provider/model selection. nil lets the server choose; unavailable values are rejected at session start.
modelOptionsModelOptions?Provider-scoped model knobs, discriminated on provider (below).
voiceVoice?How the agent sounds — the prebuilt voice id and the delivery guidance (below). nil keeps the server defaults for both.
audioAudio?The session's audio pipeline — output emission, inbound noise cancellation, ambience bed (below). nil keeps every server default.
instructionsString?System instructions; replaces the server's neutral default.
tools[Tool]?Tool set for the session (below). nil inherits defaults; an explicit [] runs with no tools. See Tools.
interruptionSensitivityInterruptionSensitivity?How readily user speech interrupts the agent. Wire values: default, high, low.
greetingString?Opening line the assistant speaks first, voiced server-side as soon as the model session opens — before the client even receives ready. nil waits for the user.
resumeSessionIdString?Resume the named prior session. Experimental — may change shape without a protocol-version bump.
maxSessionSecondsInt?Requested wall-clock cap. The server resolves the effective cap as the minimum of this and its own limits; the effective value is echoed on ready. See Session limits.
storeRecordingBool?Persist this run's recording artifacts server-side. nil keeps the server default: the session records. See Recording and privacy.
hooks[Hook]?One list, two kinds: in-process client hooks built by the seam factories (sessionStart(_:), preToolUse(matcher:_:), …) and declarative server hooks (Hook.server(_:) wrapping a SilenceTimeout). See Hooks.

SessionConfig.Voice

public struct Voice: Sendable, Equatable {
    public init(name: String? = nil, speakingStyle: String? = nil)
}
FieldTypeDescription
nameString?Provider-specific prebuilt voice id. nil lets the upstream pick per session — the voice then drifts between connects.
speakingStyleString?A "how to speak" instruction appended to the system prompt after the persona. nil keeps the server default.

An empty Voice stays off the wire entirely.

SessionConfig.Audio

public struct Audio: Sendable, Equatable {
    public init(output: Bool? = nil, noiseCancellation: Bool? = nil, ambience: Ambience? = nil)
}

public struct Ambience: Sendable, Equatable {
    public init(track: String? = nil, gainDb: Double? = nil)
}
FieldTypeDescription
outputBool?false runs the session text-only (no speech; transcription and text output unaffected). Rejected when the resolved model cannot run text-only. nil keeps the server default (on).
noiseCancellationBool?Upstream noise cancellation on input audio. nil keeps the server default (off).
ambienceAmbience?Background bed mixed into the assistant's output. Presence enables it, so nil means no bed. track names the bed (nil = default); gainDb is its level relative to full scale (-60…0).
let config = SessionConfig(
    voice: .init(name: "Puck", speakingStyle: "Warm and unhurried."),
    audio: .init(noiseCancellation: true, ambience: .init(track: "office", gainDb: -30)),
    instructions: "You are a terse voice assistant."
)

SessionConfig.ModelOptions

Each knob is honored only by its provider, so an illegal pairing is unrepresentable. model selects the concrete model within the provider.

public enum ModelOptions: Sendable, Equatable {
    case gemini(temperature: Double? = nil, maxOutputTokens: Int? = nil, thinkingLevel: ThinkingLevel? = nil)
    case openai
    case ultravox(temperature: Double? = nil, turnEndpointDelaySeconds: Double? = nil)
    case personaplex
}

ThinkingLevel wire values: minimal, low, medium, high (Gemini only).

SessionConfig.Tool

public enum Tool: Sendable, Equatable {
    case client(name: String, description: String,
                parameters: [String: JSONValue], handler: ClientToolHandler? = nil)
    case backgroundClient(name: String, description: String,
                          parameters: [String: JSONValue], handler: BackgroundClientToolHandler)
    case webSearch
    case examineImage
    case detectObjects
    case pointAtObject
}

public typealias ClientToolHandler =
    @Sendable ([String: JSONValue]) async throws -> [String: JSONValue]
  • .client — executed by this client over LiveKit RPC. parameters is the JSON Schema for the arguments (restricted dialect, top-level type: "object"). A spec without a handler is declared but not executable — invocations surface only as .toolInvocation events.
  • .backgroundClient — same wire shape; the handler receives a ClientToolJob, acks immediately (job.ack) and delivers later (job.complete / job.fail) so long-running work does not block the voice turn.
  • .webSearch / .examineImage / .detectObjects / .pointAtObject — zero-config opt-ins to server-executed tools; the server owns the model-facing declaration.

Event stream

public nonisolated let events: AsyncThrowingStream<Event, Error>

Typed server events in arrival order. Every terminal path of a live session ends with a locally synthesized .sessionEnded as the final element, after which the sequence finishes — the stream does not throw; start failures throw from start instead.

CasePayloadDescription
.readyReadyUpstream session established; carries sessionId and any soft-rejected tool specs.
.transcriptTranscriptDeltaStreaming transcript fragment (delta while isFinal is false; cumulative full text on the final event).
.modelTextModelTextText-channel fragment from the model (distinct from the spoken-audio transcript).
.turnCompleteTurnCompleteThe agent finished a turn.
.userStartedSpeaking / .userStoppedSpeakingVAD boundaries for the user.
.botStartedSpeaking / .botStoppedSpeakingAgent speech boundaries.
.botLlmStarted / .botLlmStoppedModel inference boundaries.
.botTtsStarted / .botTtsStoppedSpeech-synthesis boundaries.
.toolCallToolCallModel decided to invoke a server-executed tool.
.toolDispatchStartedToolDispatchStartedServer began executing the tool.
.toolResultToolResultServer tool finished.
.toolInvocationToolInvocationObservability: the server invoked a declared client tool (execution happens via the local handler over RPC).
.reconnectingReconnectingServer is transparently rotating the upstream session.
.cosmoCosmoEventFirst-party Cosmo event (.usage(CosmoUsage) — cumulative token usage).
.userSpeechTimeoutUserSpeechTimeoutA server-runtime silence hook fired; action reports what the server did.
.sessionEndedSessionEndedTerminal; the stream finishes after this element. reason: String?.
.errorErrorEventRecoverable or terminal — switch on fatal. Codes: auth_failed, workspace_forbidden, voice_disabled, upstream_disconnect, internal_error, invalid_message, version_mismatch.
.pongReply to ping().
.unknown(rawType: String?, payload: Data)Forward compatibility: an unrecognized or undecodable frame. Never terminal.

Payload types (Ready, TranscriptDelta, ToolCall, …) are re-exposed generated wire types under RealtimeSession.* typealiases.

states

public nonisolated let states: AsyncStream<State>

Transport lifecycle, distinct from the application-level .ready event. Yields .idle on creation and finishes after the terminal .disconnected.

public enum State: Sendable, Equatable {
    case idle
    case connecting
    case connected
    case reconnecting          // transient drop; session survives if recovery succeeds
    case reconnected           // same session, same thread
    case disconnected(reason: EndReason)   // terminal
}

public enum EndReason: Sendable, Equatable {
    case clientEnded                                    // this client called end()
    case clientClosed                                   // close() without the wire end frame
    case handshakeFailed(status: Int?, detail: String?) // server refused the start
    case serverEnded(reason: String?)                   // graceful server teardown
    case transportError(message: String)
}

Sends

All sends throw RealtimeSessionError.notConnected outside an active session.

send(text:)

public func send(text: String) async throws

Send a text turn. For a session that never speaks, configure the agent with audio.output = false.

setMuted(_:)

public func setMuted(_ muted: Bool) async throws

Sends the wire mute frame (so the agent can update VAD state) and toggles local capture. Throws if the capture toggle fails — notably a denied-permission first publish when unmuting a session that joined with micMuted: true. Re-asserted automatically after a transport reconnect.

ping()

public func ping() async throws

Keep-alive; the server replies with .pong.

send(image:mimeType:streamId:)

public func send(
    image data: String,
    mimeType: String = "image/jpeg",
    streamId: String = "video.input.default"
) async throws

Send a single image frame as base64 JSON (data is the base64-encoded bytes, not raw bytes). For one-shot captures where a continuous video track would be overkill. Oversized frames are split into envelope chunks transparently.

send(bytes:topic:)

public func send(bytes data: Data, topic: String) async throws

Stream raw bytes to the agent on a named topic, out of band from the JSON control channel — for large binary client-tool payloads that would be wasteful to base64. Delivered only to the agent participant.

sendActivityEnd()

public func sendActivityEnd() async throws

Manual-VAD end-of-turn: the user's "I'm done speaking" hint for turn-taking modes with silence detection off. No microphone side effect, unlike setMuted.

end() / close()

public func end() async     // graceful: best-effort wire end frame, reason .clientEnded
public func close() async   // abrupt local teardown, reason .clientClosed

Both are idempotent and terminal; the events stream finishes with a final .sessionEnded.


Audio levels and playback

public nonisolated var inputLevels: AsyncStream<Float>    // mic RMS, 0…1
public nonisolated var outputLevels: AsyncStream<Float>   // agent audio RMS, 0…1
public nonisolated func setAgentPlaybackVolume(_ volume: Double)

Level streams are latest-value (a slow consumer drops intermediate samples), render-callback driven, and finish when the session ends. setAgentPlaybackVolume is a software gain: 0 mutes, 1 is unity, values outside 0…1 are clamped; it re-applies to any agent track that attaches later. A read-only qoeSnapshot: SessionQoESnapshot exposes per-session WebRTC quality aggregates.


Screen share

public func startScreenShare() async throws
public nonisolated func pushScreenShareFrame(_ sampleBuffer: CMSampleBuffer)
public func stopScreenShare() async
public nonisolated func setScreenShareFrameProcessor(_ processor: ScreenShareFrameProcessor?)
public nonisolated func onScreenShareFailed(_ handler: @escaping @Sendable (Error) -> Void) -> Cancellable

startScreenShare creates the video track immediately but defers the SFU publish until the first pushScreenShareFrame (the capturer needs a frame to resolve dimensions); it is idempotent and throws .notConnected when the session is not live, or .screenShareUnavailable if the capturer cannot be created. pushScreenShareFrame is safe from a capture thread and no-ops outside an active share. ScreenShareFrameProcessor is @Sendable (CMSampleBuffer) -> CMSampleBuffer?, run before each frame reaches the capturer. onScreenShareFailed fires when the deferred publish fails; share state is cleared first, so the handler may call startScreenShare() again.


Telephony: dial

public func dial(phoneNumber: String, callerNumber: String? = nil) async throws -> String

Place an outbound phone call into this running session; returns the server-minted dial_id. Both numbers are validated as E.164 (+ followed by 8–15 digits) before the request — a malformed number throws .invalidPayload. Server rejections surface as .handshakeFailed(status:code:detail:) with the machine-readable rejection code (e.g. caller_number_not_available, minute_limit_exceeded). Declare the cosmo.end_call server tool if the model should be able to hang up. See Telephony.


Warm-up (static)

public static func prewarmConnection(origin: PrewarmOrigin = .other) async
public static func setRecordingAlwaysPrepared(_ enabled: Bool) async throws
public static func prepareSession(_ options: Options) async throws
public static func discardPreparedSession()
MethodDescription
prewarmConnection(origin:)Pre-warm LiveKit signaling (TLS + edge selection) using the most recent session's cached URL, shortening the next start's room-join. Best-effort; no-op on the first ever start. PrewarmOrigin cases: .launch, .teardown, .other.
setRecordingAlwaysPrepared(_:)Keep the OS audio engine hot so the next start's mic publish skips a cold start. No-ops without mic permission. While enabled the OS shows the mic-active indicator — scope it to a "ready to start" window.
prepareSession(_:)Pre-create a room and mint the join token off the press path. The next start against the same baseURL consumes the parked session and joins immediately while session-start runs in parallel; an absent or stale handle degrades to the serialized path. Single process-global slot.
discardPreparedSession()Drop any parked prepared session. Call on sign-out or account switch — the room grant is bound to the preparing user.

RealtimeSessionError

public enum RealtimeSessionError: Error, LocalizedError, Equatable {
    case versionMismatch(detail: String)
    case voiceDisabled
    case handshakeFailed(status: Int, code: String?, detail: String?)
    case sessionStartFailed(message: String)
    case alreadyStarted
    case notConnected
    case transportError(message: String)
    case invalidPayload(String)
    case screenShareUnavailable
    case insecureBaseURL(String)
}
CaseWhen
.versionMismatchThe server refused the start: this SDK speaks an incompatible protocol version. Upgrade the SDK.
.voiceDisabledHTTP 503 — realtime voice temporarily unavailable.
.handshakeFailedHTTP rejection with status, a machine-readable code slug for typed rejections (branch on it, not the message), and the server detail.
.sessionStartFailedStart failed (rejection without a status, or transport failure during start).
.alreadyStartedSessions are single-attempt; construct a new one instead of starting twice.
.notConnectedA send outside an active session.
.transportErrorTransport failed while starting. A mid-session drop does not throw — it ends events with a terminal .sessionEnded.
.invalidPayloadA caller-supplied payload violates a wire-protocol invariant (catalog-agent field conflict, malformed E.164 number, …).
.screenShareUnavailableThe LiveKit BufferCapturer could not be created.
.insecureBaseURLPlain http to a non-loopback host — a bearer credential must not travel over cleartext.

On this page