RealtimeSession (Swift)
Swift actor for a single Cosmo Realtime voice session.
RealtimeSession
RealtimeSession is a Swift actor — one live realtime voice session speaking the published developer protocol. One call starts it; consumption is a single typed event stream. A session is single-attempt: once it ends (by end(), by the server, or by a transport failure) it is terminal — start a new one to reconnect.
import CosmoRealtime
let session = try await RealtimeSession.start(
.init(apiKey: "key", baseURL: URL(string: "https://app.askcosmo.ai")!),
config: SessionConfig(instructions: "You are a terse assistant.")
)
for try await event in session.events {
switch event {
case .ready(let ready): print("live, session:", ready.sessionId)
case .transcript(let delta): print(delta.text)
case .sessionEnded(let ended): print("over:", ended.reason ?? "")
default: break
}
}session.events is single-consumer: iterate it from exactly one task.
RealtimeSession.Options
Client-level settings: credential, endpoint, timeouts.
public struct Options: Sendable {
public var credential: Credential // .apiKey(String) or .token(String)
public var baseURL: URL
public var connectTimeout: TimeInterval
public var requestTimeout: TimeInterval
public var verifyTLS: VerifyTLS
public var clientIdentity: ClientIdentity?
public init(
credential: Credential,
baseURL: URL,
connectTimeout: TimeInterval = 30,
requestTimeout: TimeInterval = 45,
verifyTLS: VerifyTLS = .auto,
clientIdentity: ClientIdentity? = nil
)
}Two convenience initializers pick the credential form directly (same remaining parameters and defaults):
public init(apiKey: String, baseURL: URL, ...) // Options.Credential.apiKey
public init(token: String, baseURL: URL, ...) // Options.Credential.token| Property | Type | Default | Description |
|---|---|---|---|
credential | Credential | required | Exactly one of .apiKey(String) (workspace-scoped, server-side only; can mint end-user tokens) or .token(String) (minted per-user JWT, safe to embed in a client — see End-user credentials). |
baseURL | URL | required | Backend origin. Must be https; plain http is allowed only for loopback hosts, otherwise start throws .insecureBaseURL. |
connectTimeout | TimeInterval | 30 | Media-transport join (signaling + ICE). |
requestTimeout | TimeInterval | 45 | REST session-start request. |
verifyTLS | VerifyTLS | .auto | .auto verifies remote hosts and skips verification only for loopback (self-signed local dev); .enabled always verifies; .disabled never verifies. |
clientIdentity | ClientIdentity? | nil | Identifies the calling app via X-Cosmo-Client / -Version / -Build headers. nil sends no client headers. Telemetry only, never an auth signal. |
options.canMint is true only for an .apiKey credential.
public struct ClientIdentity: Sendable, Equatable {
public init(client: String, marketingVersion: String, build: String)
}Starting a session
public static func start(
_ options: Options,
config: SessionConfig = SessionConfig(),
micMuted: Bool = false,
rpcHandlers: [String: ClientToolHandler] = [:]
) async throws -> RealtimeSessionOne REST session-start plus the media-transport join. Returns once the transport is live; await .ready on events for the agent-ready signal.
| Parameter | Description |
|---|---|
options | Credential, base URL, timeouts. |
config | Per-session SessionConfig. Empty config runs the server's neutral defaults. |
micMuted | When true, the session joins without publishing the microphone — nothing is captured or sent until the first setMuted(false). |
rpcHandlers | Client-tool handlers registered by method name but not advertised to the agent — for server-orchestrated tools invoked over RPC directly. Advertised-and-handled tools belong in SessionConfig.tools; on a name collision the rpcHandlers entry wins. |
Throws RealtimeSessionError on rejection or transport failure. A started session exposes session.sessionId (String?, set once the start succeeds).
SessionConfig
Every field is optional: the server applies neutral defaults for anything left unset, and unset fields stay off the wire.
public struct SessionConfig: Sendable, Equatable {
public init(
agentName: String? = nil,
agentInputs: [String: String]? = nil,
model: String? = nil,
modelOptions: ModelOptions? = nil,
voice: Voice? = nil,
audio: Audio? = nil,
instructions: String? = nil,
tools: [Tool]? = nil,
interruptionSensitivity: InterruptionSensitivity? = nil,
greeting: String? = nil,
resumeSessionId: String? = nil,
maxSessionSeconds: Int? = nil,
storeRecording: Bool? = nil,
hooks: [Hook]? = nil
)
}| Field | Type | Description |
|---|---|---|
agentName | String? | Machine handle of a workspace catalog agent (lowercase [a-z0-9-]). The stored config runs verbatim, so setting model, modelOptions, audio, instructions, interruptionSensitivity, greeting, or a server hook alongside it throws .invalidPayload at start. Per-run fields still apply: agentInputs, tools, voice, resumeSessionId, maxSessionSeconds, storeRecording, and client hooks. |
agentInputs | [String: String]? | String inputs for the referenced agent (template placeholders). Valid only alongside agentName. |
model | String? | Provider/model selection. nil lets the server choose; unavailable values are rejected at session start. |
modelOptions | ModelOptions? | Provider-scoped model knobs, discriminated on provider (below). |
voice | Voice? | How the agent sounds — the prebuilt voice id and the delivery guidance (below). nil keeps the server defaults for both. |
audio | Audio? | The session's audio pipeline — output emission, inbound noise cancellation, ambience bed (below). nil keeps every server default. |
instructions | String? | System instructions; replaces the server's neutral default. |
tools | [Tool]? | Tool set for the session (below). nil inherits defaults; an explicit [] runs with no tools. See Tools. |
interruptionSensitivity | InterruptionSensitivity? | How readily user speech interrupts the agent. Wire values: default, high, low. |
greeting | String? | Opening line the assistant speaks first, voiced server-side as soon as the model session opens — before the client even receives ready. nil waits for the user. |
resumeSessionId | String? | Resume the named prior session. Experimental — may change shape without a protocol-version bump. |
maxSessionSeconds | Int? | Requested wall-clock cap. The server resolves the effective cap as the minimum of this and its own limits; the effective value is echoed on ready. See Session limits. |
storeRecording | Bool? | Persist this run's recording artifacts server-side. nil keeps the server default: the session records. See Recording and privacy. |
hooks | [Hook]? | One list, two kinds: in-process client hooks built by the seam factories (sessionStart(_:), preToolUse(matcher:_:), …) and declarative server hooks (Hook.server(_:) wrapping a SilenceTimeout). See Hooks. |
SessionConfig.Voice
public struct Voice: Sendable, Equatable {
public init(name: String? = nil, speakingStyle: String? = nil)
}| Field | Type | Description |
|---|---|---|
name | String? | Provider-specific prebuilt voice id. nil lets the upstream pick per session — the voice then drifts between connects. |
speakingStyle | String? | A "how to speak" instruction appended to the system prompt after the persona. nil keeps the server default. |
An empty Voice stays off the wire entirely.
SessionConfig.Audio
public struct Audio: Sendable, Equatable {
public init(output: Bool? = nil, noiseCancellation: Bool? = nil, ambience: Ambience? = nil)
}
public struct Ambience: Sendable, Equatable {
public init(track: String? = nil, gainDb: Double? = nil)
}| Field | Type | Description |
|---|---|---|
output | Bool? | false runs the session text-only (no speech; transcription and text output unaffected). Rejected when the resolved model cannot run text-only. nil keeps the server default (on). |
noiseCancellation | Bool? | Upstream noise cancellation on input audio. nil keeps the server default (off). |
ambience | Ambience? | Background bed mixed into the assistant's output. Presence enables it, so nil means no bed. track names the bed (nil = default); gainDb is its level relative to full scale (-60…0). |
let config = SessionConfig(
voice: .init(name: "Puck", speakingStyle: "Warm and unhurried."),
audio: .init(noiseCancellation: true, ambience: .init(track: "office", gainDb: -30)),
instructions: "You are a terse voice assistant."
)SessionConfig.ModelOptions
Each knob is honored only by its provider, so an illegal pairing is unrepresentable. model selects the concrete model within the provider.
public enum ModelOptions: Sendable, Equatable {
case gemini(temperature: Double? = nil, maxOutputTokens: Int? = nil, thinkingLevel: ThinkingLevel? = nil)
case openai
case ultravox(temperature: Double? = nil, turnEndpointDelaySeconds: Double? = nil)
case personaplex
}ThinkingLevel wire values: minimal, low, medium, high (Gemini only).
SessionConfig.Tool
public enum Tool: Sendable, Equatable {
case client(name: String, description: String,
parameters: [String: JSONValue], handler: ClientToolHandler? = nil)
case backgroundClient(name: String, description: String,
parameters: [String: JSONValue], handler: BackgroundClientToolHandler)
case webSearch
case examineImage
case detectObjects
case pointAtObject
}
public typealias ClientToolHandler =
@Sendable ([String: JSONValue]) async throws -> [String: JSONValue].client— executed by this client over LiveKit RPC.parametersis the JSON Schema for the arguments (restricted dialect, top-leveltype: "object"). A spec without a handler is declared but not executable — invocations surface only as.toolInvocationevents..backgroundClient— same wire shape; the handler receives aClientToolJob, acks immediately (job.ack) and delivers later (job.complete/job.fail) so long-running work does not block the voice turn..webSearch/.examineImage/.detectObjects/.pointAtObject— zero-config opt-ins to server-executed tools; the server owns the model-facing declaration.
Event stream
public nonisolated let events: AsyncThrowingStream<Event, Error>Typed server events in arrival order. Every terminal path of a live session ends with a locally synthesized .sessionEnded as the final element, after which the sequence finishes — the stream does not throw; start failures throw from start instead.
| Case | Payload | Description |
|---|---|---|
.ready | Ready | Upstream session established; carries sessionId and any soft-rejected tool specs. |
.transcript | TranscriptDelta | Streaming transcript fragment (delta while isFinal is false; cumulative full text on the final event). |
.modelText | ModelText | Text-channel fragment from the model (distinct from the spoken-audio transcript). |
.turnComplete | TurnComplete | The agent finished a turn. |
.userStartedSpeaking / .userStoppedSpeaking | — | VAD boundaries for the user. |
.botStartedSpeaking / .botStoppedSpeaking | — | Agent speech boundaries. |
.botLlmStarted / .botLlmStopped | — | Model inference boundaries. |
.botTtsStarted / .botTtsStopped | — | Speech-synthesis boundaries. |
.toolCall | ToolCall | Model decided to invoke a server-executed tool. |
.toolDispatchStarted | ToolDispatchStarted | Server began executing the tool. |
.toolResult | ToolResult | Server tool finished. |
.toolInvocation | ToolInvocation | Observability: the server invoked a declared client tool (execution happens via the local handler over RPC). |
.reconnecting | Reconnecting | Server is transparently rotating the upstream session. |
.cosmo | CosmoEvent | First-party Cosmo event (.usage(CosmoUsage) — cumulative token usage). |
.userSpeechTimeout | UserSpeechTimeout | A server-runtime silence hook fired; action reports what the server did. |
.sessionEnded | SessionEnded | Terminal; the stream finishes after this element. reason: String?. |
.error | ErrorEvent | Recoverable or terminal — switch on fatal. Codes: auth_failed, workspace_forbidden, voice_disabled, upstream_disconnect, internal_error, invalid_message, version_mismatch. |
.pong | — | Reply to ping(). |
.unknown | (rawType: String?, payload: Data) | Forward compatibility: an unrecognized or undecodable frame. Never terminal. |
Payload types (Ready, TranscriptDelta, ToolCall, …) are re-exposed generated wire types under RealtimeSession.* typealiases.
states
public nonisolated let states: AsyncStream<State>Transport lifecycle, distinct from the application-level .ready event. Yields .idle on creation and finishes after the terminal .disconnected.
public enum State: Sendable, Equatable {
case idle
case connecting
case connected
case reconnecting // transient drop; session survives if recovery succeeds
case reconnected // same session, same thread
case disconnected(reason: EndReason) // terminal
}
public enum EndReason: Sendable, Equatable {
case clientEnded // this client called end()
case clientClosed // close() without the wire end frame
case handshakeFailed(status: Int?, detail: String?) // server refused the start
case serverEnded(reason: String?) // graceful server teardown
case transportError(message: String)
}Sends
All sends throw RealtimeSessionError.notConnected outside an active session.
send(text:)
public func send(text: String) async throwsSend a text turn. For a session that never speaks, configure the agent with audio.output = false.
setMuted(_:)
public func setMuted(_ muted: Bool) async throwsSends the wire mute frame (so the agent can update VAD state) and toggles local capture. Throws if the capture toggle fails — notably a denied-permission first publish when unmuting a session that joined with micMuted: true. Re-asserted automatically after a transport reconnect.
ping()
public func ping() async throwsKeep-alive; the server replies with .pong.
send(image:mimeType:streamId:)
public func send(
image data: String,
mimeType: String = "image/jpeg",
streamId: String = "video.input.default"
) async throwsSend a single image frame as base64 JSON (data is the base64-encoded bytes, not raw bytes). For one-shot captures where a continuous video track would be overkill. Oversized frames are split into envelope chunks transparently.
send(bytes:topic:)
public func send(bytes data: Data, topic: String) async throwsStream raw bytes to the agent on a named topic, out of band from the JSON control channel — for large binary client-tool payloads that would be wasteful to base64. Delivered only to the agent participant.
sendActivityEnd()
public func sendActivityEnd() async throwsManual-VAD end-of-turn: the user's "I'm done speaking" hint for turn-taking modes with silence detection off. No microphone side effect, unlike setMuted.
end() / close()
public func end() async // graceful: best-effort wire end frame, reason .clientEnded
public func close() async // abrupt local teardown, reason .clientClosedBoth are idempotent and terminal; the events stream finishes with a final .sessionEnded.
Audio levels and playback
public nonisolated var inputLevels: AsyncStream<Float> // mic RMS, 0…1
public nonisolated var outputLevels: AsyncStream<Float> // agent audio RMS, 0…1
public nonisolated func setAgentPlaybackVolume(_ volume: Double)Level streams are latest-value (a slow consumer drops intermediate samples), render-callback driven, and finish when the session ends. setAgentPlaybackVolume is a software gain: 0 mutes, 1 is unity, values outside 0…1 are clamped; it re-applies to any agent track that attaches later. A read-only qoeSnapshot: SessionQoESnapshot exposes per-session WebRTC quality aggregates.
Screen share
public func startScreenShare() async throws
public nonisolated func pushScreenShareFrame(_ sampleBuffer: CMSampleBuffer)
public func stopScreenShare() async
public nonisolated func setScreenShareFrameProcessor(_ processor: ScreenShareFrameProcessor?)
public nonisolated func onScreenShareFailed(_ handler: @escaping @Sendable (Error) -> Void) -> CancellablestartScreenShare creates the video track immediately but defers the SFU publish until the first pushScreenShareFrame (the capturer needs a frame to resolve dimensions); it is idempotent and throws .notConnected when the session is not live, or .screenShareUnavailable if the capturer cannot be created. pushScreenShareFrame is safe from a capture thread and no-ops outside an active share. ScreenShareFrameProcessor is @Sendable (CMSampleBuffer) -> CMSampleBuffer?, run before each frame reaches the capturer. onScreenShareFailed fires when the deferred publish fails; share state is cleared first, so the handler may call startScreenShare() again.
Telephony: dial
public func dial(phoneNumber: String, callerNumber: String? = nil) async throws -> StringPlace an outbound phone call into this running session; returns the server-minted dial_id. Both numbers are validated as E.164 (+ followed by 8–15 digits) before the request — a malformed number throws .invalidPayload. Server rejections surface as .handshakeFailed(status:code:detail:) with the machine-readable rejection code (e.g. caller_number_not_available, minute_limit_exceeded). Declare the cosmo.end_call server tool if the model should be able to hang up. See Telephony.
Warm-up (static)
public static func prewarmConnection(origin: PrewarmOrigin = .other) async
public static func setRecordingAlwaysPrepared(_ enabled: Bool) async throws
public static func prepareSession(_ options: Options) async throws
public static func discardPreparedSession()| Method | Description |
|---|---|
prewarmConnection(origin:) | Pre-warm LiveKit signaling (TLS + edge selection) using the most recent session's cached URL, shortening the next start's room-join. Best-effort; no-op on the first ever start. PrewarmOrigin cases: .launch, .teardown, .other. |
setRecordingAlwaysPrepared(_:) | Keep the OS audio engine hot so the next start's mic publish skips a cold start. No-ops without mic permission. While enabled the OS shows the mic-active indicator — scope it to a "ready to start" window. |
prepareSession(_:) | Pre-create a room and mint the join token off the press path. The next start against the same baseURL consumes the parked session and joins immediately while session-start runs in parallel; an absent or stale handle degrades to the serialized path. Single process-global slot. |
discardPreparedSession() | Drop any parked prepared session. Call on sign-out or account switch — the room grant is bound to the preparing user. |
RealtimeSessionError
public enum RealtimeSessionError: Error, LocalizedError, Equatable {
case versionMismatch(detail: String)
case voiceDisabled
case handshakeFailed(status: Int, code: String?, detail: String?)
case sessionStartFailed(message: String)
case alreadyStarted
case notConnected
case transportError(message: String)
case invalidPayload(String)
case screenShareUnavailable
case insecureBaseURL(String)
}| Case | When |
|---|---|
.versionMismatch | The server refused the start: this SDK speaks an incompatible protocol version. Upgrade the SDK. |
.voiceDisabled | HTTP 503 — realtime voice temporarily unavailable. |
.handshakeFailed | HTTP rejection with status, a machine-readable code slug for typed rejections (branch on it, not the message), and the server detail. |
.sessionStartFailed | Start failed (rejection without a status, or transport failure during start). |
.alreadyStarted | Sessions are single-attempt; construct a new one instead of starting twice. |
.notConnected | A send outside an active session. |
.transportError | Transport failed while starting. A mid-session drop does not throw — it ends events with a terminal .sessionEnded. |
.invalidPayload | A caller-supplied payload violates a wire-protocol invariant (catalog-agent field conflict, malformed E.164 number, …). |
.screenShareUnavailable | The LiveKit BufferCapturer could not be created. |
.insecureBaseURL | Plain http to a non-loopback host — a bearer credential must not travel over cleartext. |