Audio

Audio I/O is handled by LiveKit. Your application publishes the local microphone track; the server publishes the agent's TTS audio as a remote track. The SDK wires both sides for you.

Mic capture

When you call the SDK's microphone-enable method, the SDK:

1. Creates a LocalAudioTrack from the default microphone device.
2. Publishes it to the LiveKit room with source: MICROPHONE.
3. Sends a mute control frame ({ "type": "mute", "muted": false }) so the server starts processing inbound audio from your participant.

// Unmute (enable mic)
await session.setMuted(false);

// Mute without disconnecting
await session.setMuted(true);

import { MicToggle } from 'cosmo-ai/react';
<MicToggle />

await session.set_microphone_enabled(True)   # publish track + send mute(false)
await session.set_microphone_enabled(False)  # unpublish track + send mute(true)

try await session.setMuted(false)   // unmute: mute frame + local capture on
try await session.setMuted(true)    // mute: mute frame + local capture off

Agent audio playback

The agent's TTS output arrives as a remote audio track in the LiveKit room. The TypeScript SDK handles this with <RealtimeAudio />:

import { RealtimeAudio } from 'cosmo-ai/react';
<RealtimeAudio />

<RealtimeAudio /> creates a hidden <audio> element, attaches it to the client with client.attachAudioElement(), and manages browser autoplay state. The StartAudio component provides a user-gesture gate for browsers that block autoplay on page load:

import { StartAudio } from 'cosmo-ai/react';
<StartAudio />  // renders a button the user clicks to unblock audio

In Python and Swift, LiveKit's platform audio pipeline plays the remote track automatically once the room is joined.

Level metering

For level meters / waveform UIs, every SDK exposes per-direction RMS (0…1). The cadence is per-SDK: TypeScript emits volume events once per animation frame (display-refresh driven), Python's audio_levels() samples at ~20 Hz, and Swift's level streams throttle to ~30 Hz. Each stream yields only once the corresponding track is published / subscribed.

import { useMicLevel, useOutputLevel } from 'cosmo-ai/react';

const mic = useMicLevel();
const out = useOutputLevel();

session.on('volume', ({ mic, output }) => { /* … */ });

async for levels in session.audio_levels():
    print(f"mic={levels.mic:.2f} agent={levels.agent:.2f}")

Task {
    for await level in session.inputLevels {
        // 0…1 RMS of the local mic track
    }
}

Task {
    for await level in session.outputLevels {
        // 0…1 RMS of the agent's remote track
    }
}

Background voice cancellation

Background voices showing up in the transcript, or the agent answering someone who isn't the user? That symptom has one switch: audio.noise_cancellation (noiseCancellation in TypeScript and Swift) on the agent config. It is off by default, and it takes a mode:

The distinction matters most when two people are meant to be heard. voice_focus decides for itself which speaker is primary and treats the rest of the room as background, so on a shared microphone it attenuates whoever is further away — including the person you wanted. Reach for denoise there.

const agent = client.agent({
  instructions: '…',
  audio: { noiseCancellation: 'denoise' },
});

from cosmo_ai import AudioConfig, NoiseCancellation

agent = client.agent(
    instructions="…",
    audio=AudioConfig(noise_cancellation=NoiseCancellation.DENOISE),
)

let agent = try client.agent(
    instructions: "…",
    audio: AudioConfig(noiseCancellation: .denoise)
)

Whichever mode you pick, the filtered signal is also what turn-taking reads, so it costs some barge-in responsiveness — filter for noisy environments (cafés, open offices, call centers) and leave it off for quiet-room or headset use.

Both modes apply to managed WebRTC sessions. A phone leg — session.dial(...) or an inbound call — gets a lighter noise suppressor instead, whatever the mode: it removes background noise from the line but does not single out competing voices, so it already behaves as denoise. The local OSS cosmo-server runs neither filter and rejects any audio block at session start (option_unsupported).

It removes background voices, which no client-side processing does: capture-level noise suppression targets steady noise, not other talkers — and the TypeScript SDK keeps the browser's noise suppression and auto-gain control off, because they duck speech during double-talk and break barge-in. Acoustic echo cancellation (below) is a different job again: it removes the agent's own voice from the microphone, and is on by default in every SDK.

Acoustic echo cancellation (AEC)

AEC — the bit of the audio pipeline that subtracts the speaker's output from the mic input so the agent doesn't hear itself — runs inside the browser's WebRTC stack (for TypeScript) or the LiveKit native SDK (for Python and Swift). You don't configure it. The server relies on AEC being active — it never soft-mutes the mic when the agent is speaking. Soft-muting on bot speech events (bot-started-speaking) would defeat AEC and cause interruption artifacts.

Continuous streaming

The mic streams continuously for the whole session; server-side VAD decides when the user is speaking. What mute does locally differs by SDK: TypeScript and Swift disable the local microphone track before sending the frame, so capture genuinely stops; Python leaves the track publishing and only signals the server. Either way the server stops processing your audio. If you need manual turn boundaries (push-to-talk) or different barge-in behavior, that's turn-taking configuration, not audio configuration. See Turn-taking for interruption_sensitivity, activity-end, and silence-timeout hooks.

Sample rates

LiveKit negotiates the audio codec automatically, and no SDK asks you for a sample rate. In Python, set_microphone_enabled(True) captures through WebRTC's audio device module, which takes the device's own rate and resamples; the browser WebRTC stack does the same in TypeScript.

Pitfalls

• In the TypeScript SDK, mute calls are usable the moment agent.start() resolves; they throw SessionStateError with code not_connected once the session has ended. A session reference obtained before the start resolved (the onSession callback) can await session.waitUntilReady() first.
• In Python, set_microphone_enabled() raises AudioUnavailableError when no input device can be opened — a headless host, a denied permission, or a device another process holds exclusively. set_speaker_enabled() plays through PortAudio and raises the same error when that library is missing; the sounddevice wheel bundles it on macOS and Windows, but on Linux it comes from the distribution.
• <RealtimeAudio /> must be rendered inside <RealtimeProvider>. It attaches to the provider's session internally.