Audio

Audio I/O is handled by LiveKit. Your application publishes the local microphone track; the server publishes the agent's TTS audio as a remote track. The SDK wires both sides for you.

Mic capture

When you call the SDK's microphone-enable method, the SDK:

1. Creates a LocalAudioTrack from the default microphone device.
2. Publishes it to the LiveKit room with source: MICROPHONE.
3. Sends a mute control frame ({ "type": "mute", "muted": false }) so the server starts processing inbound audio from your participant.

TypeScript — use session.setMuted():

// Unmute (enable mic)
await session.setMuted(false);

// Mute without disconnecting
await session.setMuted(true);

The MicToggle component wraps this:

import { MicToggle } from 'cosmo-ai';
<MicToggle />

Python

await session.set_microphone_enabled(True)   # publish track + send mute(false)
await session.set_microphone_enabled(False)  # unpublish track + send mute(true)

Swift

try await session.setMuted(false)   // unmute: mute frame + local capture on
try await session.setMuted(true)    // mute: mute frame + local capture off

Agent audio playback

The agent's TTS output arrives as a remote audio track in the LiveKit room. The TypeScript SDK handles this via <RealtimeAudio />:

import { RealtimeAudio } from 'cosmo-ai';
<RealtimeAudio />

<RealtimeAudio /> creates a hidden <audio> element, attaches it to the client via client.attachAudioElement(), and manages browser autoplay state. The StartAudio component provides a user-gesture gate for browsers that block autoplay on page load:

import { StartAudio } from 'cosmo-ai';
<StartAudio />  // renders a button the user clicks to unblock audio

In Python and Swift, LiveKit's platform audio pipeline plays the remote track automatically once the room is joined.

Level metering

For level meters / waveform UIs, every SDK exposes per-direction RMS (0…1) throttled to ~30 Hz. Each stream yields only once the corresponding track is published / subscribed.

Swift (typed AsyncStream<Float>)

Task {
    for await level in session.inputLevels {
        // 0…1 RMS of the local mic track
    }
}

Task {
    for await level in session.outputLevels {
        // 0…1 RMS of the agent's remote track
    }
}

Latest-value buffered, so a slow consumer drops old values rather than backing up LiveKit's render callback.

Python (async iterator)

async for levels in session.audio_levels():
    print(f"mic={levels.mic:.2f} agent={levels.agent:.2f}")

TypeScript (React hooks or raw volume event)

import { useMicLevel, useOutputLevel } from 'cosmo-ai';

const mic = useMicLevel();
const out = useOutputLevel();

Or, non-React:

session.on('volume', ({ mic, output }) => { /* … */ });

Acoustic echo cancellation (AEC)

AEC — the bit of the audio pipeline that subtracts the speaker's output from the mic input so the agent doesn't hear itself — runs inside the browser's WebRTC stack (for TypeScript) or the LiveKit native SDK (for Python and Swift). You do not configure it. The server relies on AEC being active — it does not soft-mute the mic when the agent is speaking. Soft-muting on bot speech events (bot-started-speaking) would defeat AEC and cause interruption artifacts.

Continuous streaming

The mic streams continuously for the whole session; server-side VAD decides when the user is speaking. There is no client-side audio gating in the protocol — if you need manual turn boundaries (push-to-talk) or different barge-in behavior, that is turn-taking configuration, not audio configuration. See Turn-taking for interruption_sensitivity, activity-end, and silence-timeout hooks.

Sample rates

LiveKit negotiates the audio codec automatically. The Python SDK creates a LocalAudioTrack at 48 kHz / mono when set_microphone_enabled(True) is called:

source = rtc.AudioSource(sample_rate=48000, num_channels=1)
track = rtc.LocalAudioTrack.create_audio_track("mic", source)

The browser WebRTC stack handles resampling. You do not specify a sample rate in the TypeScript SDK.

Pitfalls

• In the TypeScript SDK, mute calls throw RealtimeNotReadyError if made before the server's ready event. Use await session.waitUntilReady() first.
• In Python, set_microphone_enabled() requires the livekit extra. Calling it without the extra raises ImportError.
• <RealtimeAudio /> must be rendered inside <CosmoRealtimeProvider>. It attaches to useRealtimeClient() internally.

See also

• Lifecycle — bot-started-speaking and bot-stopped-speaking events
• Turn-taking — interruption sensitivity, manual turn boundaries, silence handling
• Transport — how audio tracks relate to the data channel