Integrating Voice AI with FreeSWITCH Using a Custom Media Bug Module

FreeSWITCH is a strong foundation for programmable telephony because the core already solves the hard PBX problems: SIP signaling, RTP, codec negotiation, call state, bridging, dialplan execution, and module loading. When you want to build a real-time Voice AI integration, the goal is usually not to replace those pieces. The goal is to attach to a live session, stream caller audio to an AI service, receive synthesized audio back, and inject that audio into the call with low latency.

This post explains how to design that integration with a custom FreeSWITCH module using the Media Bug API. It focuses on architecture, FreeSWITCH APIs, callback flow, and pseudocode. It is not a complete production implementation.

The High-Level Architecture

The cleanest custom-module architecture is to keep call control inside FreeSWITCH and keep AI orchestration outside the media callback thread.

The FreeSWITCH module should do three things:

1. Register a dialplan application or API command that starts and stops the AI integration on a session.
2. Attach a media bug to the session so the module can read live audio frames and optionally replace outbound audio frames.
3. Run network I/O, AI protocol handling, buffering, and timing logic outside the media callback.

The media callback must stay fast. Do not make blocking API calls to ASR, LLM, TTS, databases, or HTTP services directly inside the callback.

Why Use a Custom Media Bug Module?

There are several ways to connect FreeSWITCH to speech systems:

Use a custom media bug when you need direct access to the live audio stream and you need control over both sides of the conversation. That is the common requirement for:

• Conversational AI agents
• Real-time transcription and agent assist
• Barge-in detection
• Streaming TTS playback
• Voice activity detection
• Audio analytics
• Custom provider protocols that do not fit MRCP

If your use case only needs traditional ASR or TTS behavior through existing FreeSWITCH speech applications, the FreeSWITCH ASR and speech interfaces may be a better fit. If you need full-duplex media control, media bugs give you lower-level access.

FreeSWITCH APIs You Will Use

Module Lifecycle

A FreeSWITCH module is a dynamic library loaded by the core. The usual structure is based on mod_skel:

SWITCH_MODULE_LOAD_FUNCTION(mod_voice_ai_load);
SWITCH_MODULE_SHUTDOWN_FUNCTION(mod_voice_ai_shutdown);
SWITCH_MODULE_DEFINITION(mod_voice_ai, mod_voice_ai_load, mod_voice_ai_shutdown, NULL);

SWITCH_MODULE_LOAD_FUNCTION(mod_voice_ai_load)
{
    switch_application_interface_t *app_interface;
    switch_api_interface_t *api_interface;

    *module_interface = switch_loadable_module_create_module_interface(pool, modname);

    SWITCH_ADD_APP(app_interface,
        "voice_ai_start",
        "Start Voice AI",
        "Attach Voice AI media bug to this session",
        voice_ai_start_function,
        "<profile-name>",
        SAF_NONE);

    SWITCH_ADD_API(api_interface,
        "voice_ai",
        "Voice AI control API",
        voice_ai_api_function,
        "start|stop|status <uuid>");

    return SWITCH_STATUS_SUCCESS;
}

A dialplan application is convenient when the AI agent should start during call routing. An API command is useful for external control, debugging, or attaching to an existing UUID.

Session And Channel Access

The session is the call leg. The channel holds call state, variables, caller profile data, flags, and hangup status.

Important APIs:

• switch_core_session_t: call-leg object passed to dialplan applications.
• switch_core_session_get_channel(session): get the channel for variables and state.
• switch_core_session_get_pool(session): allocate session-scoped data.
• switch_core_session_locate(uuid): safely locate and lock a session from another thread.
• switch_core_session_rwunlock(session): release a session found with switch_core_session_locate.

Use the session memory pool for data that should live exactly as long as the call. Use a module-level pool or explicit allocation for global state that outlives a session.

Media Bug Attachment

The core API is:

switch_core_media_bug_add(
    session,
    "voice_ai",
    NULL,
    voice_ai_media_callback,
    context,
    0,
    flags,
    &context->bug
);

Typical flags for a full-duplex Voice AI integration:

switch_media_bug_flag_t flags =
    SMBF_READ_STREAM |
    SMBF_WRITE_STREAM |
    SMBF_READ_PING |
    SMBF_WRITE_REPLACE;

The exact flags depend on your use case:

• SMBF_READ_STREAM: include the read stream so caller audio can be captured.
• SMBF_WRITE_STREAM: include the write stream when you need visibility into outbound audio.
• SMBF_READ_PING: ask FreeSWITCH to call you on read-side pings.
• SMBF_WRITE_REPLACE: allow your callback to replace outbound audio frames with generated audio.

For one-way transcription, you may only need read-side access. For a conversational AI agent that talks back to the caller, SMBF_WRITE_REPLACE is the important piece.

The Media Bug Callback Model

The callback receives a switch_abc_type_t value. The common cases are:

• SWITCH_ABC_TYPE_INIT: initialize callback-local behavior.
• SWITCH_ABC_TYPE_READ or SWITCH_ABC_TYPE_READ_PING: read caller audio.
• SWITCH_ABC_TYPE_WRITE_REPLACE: inject or mix AI audio into the outbound stream.
• SWITCH_ABC_TYPE_CLOSE: stop work and release resources.

Pseudocode: Session Start

The dialplan application should allocate a context, parse configuration, start worker resources, and attach the media bug.

typedef struct voice_ai_context_s {
    switch_core_session_t *session;
    switch_channel_t *channel;
    switch_media_bug_t *bug;
    switch_memory_pool_t *pool;

    switch_mutex_t *mutex;
    switch_queue_t *audio_to_ai;
    switch_queue_t *audio_from_ai;

    switch_bool_t running;
    switch_bool_t caller_is_speaking;
    switch_bool_t agent_is_speaking;
    char *profile_name;
    char *ai_endpoint;
    ai_connection_t *ai_connection;
} voice_ai_context_t;

SWITCH_STANDARD_APP(voice_ai_start_function)
{
    voice_ai_context_t *ctx;
    switch_channel_t *channel = switch_core_session_get_channel(session);
    switch_memory_pool_t *pool = switch_core_session_get_pool(session);
    switch_media_bug_flag_t flags;

    ctx = switch_core_session_alloc(session, sizeof(*ctx));
    memset(ctx, 0, sizeof(*ctx));

    ctx->session = session;
    ctx->channel = channel;
    ctx->pool = pool;
    ctx->profile_name = switch_core_session_strdup(session, data);
    ctx->running = SWITCH_TRUE;

    create_queues_and_mutexes(ctx);
    load_profile_settings(ctx);
    start_ai_worker_thread(ctx);

    flags = SMBF_READ_STREAM | SMBF_READ_PING | SMBF_WRITE_REPLACE;

    if (switch_core_media_bug_add(
            session,
            "voice_ai",
            NULL,
            voice_ai_media_callback,
            ctx,
            0,
            flags,
            &ctx->bug) != SWITCH_STATUS_SUCCESS) {
        ctx->running = SWITCH_FALSE;
        stop_ai_worker_thread(ctx);
        switch_channel_hangup(channel, SWITCH_CAUSE_DESTINATION_OUT_OF_ORDER);
        return;
    }

    switch_channel_set_private(channel, "voice_ai_context", ctx);
}

The important design decision is that the media bug callback owns real-time audio movement, while the worker thread owns slow work such as network I/O and AI protocol handling.

Pseudocode: Reading Caller Audio

On the read side, the callback pulls frames from FreeSWITCH and pushes normalized audio into a queue.

static switch_bool_t voice_ai_media_callback(
    switch_media_bug_t *bug,
    void *user_data,
    switch_abc_type_t type)
{
    voice_ai_context_t *ctx = (voice_ai_context_t *) user_data;

    switch (type) {
    case SWITCH_ABC_TYPE_INIT:
        return SWITCH_TRUE;

    case SWITCH_ABC_TYPE_READ:
    case SWITCH_ABC_TYPE_READ_PING:
        return handle_read_frame(ctx, bug);

    case SWITCH_ABC_TYPE_WRITE_REPLACE:
        return handle_write_replace(ctx, bug);

    case SWITCH_ABC_TYPE_CLOSE:
        ctx->running = SWITCH_FALSE;
        signal_worker_to_stop(ctx);
        return SWITCH_TRUE;

    default:
        return SWITCH_TRUE;
    }
}

static switch_bool_t handle_read_frame(
    voice_ai_context_t *ctx,
    switch_media_bug_t *bug)
{
    switch_frame_t frame = { 0 };
    switch_byte_t data[SWITCH_RECOMMENDED_BUFFER_SIZE];

    frame.data = data;
    frame.buflen = sizeof(data);

    while (switch_core_media_bug_read(bug, &frame, SWITCH_TRUE) == SWITCH_STATUS_SUCCESS) {
        if (!frame.datalen) {
            break;
        }

        /*
         * Keep this fast:
         * 1. Copy frame data into a small audio packet.
         * 2. Optionally mark timestamp, sample rate, and sequence number.
         * 3. Push to a non-blocking queue.
         */
        enqueue_audio_for_ai(ctx->audio_to_ai, frame.data, frame.datalen);
    }

    return SWITCH_TRUE;
}

In production, normalize the audio format expected by your AI provider. Many streaming ASR services expect 16-bit PCM at 8 kHz, 16 kHz, or 24 kHz. FreeSWITCH may negotiate a different codec or sample rate on the call, so inspect the session media implementation and resample when needed.

Pseudocode: AI Worker Thread

The worker thread is the boundary between FreeSWITCH real-time media and external AI services.

static void *SWITCH_THREAD_FUNC ai_worker_thread(
    switch_thread_t *thread,
    void *obj)
{
    voice_ai_context_t *ctx = (voice_ai_context_t *) obj;
    ai_connection_t *conn = ai_connect(ctx->ai_endpoint);

    while (ctx->running && ai_connection_is_open(conn)) {
        audio_packet_t *packet = dequeue_audio(ctx->audio_to_ai, 10);

        if (packet) {
            ai_send_audio(conn, packet->data, packet->len);
            release_audio_packet(packet);
        }

        while (ai_has_event(conn)) {
            ai_event_t event = ai_next_event(conn);

            switch (event.type) {
            case AI_EVENT_PARTIAL_TRANSCRIPT:
                publish_transcript_event(ctx, event.text, SWITCH_FALSE);
                break;

            case AI_EVENT_FINAL_TRANSCRIPT:
                publish_transcript_event(ctx, event.text, SWITCH_TRUE);
                break;

            case AI_EVENT_TTS_AUDIO:
                enqueue_tts_audio(ctx->audio_from_ai, event.audio, event.audio_len);
                break;

            case AI_EVENT_END_OF_TURN:
                mark_agent_turn_complete(ctx);
                break;
            }
        }
    }

    ai_close(conn);
    return NULL;
}

This layer is also where provider-specific logic belongs:

• WebSocket authentication
• ASR message format
• VAD configuration
• LLM prompt state
• Function calls or tool calls
• TTS voice selection
• Audio chunk sizing
• Retry and reconnect behavior

Keeping that logic out of the media callback makes the module easier to reason about and safer under load.

Pseudocode: Injecting AI Audio

For the AI to speak back to the caller, use the write-replace callback. FreeSWITCH provides a write frame; your module can leave it alone, replace it, or mix generated audio into it.

static switch_bool_t handle_write_replace(
    voice_ai_context_t *ctx,
    switch_media_bug_t *bug)
{
    switch_frame_t *frame = switch_core_media_bug_get_write_replace_frame(bug);
    audio_packet_t *tts = NULL;

    if (!frame || !frame->data || !frame->datalen) {
        return SWITCH_TRUE;
    }

    tts = dequeue_tts_audio(ctx->audio_from_ai, 0);

    if (!tts) {
        /*
         * No AI audio is ready. Leave the original write frame unchanged.
         * Depending on the product behavior, you may also inject comfort
         * noise or silence, but do that deliberately.
         */
        return SWITCH_TRUE;
    }

    /*
     * The TTS packet must match the frame's expected format.
     * If it does not, resample/reframe before this point.
     */
    copy_or_mix_audio(frame->data, frame->datalen, tts->data, tts->len);
    switch_core_media_bug_set_write_replace_frame(bug, frame);

    release_audio_packet(tts);
    return SWITCH_TRUE;
}

Two production details matter here:

1. Match the frame size and sample format. Do not push arbitrary TTS chunks directly into the write frame.
2. Decide whether the AI audio should replace the outbound stream or mix with it. For a bot-only call, replacement is usually fine. For agent assist or conference use cases, mixing may be required.

Barge-In And Turn Taking

Conversational agents need interruption handling. If the caller starts speaking while TTS is playing, the module should stop or fade the current agent audio and notify the AI service.

Pseudocode:

if (caller_energy_above_threshold(frame) && ctx->agent_is_speaking) {
    ctx->caller_is_speaking = SWITCH_TRUE;
    clear_queue(ctx->audio_from_ai);
    ai_send_cancel_response(ctx->ai_connection);
    publish_barge_in_event(ctx);
}

You can implement speech detection inside the module, delegate VAD to the AI provider, or combine both. Local VAD is useful because it can stop TTS playback before the remote service finishes processing the new audio.

Events And Control Surface

The module should expose enough observability for operations and application logic. Common events:

• voice_ai::started
• voice_ai::partial_transcript
• voice_ai::final_transcript
• voice_ai::agent_response_started
• voice_ai::agent_response_completed
• voice_ai::barge_in
• voice_ai::error
• voice_ai::stopped

Common commands:

voice_ai start <uuid> <profile>
voice_ai stop <uuid>
voice_ai status <uuid>
voice_ai mute-input <uuid> on|off
voice_ai interrupt <uuid>

When an API command locates a session by UUID, use switch_core_session_locate and always release it with switch_core_session_rwunlock.

SWITCH_STANDARD_API(voice_ai_api_function)
{
    switch_core_session_t *session;

    session = switch_core_session_locate(uuid);
    if (!session) {
        stream->write_function(stream, "-ERR session not found\n");
        return SWITCH_STATUS_SUCCESS;
    }

    control_voice_ai_session(session, cmd, stream);
    switch_core_session_rwunlock(session);

    return SWITCH_STATUS_SUCCESS;
}

Memory, Locking, And Threading Rules

FreeSWITCH modules run inside the media server process, so small mistakes can affect live calls. Treat module code like infrastructure code.

Use these rules:

• Allocate call-scoped context from the session pool.
• Do not store a raw session pointer in a long-running external thread unless you understand the lifetime and locking implications.
• If another thread needs a session, locate it by UUID and unlock it when finished.
• Keep the media bug callback non-blocking.
• Use queues or ring buffers between the callback and worker thread.
• Put upper limits on queue size to avoid unbounded memory growth.
• Handle backpressure by dropping, compressing, or summarizing audio, not by blocking the media thread.
• Make shutdown idempotent because hangup, stop command, and module unload can race.

Audio Format Checklist

Before connecting to an AI provider, define the audio contract:

• Sample rate: 8000, 16000, 24000, or 48000 Hz
• Sample format: usually signed 16-bit linear PCM
• Channels: mono unless the use case needs stereo
• Frame size: commonly 10 ms or 20 ms
• Direction: caller-only, agent-only, mixed, or stereo separated
• Resampling location: inside module, gateway service, or provider SDK
• Silence behavior: pass through, suppress, or synthesize comfort noise

Most integration bugs come from mismatched audio assumptions, not from the media bug attachment itself.

Failure Handling

A production module should define what happens when the AI side fails:

• If ASR disconnects, should the call continue without AI or hang up?
• If TTS is late, should FreeSWITCH play silence, a fallback prompt, or transfer to a human?
• If the AI service returns malformed audio, should the module drop the chunk or reset the stream?
• If the queue is full, which audio is more important: oldest audio or newest audio?
• If the caller hangs up, how quickly does the worker thread stop?

The safest default is to keep the call alive when possible, stop the media bug cleanly, emit an error event, and let dialplan or external orchestration decide the next action.

Security And Privacy Considerations

Voice AI integrations often process sensitive audio. The custom module should make privacy and security explicit:

• Use TLS for external AI connections.
• Keep provider credentials outside dialplan XML.
• Avoid logging raw transcripts unless explicitly enabled.
• Redact account numbers, phone numbers, and secrets from events.
• Apply tenant-level configuration so one customer cannot use another customer profile.
• Define retention rules for audio and transcript data.

Minimal Development Plan

Build the module in stages:

1. Start with a dialplan app that attaches and removes a media bug.
2. Read caller audio and write frame counters to logs.
3. Add a worker thread and queue audio frames to it.
4. Stream audio to a test WebSocket server.
5. Receive generated PCM audio and inject it with SMBF_WRITE_REPLACE.
6. Add transcript events, barge-in, and operational commands.
7. Add reconnect, backpressure, metrics, and load testing.

Do not start with the LLM. First prove the media timing path.

Summary

A custom FreeSWITCH Voice AI module is mainly a real-time systems problem. The Media Bug API gives you the hook into call audio, but the quality of the integration depends on what you build around it: buffering, frame timing, resampling, thread isolation, interruption handling, eventing, and graceful failure behavior.

The core pattern is:

FreeSWITCH session
  -> media bug read callback
  -> non-blocking audio queue
  -> AI worker / gateway
  -> ASR + dialog + TTS
  -> TTS audio queue
  -> media bug write-replace callback
  -> caller hears AI response

That pattern keeps FreeSWITCH responsible for telephony and media scheduling while your AI layer focuses on conversation intelligence.

References

• FreeSWITCH mod_skel module pattern: signalwire/freeswitch/src/mod/applications/mod_skel/mod_skel.c
• FreeSWITCH media bug API declarations: signalwire/freeswitch/src/include/switch_core.h
• FreeSWITCH media bug flags and callback types: signalwire/freeswitch/src/include/switch_types.h
• FreeSWITCH examples using write-replace callbacks: signalwire/freeswitch/src/switch_ivr_async.c
• FreeSWITCH ASR and TTS module reference: SignalWire FreeSWITCH Users Manual