Pipecat backchannel detector

Pipecat backchannel detector

Listens to a user while a voice bot is talking and tells apart a quick "mm-hmm" from a real interruption, so the bot keeps going or stops. A tiny streaming audio model for voice agent builders.

Made by pipecat-ai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a small streaming audio model for voice agents. While the bot is speaking, it decides whether the user is only nodding along, as with a short "mm-hmm" or "right", or is taking the floor, as with "wait" or "no". The bot then keeps talking or stops.

It works on the audio itself, so it decides ahead of the transcript. The model is tiny, at about 4 MB in full precision, and uses roughly 0.3 ms of CPU for each 40 ms chunk of audio.

Who it suits: It suits developers who build voice agents and want interruptions handled quickly, without a bot that either talks over people or stops at every small sound.

Industry | Who uses it | How they use it | Example
Software | Voice agent developers | Add interruption handling to a spoken bot | Keep talking through an mm-hmm
Customer support | Contact center teams | Let callers cut in naturally | Stop at a caller saying wait
Voice products | Product teams | Make bot turn-taking feel natural | Tune the floor threshold
Research | Dialogue researchers | Study backchannel detection in speech | Replay held-out conversations

If your bot does not speak, or you handle languages other than English, you can skip it.

Where to use it: It is meant to sit inside a streaming voice agent pipeline and run on a CPU. You fetch the files on Windows, macOS or Linux with Python.

• A Pipecat style voice agent loop
• An ONNX runtime on a CPU
• English voice conversations

Use case | Who | What they get back
Backchannel check | Voice agent developers | A probability that the user is only nodding
Interruption check | Voice agent developers | A probability that the user is taking the floor
Voice activity | Researchers | An auxiliary user voice activity signal

How to set it up: You need Python with pip to fetch the files and an ONNX runtime to run the model in your voice agent. The repository holds a default fp32 model and an int8 model.

• Python with pip
• An ONNX runtime
• A voice agent that can send 16 kHz audio chunks

1. Install the Hugging Face hub package.
2. Download the repository files.
3. Load the fp32 ONNX file, or the int8 one for a smaller model.

pip install -U huggingface_hub
hf download pipecat-ai/backchannel-detector

How to use it: You call the model once for every 40 ms chunk of the user's microphone audio, after echo cancellation. You also pass the bot's speaking state, and feed each cache output back in on the next call.

Tool | What it does
backchannel-fp32.onnx | The default streaming model
backchannel-int8.onnx | An int8 quantized version
model.safetensors | PyTorch weights for the same network
model.json | Config, chunk size and state tensor names

The output is a set of probabilities over five labels, with backchannel meaning the continuer and assessment labels, and floor meaning the negative and floor-take labels.

Example: The bot is mid-sentence and the user says "mm-hmm". The model reports a high backchannel probability, so the bot keeps talking. If the user says "wait", the floor probability rises and the bot stops.

For newcomers and experts: If you are new, think of it as a listener that tells a voice bot when to stop talking. The first thing to try is to read its model.json to see the chunk size and states it needs.

If you know the field, a causal convolution encoder feeds a GRU, with extra auxiliary outputs for voice activity and a voice activity projection. The card tests it on saved conversations that are played back in real time to the agent.

User audio arrives in 40 ms chunks
The model reads it with bot state
It outputs intent probabilities
The bot keeps talking or stops

Good to know: The card reports results on two sets of held-out conversations, and a median time of about 280 to 300 ms to catch an interruption.

Measure | Open Yap | otoSpeech
Interruptions caught within 700 ms | 82.9 % | 81.2 %
Interruptions lost as backchannels | 2.6 % | 4.1 %
Backchannels the bot talks through | 86.5 % | 75.2 %

• License: bsd-2-clause
• Cost to run: none named, since the model runs on a CPU
• Data it reaches: the audio chunks you feed it, and no training data is included
• Not for: languages other than English, or people talking to a bot, since it was trained on human-to-human talk
• Not for: real phone codecs, which are untested, or real echo cancellation residue

On Hugging Face, 13 November 2026

• Task: audio classification
• Library: onnx
• Parameters: 961K
• Weights: about 4 MB in F32; about 1 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 0
• Likes: 17
• License: bsd-2-clause
• Languages: en
• Tags: onnx, safetensors, pipecat, voice-agents, turn-taking, backchannel, streaming, audio-classification

Links

• Model: https://huggingface.co/pipecat-ai/backchannel-detector
• Files: https://huggingface.co/pipecat-ai/backchannel-detector/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download pipecat-ai/backchannel-detector

Then follow the usage in its model card. Nothing in this kit runs by itself.