VibeVoice

What it is

VibeVoice is Microsoft’s family of open-source voice models, covering speech to text and text to speech. This listing covers what the repository still lets you run. VibeVoice-ASR takes up to 60 minutes of audio in one pass and returns who spoke, when and what they said. A streaming version transcribes while the audio arrives. VibeVoice-Realtime speaks streaming text in one voice. The long multi-speaker TTS model is left out, because Microsoft removed its code after misuse.

1. You give it a recording of up to 60 minutes
2. You add hotwords such as names and technical terms
3. One pass reads the whole recording, with no slicing
4. It returns who spoke, when and what was said

Who it suits

Developers and researchers who want speech recognition or speech synthesis they can run themselves. The ASR model fits long recordings with several speakers, such as meetings and interviews, where a transcript that says who spoke is the point. The Realtime model is built so that a language model can start speaking from its first tokens. Yet the page’s TODO list still has “Implement streaming text input function to feed new tokens while audio is still being generated” unchecked. Its two usages are a websocket demo and a text-file script. The docs’ install route assumes an NVIDIA GPU in the NVIDIA PyTorch container. The ASR demo script also runs on mps, xpu or cpu. VibeASR.cpp runs the ASR model on a CPU, and the Realtime page names a Mac M4 Pro among the devices that reached real-time speed. Microsoft calls them research models and does not recommend commercial use without further testing.

Skip it if you need a voice of your own or long multi-speaker speech from text: that model is disabled, and the Realtime model ships embedded voices only.

Where to use it

VibeVoice is not an agent skill. It is a set of models you run on your own GPU machine, in a container, or try in a hosted demo.

• An NVIDIA container: the ASR, streaming and Realtime pages all start from the NVIDIA PyTorch container
• A vLLM server in Docker: serves the ASR models as an OpenAI-compatible API, with a streaming variant
• Google Colab: the Realtime demo notebook
• The ASR Playground and Azure AI Foundry Labs: hosted ways to try the ASR model
• Hugging Face Transformers: the VibeVoice-ASR-HF model
• A CPU: VibeASR.cpp, the ASR engine the README links for machines with no GPU

How to set it up

Get it saves a zip of Mindket’s setup notes in your downloads folder, with each model’s steps from Microsoft’s docs. Every route with commands in Microsoft’s docs starts from a clone of the repository.

1. Start the container the docs recommend. Or use your own CUDA environment.
2. Clone the repository and install it with pip.
3. Install ffmpeg for the demos.
4. Start the demo or server for your model.
5. Point it at a recording or a text file.

git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice

pip install -e .

The Realtime model installs with an extra, which pins its own transformers version.

pip install -e .[streamingtts]

How to use it

Pick the script that fits the job. For a recording, transcribe a file or launch the Gradio demo. For text, run the Realtime demo or speak a text file.

python demo/vibevoice_asr_inference_from_file.py --model_path microsoft/VibeVoice-ASR --audio_files [add an audio path here]

python demo/realtime_model_inference_from_file.py --model_path microsoft/VibeVoice-Realtime-0.5B --txt_path demo/text_examples/1p_vibevoice.txt --speaker_name Carter

The streaming ASR script takes --context_info to bias recognition toward the terms you list. The vLLM test scripts take --hotwords for the same job. The docs’ Gradio command ends with --share, which the script describes as “Create a public link”. Drop it, and add --host 127.0.0.1, to keep the demo on your machine: the script binds to 0.0.0.0 on port 7860 by default.

Example

You have an hour-long recording of a team meeting and want a transcript that says who spoke. You run the file command with your own path in place of the bracket.

python demo/vibevoice_asr_inference_from_file.py --model_path microsoft/VibeVoice-ASR --audio_files meeting.wav

The model reads the hour in one pass instead of slicing it into short chunks. The docs say that keeps speaker tracking consistent across the whole recording, and that the output holds who, when and what. You get lines with a speaker and a time range, ready to search or edit.

For newcomers and experts

If you are new. ASR means automatic speech recognition, which is speech to text. TTS is the reverse. Diarization is the “who spoke” part, which VibeVoice-ASR does together with the transcript and the timestamps. A GPU is a graphics card that runs the model fast, and the docs’ container route assumes an NVIDIA one. Start with the ASR Playground or the Realtime Colab notebook, which need no install. Move to your own machine when the audio must stay there.

For experts. The README describes next-token diffusion. A language model reads the text and the dialogue flow, and a diffusion head generates the acoustic detail from continuous tokens at 7.5 Hz. The Realtime model drops the semantic tokenizer and keeps only the acoustic one, with an interleaved, windowed design.

The streaming ASR checkpoint sets its own chunk and lookahead in preprocessor_config.json. The vLLM streaming server reuses the cache of every earlier chunk, so the cost stays flat as a session grows. It has no --dp option, because a session lives in one replica. Run one server per GPU and route whole sessions. The non-streaming server does take --dp and --tp.

The vLLM streaming server speaks WebSocket at /v1/stream and REST at /v1/transcribe and /v1/chat/completions. Its HTTP endpoints take inline base64 audio only, with no HTTP(S) audio URLs, up to 256 MiB decoded per file. The WebSocket takes float32 PCM frames. For fine-tuning, the LoRA defaults are rank 16 and alpha 32. The bundled toy dataset is synthetic audio from VibeVoice TTS, for demonstration only, not a full fine-tuning dataset.

Good to know

Read the maker’s warnings before you build on it.

• Research use: the README says the models are for research and development, and Microsoft does not recommend commercial use or use in live products without further testing
• Misuse: Microsoft removed the VibeVoice-TTS code on 2025-09-05 after finding uses it called inconsistent with its intent, and the TTS page reads “Disabled due to widespread misuse”. The risk text warns of impersonation, fraud and disinformation, and calls it best practice to disclose AI-made audio when you share it
• Wrong output: the models can give output that is unexpected, biased or inaccurate. The TTS and Realtime pages say they inherit the biases of a Qwen2.5 base model. Check a transcript before you rely on it
• Realtime limits: one speaker and English only, with nine experimental languages not extensively tested. Voices are embedded, and for voice customization the page says to reach out to Microsoft’s team. It reads no code or formulas, and is unstable on input of three words or fewer. Streaming text input is still unchecked on the page’s TODO list
• Public links: the docs’ Gradio command ends with --share, and the vLLM demo’s --cloudflared flag downloads a binary and makes one. Leave both off for private audio, and pass --host 127.0.0.1: both demos bind to 0.0.0.0 by default
• Container rights: the docs’ docker run line uses --privileged and --net=host, which give the container wide access to your machine
• Downloads: the demos fetch model weights from Hugging Face, and the experimental voices script pulls nine archives with wget. The model cards on Hugging Face may carry their own terms, and Mindket’s notes did not read them
• Removal: the docs give no uninstall steps. Mindket’s notes give pip uninstall vibevoice, and deleting the clone and the Hugging Face cache
• License: MIT. The LICENSE file reads “Copyright (c) 2025 Microsoft”