RubatoCodec

RubatoCodec

Compresses speech into a fixed number of variable-length events with durations, and decodes them back as it streams. A neural speech codec for people building streaming text-to-speech in English, Japanese and Chinese.

Made by third-intelligence on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: RubatoCodec is a neural speech codec. It splits speech into a fixed number of events, each with discrete code indices and a duration. Because the event count follows the utterance length exactly, a speech generator knows in advance how many tokens it must produce. A causal decoder renders finished events step by step, which suits streaming text-to-speech.

Who it suits: Researchers and developers who build speech generation systems and need a predictable token budget with streaming decoding.

Industry | Who uses it | How they use it | Example
Speech AI | Researchers | Use the codec as the token layer for speech generators | Train a text-to-speech model on its codes
Software | Developers | Add streaming speech decoding to an app | Render speech as events arrive
Research | Audio engineers | Study low-bitrate speech coding | Compare reconstructions at two depths

If you want a ready speech enhancement or transcription tool, you can skip it.

Where to use it:

• Your own computer with Python and PyTorch, preferably with a CUDA GPU
• Streaming text-to-speech pipelines
• Codec comparison and speech generation research

Use case | Who | What they get back
Encoding speech | Researchers | Event codes and durations from a waveform
Decoding speech | Developers | Audio at 44.1 kHz
Lower bitrate decoding | Developers | Audio from only the first layers

How to set it up:

• Python and the uv tool
• The project's package from its GitHub repository, which the card says to install
• The soundfile and torch libraries used in the example
• Room for the weights, about 984 MB in F32, plus the w2v-BERT 2.0 model that loading downloads for encoding

1. Clone the RubatoCodec repository from the project's GitHub page.
2. Move into the folder and sync the environment.
3. Run the encode and decode script below.

cd RubatoCodec
uv sync

To fetch only the codec weights, use the Hugging Face steps.

pip install -U huggingface_hub
hf download third-intelligence/RubatoCodec

How to use it: Load the codec, read a mono waveform, encode it to codes and durations, then decode. Passing a depth value decodes with fewer layers for a lower bitrate.

import soundfile as sf
import torch

from rubatocodec import RubatoCodec

codec = RubatoCodec.from_pretrained("third-intelligence/RubatoCodec", device="cuda")

data, sr = sf.read("speech.wav", dtype="float32")  # mono waveform
wav = torch.from_numpy(data)

enc = codec.encode(wav, sr=sr)       # {"codes": (16, K) int64, "durs": (K,) int64}
audio = codec.decode(enc)            # (1, 1, T_audio) at 44.1 kHz, full depth
audio = codec.decode(enc, depth=8)   # lower bitrate with the first 8 layers

sf.write("recon.wav", audio[0, 0].cpu().numpy(), codec.output_sr)

Tool | What it does
codec.encode | Turns a waveform into event codes and durations
codec.decode | Rebuilds audio from the codes, at full depth or less
depth argument | Uses only the first layers for a lower bitrate

Example: You encode a short mono recording named speech.wav, then decode it with depth 8. The script writes recon.wav, a 44.1 kHz reconstruction made at a lower bitrate than the full-depth decode.

For newcomers and experts: If you are new, run the script on a short clip and compare recon.wav with the original by ear. A codec compresses audio into small codes and rebuilds it, so some detail is lost.

For experts, the quantizer has 16 layers of 4,096 entries each. Layer 0 is semantically supervised FSQ, and layers 1 to 15 are residual VQ. The latent grid runs at 50 Hz and keeps 12.5 events per second. You can decode at any depth from 1 up to 16 layers.

Speech at 16 kHz is encoded
Adaptive firing picks the event positions
Each event gets codes and a duration
A causal decoder renders audio at 44.1 kHz

Good to know: The card reports decoding one event takes about 11 ms on an H200, far less than the audio it represents.

Title: Event decoding time against audio per event
Decode time per event on an H200: 11 ms
Audio each event represents on average: 80 ms

• License: Apache 2.0, as the Hugging Face record lists it.
• Cost to run: no API key or account is named. A CUDA GPU is used in the example.
• Data it reaches: loading downloads the facebook w2v-BERT 2.0 model, used only for encoding. It reads the audio files you give it.
• Not for: the card shows the best intelligibility on English and Japanese benchmarks, but says the strongest comparison codecs score higher on PESQ-WB and STOI.

On Hugging Face, 13 November 2026

• Task: audio to audio
• Parameters: 246M
• Weights: about 984 MB in F32; about 123 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 670
• Likes: 18
• License: apache-2.0
• Languages: en, ja, zh
• Tags: safetensors, audio, speech, neural-audio-codec, speech-tokenizer, variable-frame-rate, streaming, text-to-speech, audio-to-audio, multilingual

Links

• Model: https://huggingface.co/third-intelligence/RubatoCodec
• Files: https://huggingface.co/third-intelligence/RubatoCodec/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download third-intelligence/RubatoCodec

Then follow the usage in its model card. Nothing in this kit runs by itself.