SimulS2ST-Omni

SimulS2ST-Omni

Holds the model files for simultaneous speech-to-speech translation between English and Chinese, built on OmniTalker. For researchers and developers who want to run or study the EMNLP 2026 paper's system.

Made by HA-SA-ki on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: SimulS2ST-Omni is a set of model files for speech-to-speech translation, in English and Chinese, where the output can start before the speaker finishes. The repository bundles a merged OmniTalker model, adapters for streaming use, and the audio components they rely on. The card says the work was accepted by EMNLP 2026.

Who it suits: It suits speech researchers and developers who want to try live spoken translation or reproduce what the paper describes.

Industry | Who uses it | How they use it | Example
Research | Speech and translation researchers | Run the released weights to study simultaneous translation | Comparing offline and streaming modes
Software | Developers of voice apps | Prototype spoken translation between English and Chinese | A demo that translates speech as it comes in
Education | Language teachers and students | Listen to translated speech as an aid to practice | Hearing a short talk rendered in the other language

Skip it if you want a ready-made app, since the card lists files and paths and no end-user program.

Where to use it:

• A Python environment with the transformers library, which the model is listed for
• Windows, macOS or Linux with Python, for downloading the files
• Your own research or product code that reads the model paths from the card

Use case | Who | What they get back
Offline translation | Researchers | Translated speech from a model that sees the whole input
Streaming translation | Developers | Translated speech produced while the source is still being spoken
Component study | Audio engineers | Separate VoiceBox, Vocos and DualCodec files to inspect

How to set it up: You need these first.

• Python with pip, to get the Hugging Face tools
• About 19 GB of disk space, adding up the sizes the card lists for each folder
• Runtime code of your own that can read these model paths, since the card does not name one

1. Install the Hugging Face hub library with pip.
2. Download the repository into a local models folder.
3. Check the files against the manifest in the repository root.
4. Point your code at the paths below.

pip install -U huggingface_hub
hf download HA-SA-ki/SimulS2ST-Omni

hf download HA-SA-ki/SimulS2ST-Omni \
  --local-dir models/SimulS2ST-Omni

models/SimulS2ST-Omni/offline
models/SimulS2ST-Omni/simuls2st_adapter
models/SimulS2ST-Omni/voicebox/voicebox.safetensors
models/SimulS2ST-Omni/voicebox/vocos.safetensors
models/SimulS2ST-Omni/voicebox/voicebox_config.json
models/SimulS2ST-Omni/dualcodec/dualcodec.safetensors
models/SimulS2ST-Omni/dualcodec/w2v_bert_stats.pt
models/SimulS2ST-Omni/w2v

How to use it: You choose a mode by choosing paths. For offline inference, give the offline folder as both the model and the checkpoint. For streaming, give the offline folder as the base model and the adapter folder as the checkpoint.

Tool | What it does
offline/ | Holds the merged OmniTalker model and its talker, processor and tokenizer files
simuls2st_adapter/ | Holds the LoRA adapters and talker parts for streaming translation
voicebox/ | Holds the VoiceBox and Vocos files and their config
dualcodec/ | Holds the DualCodec weights and normalization statistics
w2v/ | Holds a W2V-BERT audio feature model that supports the codec tokenizer

Example: To translate in streaming mode, you load the offline folder as the base and the adapter folder as the checkpoint, then feed it speech in English or Chinese. The card does not show the call, so the exact output depends on the runtime code you use.

For newcomers and experts: If you are new, this repository is the parts box, not the finished machine. A first small step is to download it and confirm the files match the manifest, before you look for runtime code to drive them.

If you know the field, the layout separates a merged offline model from thinker and talker LoRA adapters used for simultaneous work. The audio side adds a codec path with DualCodec, W2V-BERT statistics, VoiceBox and Vocos. A config file records every component's relative path. The trade-off is that the card gives no benchmarks or run script.

Speech comes in
Offline model or streaming adapter reads it
Codec components handle the audio
Translated speech goes out

Good to know:

• Size: about 13 GB for the offline folder, 2.3 GB for voicebox and 2.2 GB for w2v
• Integrity: a manifest file lets you verify every runtime asset
• Languages: English and Chinese
• Maturity: the card has no usage code, metrics or examples

• License: Apache 2.0
• Cost to run: none named, the source lists no API key, account or paid service
• Data it reaches: the audio you pass in and the model files on your disk, and the source names no web or account access
• Not for: the source forbids no uses, but it offers no ready-made app or run script

On Hugging Face, 13 November 2026

• Task: audio to audio
• Library: transformers
• Formats: Safetensors
• Downloads in the last 30 days: 0
• Likes: 2
• License: apache-2.0
• Languages: en, zh
• Tags: transformers, safetensors, speech-to-speech, simultaneous-translation, streaming, qwen2.5-omni, peft, audio-to-audio

Links

• Model: https://huggingface.co/HA-SA-ki/SimulS2ST-Omni
• Files: https://huggingface.co/HA-SA-ki/SimulS2ST-Omni/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download HA-SA-ki/SimulS2ST-Omni

Then follow the usage in its model card. Nothing in this kit runs by itself.