VieGem-ASR

VieGem-ASR

Transcribes Vietnamese and mixed Vietnamese and English speech, offline or in 0.48 second streaming chunks, with a 359M-parameter model on Google's EmbeddingGemma 2 audio encoder. Licensed for non-commercial use.

Made by 5CD-AI on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: VieGem-ASR is a speech recognition model for Vietnamese and for Vietnamese mixed with English. It is a 359M-parameter streaming CTC model built on the audio encoder of Google's EmbeddingGemma 2. The author says it was inspired by QUAN-Transcribe-1.0. The same model works on whole files and on live audio.

Who it suits: It suits developers and researchers who need fast Vietnamese transcription and can work within a non-commercial license.

Industry | Who uses it | How they use it | Example
Media | Vietnamese subtitle and transcript teams | Transcribe recorded speech into text | Turn a long interview recording into a draft transcript
Software | Developers building voice features | Feed audio in short chunks for live results | Show captions as someone speaks
Healthcare technology | Teams handling speech with English terms | Check how mixed-language terms are written | Test Vietnamese speech that includes English medical words
Research | Speech researchers | Compare it with other recognizers on the same sets | Score it on the card's Vietnamese test sets

You can skip it if your work is commercial, or if your audio is in a language other than Vietnamese or English.

Where to use it:

• Python with PyTorch and transformers, on a CUDA GPU, as the card's example does
• Windows, macOS and Linux with Python, to download the files
• Offline file transcription, or streaming in 0.48 second chunks with no look-ahead

Use case | Who | What they get back
File transcription | Media teams | The text of a recording of up to 30 seconds per clip
Live captions | App developers | Text from streaming audio chunks
Language-model decoding | Researchers | A transcript decoded with the included n-gram model

How to set it up: You need these first:

• Python, with torch, soundfile and transformers installed
• A CUDA GPU, since the card's example moves the model onto it
• Acceptance of Google's terms for EmbeddingGemma 2, which the code comment asks for
• A 16 kHz mono audio file of at most 30 seconds

1. Install or update the Hugging Face download client.
2. Download this repository, which holds the weights, the asr code folder and the language model.
3. Run the usage script from the folder that contains asr.
4. Optionally install kenlm to decode with the language model.

pip install -U huggingface_hub
hf download 5CD-AI/VieGem-ASR

pip install kenlm

How to use it: You load the checkpoint, read a mono audio file and run the model, then decode the output into text. Set streaming to True for 0.48 second chunks.

import torch, soundfile as sf
from transformers import AutoTokenizer
from asr.model import CTCModel, ModelConfig, greedy_decode     # asr/ folder of this repo
from asr.text import Units

ck = torch.load("viegem_asr.pt", map_location="cpu")
cfg = ModelConfig.from_dict(ck["model_config"])
units = Units(ck["gemma_ids"])
tok = AutoTokenizer.from_pretrained(cfg.encoder_id)               # google/embeddinggemma-2 (accept its terms)
model = CTCModel(cfg, units.gemma_ids)
model.load_state_dict(ck["model"], strict=False)
model = model.cuda().eval()

wav, sr = sf.read("audio.wav", dtype="float32")                  # 16 kHz mono, ≤ 30 s
x = torch.from_numpy(wav)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits, lens = model(x, torch.tensor([x.shape[1]], device="cuda"), streaming=False)  # streaming=True: 0.48 s chunks
print(units.decode(tok, greedy_decode(logits, lens)[0]))

With the language model installed, you decode the same logits through the LM decoder instead.

from asr.lm_decode import LMDecoder

words = open("lm/viegem_lm.unigrams.txt", encoding="utf-8").read().split()
dec = LMDecoder("lm/viegem_lm.bin", unigrams=words, alpha=0.35, beta=1.5)
pieces = ["", ""] + tok.convert_ids_to_tokens(units.gemma_ids)
lp = torch.log_softmax(logits[0, : lens[0]].float(), -1).cpu().numpy()
print(units.decode(tok, dec.decode(lp, pieces, beam_width=32)))

Tool | What it does
CTCModel | Builds the recognizer from the checkpoint settings
greedy_decode | Turns the model output into text by taking the best choice at each step
LMDecoder | Decodes with the included 4-gram language model and a beam search
streaming=True | Processes audio in 0.48 second chunks instead of whole clips

Example: You save a 16 kHz mono clip of Vietnamese speech with a few English terms as audio.wav, then run the script. It prints the recognized text for that clip.

For newcomers and experts: This model turns spoken Vietnamese into text. A good first try is one short clip of clear speech, run through the script above on a machine with a CUDA GPU. Try the language model after that.

For readers who know the field, the encoder is the EmbeddingGemma 2 audio tower, with 12 conformer layers at 25 frames per second and its top 4 layers fine-tuned. Six transformer blocks with chunked attention sit on top, so one model serves offline and streaming use. A CTC head over 21,031 Gemma tokens produces the text. Training used about 5,800 hours of audio.

Read a 16 kHz mono audio clip
Encode it with the EmbeddingGemma 2 audio tower
Run the reader blocks and the CTC head
Decode greedily or with the 4-gram model
Print the transcript

Good to know:

• Hardware: the card's example runs on a CUDA GPU and reports its speed on one H100.
• Speed: the card says about 3,500 times real time batched, and a 10-minute video in 0.49 seconds.
• Results: the numbers come from the authors' own runs with one text normalizer, so check them on your data.
• Downloads: the code fetches the EmbeddingGemma 2 tokenizer by name, which needs Google's terms accepted.

Title: Mean word error rate over 8 Vietnamese test sets, lower is better
VieGem-ASR + LM: 5.29%
VieGem-ASR: 6.34%
QUAN-Transcribe-1.0: 6.89%
Gemini-Flash-3.8: 6.92%
ElevenLabs Scribe v2: 7.08%
PhoWhisper-large: 7.21%
Qwen3-ASR-0.6B: 9.73%
Whisper large-v3: 18.04%

• License: CC BY-NC-SA 4.0, which is non-commercial.
• Cost to run: none named. You need a CUDA GPU.
• Data it reaches: the audio you pass in, plus a download of the encoder's tokenizer from Hugging Face.
• Not for: commercial use, or languages other than Vietnamese and English.

On Hugging Face, 13 November 2026

• Task: automatic speech recognition
• Library: pytorch
• Downloads in the last 30 days: 0
• Likes: 12
• License: cc-by-nc-sa-4.0
• Languages: vi, en
• Based on: google/embeddinggemma-2
• Tags: pytorch, vietnamese, code-switching, streaming, embeddinggemma, automatic-speech-recognition

Links

• Model: https://huggingface.co/5CD-AI/VieGem-ASR
• Files: https://huggingface.co/5CD-AI/VieGem-ASR/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download 5CD-AI/VieGem-ASR

Then follow the usage in its model card. Nothing in this kit runs by itself.