Before you start
Dubbing replaces the speech in a video with speech in another language, and it splits into four jobs. Transcribe the original speech into text. Translate the text. Speak the translation in a voice. Fit the new audio to the video, which for a speaker on screen includes lip sync. Each job hands its output to the next, so a mistake in the transcript turns into a mistake in the voice.
Two earlier lessons cover the voice. Text to speech is how you speak the translation, and voice cloning is how you keep the original speaker's voice, with their consent. Use a video you made or have permission to translate, and start with 30 seconds of it.
Step 1: transcribe the speech
Dubbing starts from text. OpenAI's file transcription guide says to start with gpt-transcribe for recorded speech in its original language. Files can be up to 25 MB, in mp3, mp4, mpeg, mpga, m4a, wav or webm. For a longer video, the guide says to compress the audio or split it into chunks of 25 MB or less, and to avoid cutting in the middle of a sentence.
Three optional fields help with names and terms. prompt gives context in plain words, keywords lists the exact terms you expect to hear, and languages names the languages spoken. The guide calls keywords hints, not required output, so check that they help without making unspoken terms appear. The request below is the guide's example with a new file name and new example words.
curl https://api.openai.com/v1/audio/transcriptions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F model="gpt-transcribe" \
-F file="@/path/to/file/talk.mp3" \
-F 'prompt=A product demo for a note-taking app called Notely.' \
-F 'keywords[]=Notely' \
-F 'languages[]=en'- The reply is JSON with the transcript in
textand alanguageslist holding the detected language codes. The list is empty when detection is not reliable. - To keep the audio on your own computer, use Whisper. Its code and weights are MIT-licensed,
pip install -U openai-whisperinstalls it, and it needs ffmpeg.whisper audio.mp3 --model turbotranscribes, and the README says the default works well for English. - The guide's examples for word timestamps and subtitle formats use deprecated models, and those models shut down on 26 February 2027. For timed text, use a tool that aligns words itself, such as VideoLingo below.
- Read the transcript once while the video plays. Every error here is copied into the translation and the voice.
Step 2: translate the transcript
There are three ways to get a translation. Whisper can translate speech straight into English: whisper japanese.wav --model medium --language Japanese --task translate. Its README warns that the turbo model returns the original language even when --task translate is set, so use medium or large.
OpenAI's older /v1/audio/translations endpoint also translates a recording into English text only, and the guide lists it as deprecated, shutting down on 26 February 2027. For speech that is still being spoken, the guide points to gpt-realtime-translate, which streams translated audio and a transcript while the speaker talks. It needs one session per target language, so English to Spanish and English to French take two.
For any other pair, and for a recording, send the transcript to a chat model. This prompt is the lesson's own, not a vendor's. Paste a few numbered lines at a time.
Translate these subtitle lines from English into Spanish.
Keep the numbers and the number of lines the same.
Keep each line close to the length of the original, because a voice will say it in the same time.
Leave these terms in English: Notely, workspace.
Return only the numbered lines.
1. Welcome to Notely.
2. In this video, I will show you how to share a workspace.- Keep the numbering, so each translated line maps back to its timestamps.
- Give a glossary of names and terms that stay in English. VideoLingo uses custom and AI-generated terminology for the same goal, a coherent translation from start to end.
- Ask for lines about as long as the originals. The README says dubbing timing depends on the translation, the voice service and the speech rate.
- Have someone who speaks the language read the lines before you voice them. The README says translation quality depends on the source audio, the language and the models.
Step 3: speak the translation
This is the text to speech from lesson one, run once per line. Pick a stock voice, or clone the speaker's voice with their consent, as lesson two explains. OpenAI's guide says its voices are optimized for English and follow the Whisper language list for other languages. VideoLingo says its dubbing languages depend on the voice method you choose, and lists OpenAI, Fish Audio, SiliconFlow Fish and CosyVoice2, GPT-SoVITS, Edge TTS and F5-TTS, plus a custom adapter.
VideoLingo's README is blunt about this step: speed adjustment does not guarantee natural delivery or perfect synchronization. The lesson's advice follows from that, which is to shorten a sentence that is too long for its slot in the translation instead of leaning on speed. The README also says the dubbing workflow does not automatically assign a separate voice to each speaker, so for a two-person interview, voice each speaker's lines separately.
Step 4: lip sync, only if a face is on screen
New words change what the mouth should be doing. A lip-sync model changes the face in the video so it matches the new audio, and the Wav2Lip README says it works for any identity, voice and language. If the speaker is off screen, or the video shows slides or a screen recording, skip this step.
The three open projects below differ most in license. The Wav2Lip command shows the shape of the job: a video, an audio file and a checkpoint go in, and the result lands in results/result_voice.mp4. Its README says the audio can be any file ffmpeg reads, including a video.
python inference.py --checkpoint_path <ckpt> --face <video.mp4> --audio <an-audio-source>| Wav2Lip | MuseTalk | LatentSync | |
|---|---|---|---|
| License, as the README words it | Research, academic and personal use only | Code MIT, model "available for any purpose, even commercially" | Not stated in the README, so check the repository |
| Hardware the README names | Not stated | Tested on a 4 GB laptop GPU in fp16, where 8 seconds of video took about 5 minutes | 8 GB for version 1.5, 18 GB for version 1.6 |
| How you start it | The inference.py command above | python app.py opens a Gradio page | python gradio_app.py opens a Gradio page |
| A tip from the README | 720p often looks better than 1080p | Use input video at 25 fps | inference_steps from 20 to 50: higher looks better and runs slower |
Sources: GitHub: Rudrabha/Wav2Lip README (read November 6, 2026), GitHub: TMElyralab/MuseTalk README (read November 6, 2026), GitHub: bytedance/LatentSync README (read November 6, 2026)
One tool for steps 1 to 3: VideoLingo in a browser
VideoLingo is an Apache 2.0 project that combines speech recognition, subtitle translation, segmentation and dubbing in a Streamlit interface. It produces subtitle files and, if you ask, subtitled or dubbed videos. Its input languages are English, Russian, French, German, Italian, Spanish, Japanese and Chinese. On Windows the README gives a one-click route.
git clone https://github.com/Huanshere/VideoLingo.git && cd VideoLingo
uv run start.py- The two lines above are the README's route for macOS, Linux and Windows from source. Run
uv run start.pyagain from the folder to start it later. - Speech recognition runs on your computer by default. If you pick a cloud provider in the sidebar, the README says your audio goes to that provider and may incur charges.
- Download Source code (zip) from the latest Release on the project's GitHub page, extract it and open the folder.
- Double-click
OneKeyStart.batand keep the window open. The first run installs uv, Python 3.12, the dependencies and FFmpeg, and it needs an internet connection. - When VideoLingo opens in your browser, enter your API URL, key and model in the sidebar. The model must come from an OpenAI-compatible chat provider and be able to return the structured JSON the workflow needs.
- Start with a short video, and read the subtitles it makes before you turn on dubbing.
Where each step goes wrong
VideoLingo's README keeps a list of current limitations, and Whisper's README adds one. They are a fair map of the places a dub breaks, whichever tool you use.
| Step | What goes wrong | First thing to check |
|---|---|---|
| Transcribe | Background noise and language-specific alignment models affect recognition and word timestamps | Compare the loudest scene with its subtitles, and try separating the voice from the music first |
| Transcribe | Numbers and symbols may lack reliable word timings | Read every subtitle line that holds a number |
| Transcribe | Whisper's performance varies widely by language | Test a 30-second clip in your language before the whole video |
| Transcribe | Mixed-language speech is not guaranteed to keep accurate text and timing in every language | Split the video where the language changes |
| Translate | The chat model's reply has to satisfy the workflow's JSON structure | If a run fails, the README says to inspect output/gpt_log/error.json instead of deleting all output first |
| Speak | Timing and delivery depend on the translation, the voice service and the speech rate | Listen to the longest sentences first and shorten their translation |
| Speak | No separate voice is assigned to each speaker automatically | Voice each speaker's lines on their own |
Check the dub before you publish it
Play the dub beside the original and look for four things.
- Drift. Compare the end of each scene with the original, because a voice that runs long falls behind the picture there.
- Names and numbers. These are the words transcription misses most, and one wrong digit in a price or a date is a real error.
- A native ear. Ask one speaker of the target language to listen once with the sound on.
- The disclosure. OpenAI's text to speech guide says its usage policies require clear disclosure to end users that the voice is AI-generated, and its custom voices guide asks you to keep any provenance metadata and watermarks.
- Where the audio went. A cloud provider receives the recording: VideoLingo's README says audio is sent to the selected provider and may incur charges. Read that provider's own terms for audio before you upload.
Common mistakes
- Dubbing a whole video before testing 30 seconds of it.
- Skipping the transcript read, then finding a wrong name in every sentence.
- Translating word for word, so the voice runs out of time before the sentence ends.
- Running
--model turbowith--task translateand getting the original language back. - Giving two speakers one voice in a tool that does not assign voices.
- Using Wav2Lip on a paid job, when its README prohibits commercial use.
- Leaving out the line that tells viewers the voice is AI-generated.
Try it yourself
- Cut a 30-second clip with one speaker, transcribe it with
gpt-transcribeorwhisper --model turbo, and mark every error. - Translate the transcript with the prompt above and ask someone who speaks the language to read it.
- Install VideoLingo, make subtitles only, and compare them with your transcript.
- Voice the three longest sentences and time each one against the original.
Next course: Vibe coding
You can now make speech, a voice, a song and a dub. Vibe coding is a different kind of making: you describe an app to an AI tool, then check what comes back.
Check yourself
In what order do the four dubbing jobs run?
Show answer
C. Transcribe, translate, speak, fit The text has to exist before it can be translated, and the translation has to exist before a voice can say it. Fitting the audio to the picture comes last.
Which Whisper model returns the original language even when you set --task translate?
Show answer
D. turbo The README says turbo is not trained for translation and tells you to use medium or large for it.
What does the Wav2Lip README allow?
Show answer
A. Research, academic and personal use only The README says its results should be used for research, academic or personal purposes only and that any form of commercial use is strictly prohibited.
Does VideoLingo's dubbing workflow automatically give each speaker a separate voice?
Show answer
B. No, it does not assign one automatically The README lists this among its current limitations, so a video with several speakers needs each speaker's lines voiced by hand.
Frequently asked questions
How do I dub a video into another language with AI?
Start small. Run one speaker and one language through every job, play the result beside the original, and only then repeat it on the full video. The steps above show each job and where it tends to break.
Is there a free AI dubbing tool?
VideoLingo is free to download under the Apache 2.0 license and runs on your own computer. The services it calls, such as a chat model with your API key or a cloud speech provider, can charge you, so check each one before you start.
Can Whisper translate into languages other than English?
Its README shows translation into English only. For another target language, transcribe first and send the text to a chat model.
Do I need lip sync?
Only when the speaker's mouth is visible. For slides, a screen recording or a speaker off screen, a voice-over is enough. Check the license first, because Wav2Lip's is non-commercial.
Which languages can I dub into?
The voice sets the limit, not the translation. Chatterbox's Multilingual V3 lists 23 or more languages, though that project was retired in August 2026, and OpenAI's voices are tuned for English. Check a voice's language list before you translate into it.
Sources
- OpenAI API docs: File transcription https://developers.openai.com/api/docs/guides/speech-to-text
- OpenAI API docs: Live translation https://developers.openai.com/api/docs/guides/realtime-translation
- OpenAI API docs: Text to speech https://developers.openai.com/api/docs/guides/text-to-speech
- OpenAI API docs: Custom voices https://developers.openai.com/api/docs/guides/custom-voices
- GitHub: openai/whisper README https://github.com/openai/whisper
- GitHub: Huanshere/VideoLingo README https://github.com/Huanshere/VideoLingo
- GitHub: Rudrabha/Wav2Lip README https://github.com/Rudrabha/Wav2Lip
- GitHub: TMElyralab/MuseTalk README https://github.com/TMElyralab/MuseTalk
- GitHub: bytedance/LatentSync README https://github.com/bytedance/LatentSync
- GitHub: resemble-ai/chatterbox README https://github.com/resemble-ai/chatterbox