Voice cloning with AI: how it works, what consent means and which models to use

Before you start

Voice cloning means building a voice from a recording of one particular person, so new text comes out sounding like them. The open-model examples below have no training step: you give a short reference clip and new text, and the model speaks the text in that voice. Chatterbox's README calls this zero-shot. The text to speech lesson covers choosing a built-in voice, and this one starts where that stops.

A described voice comes from a text description, and nobody is copied. A cloned voice comes from a recording of a real person, so it carries that person's likeness, and every rule in this lesson follows from that. The steps below need an OpenAI API key for the hosted route and Python for the open models.

No recording needed: describe a voice

OpenAI's prompt-to-voice creates and saves a voice from a description, with no reference recording and no consent recording. It is generally available to all API customers. The saved voice works with GPT-Live, and the guide says it is not supported by the Text to Speech, Realtime or Chat Completions APIs. Creating one is free, and using it in GPT-Live is billed at regular session rates.

You describe qualities, not a person. The guide lists nine, and they are words in your prompt rather than separate API parameters. It also says not to ask for imitation of a named person, a public figure or a fictional character, and that children's voices are not allowed.

curl --fail-with-body https://api.openai.com/v1/audio/voices \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "prompt",
    "name": "Support assistant",
    "prompt": "A calm, reassuring adult voice for a customer support assistant. Speak with an American English accent, a mid-range pitch, a smooth, warm tone, and a steady, conversational pace.",
    "model": "auto"
  }' \
  -o voice.json

1. Write the description from the qualities in the table below, in plain language.
2. Send the request above. Each successful request saves a new voice and returns its metadata, including an id such as voice_123, with no audio or preview.
3. Keep the id and the original description. The guide says to reuse a saved voice when you want the same voice in another session.
4. Listen in a GPT-Live session by setting session.audio.output.voice.id to that id. For voice performance, the guide says to include the description in session.instructions as well.
5. Change one quality at a time and compare using the same short passage.

Clone a voice from a recording: OpenAI's consent flow

A voice from a recording is limited to eligible customers. The guide says to contact OpenAI's sales team, and once your organization is enabled a Voices tab appears under Audio. It takes two separate recordings from the same person: a consent recording, where the speaker reads one set phrase, and a sample recording, which the model tries to copy.

The English phrase is: "I am the owner of this voice and I consent to OpenAI using this voice to create a synthetic voice model." The guide lists the phrase for 16 languages, including Chinese, and warns that any divergence from the script makes the upload fail. One consent can be reused when the same speaker makes more than one voice.

curl https://api.openai.com/v1/audio/voice_consents \
  -X POST \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F "name=test_consent" \
  -F "language=en" \
  -F "recording=@$HOME/tmp/voice_consent/consent_recording.wav;type=audio/x-wav"

curl https://api.openai.com/v1/audio/voices \
  -X POST \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F "name=test_voice" \
  -F "audio_sample=@$HOME/tmp/voice_consent/audio_sample_recording.wav;type=audio/x-wav" \
  -F "consent=cons_123abc"

1. Get access. Without it, the voice endpoints answer with an error such as a 404.
2. Record the consent phrase exactly as written, in a quiet room.
3. Upload it to /v1/audio/voice_consents with a name and its language. The response holds a consent ID.
4. Record the sample, 60 seconds or less, in one of mpeg, wav, ogg, aac, flac, webm or mp4.
5. Create the voice at /v1/audio/voices with the sample and the consent ID.
6. Use the voice's ID in a Realtime session with gpt-realtime-2.1-mini, or in GPT-Live. Choose it at the start, because a different voice needs a new session.

What the sample has to be

The guide says the sample sets the quality. Its advice is a quiet space with little echo, a professional XLR microphone, and a spot about 7 to 8 inches from the mic with a pop filter in between, kept the same throughout. The model copies exactly what it hears, tone, cadence, energy, pauses and habits, so record the voice you want the clone to have. A sample that wanders in energy or accent wanders in the result.

The open models give less advice. Chatterbox's README warns that a reference clip in a different language from the target can pass its accent along, and suggests setting cfg_weight to 0 to reduce that. XTTS-v2's card names 6 seconds as the clip length. The advice above comes from OpenAI's guide, but a quiet room and a steady delivery cost nothing, so keep them for any clip you record.

Clone on your own computer with an open model

Open models need no approval and send nothing to a vendor. None of them has a consent step, so the duty sits with you. F5-TTS is one with a plain command line. It takes a reference recording and, if you have it, the words spoken in that recording, then speaks new text in that voice. Leave the reference text empty and an ASR model transcribes the recording, at the cost of extra GPU memory.

Chatterbox does the same through Python: model.generate(text, audio_prompt_path="your_10s_ref_clip.wav"), where the README's example file is named for a 10-second clip, though the README states no length. Its README says the project was retired in August 2026 and is no longer maintained, so the code stays public as an archive. The commands look alike. The licenses on the weights do not.

pip install f5-tts

# Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage)
f5-tts_infer-cli --model F5TTS_v1_Base \
--ref_audio "provide_prompt_wav_path_here.wav" \
--ref_text "The content, subtitle or transcription of reference audio." \
--gen_text "Some text you want TTS model generate for you."

The rules before you clone anything

Laws about a person's voice differ by country and by state, and this lesson does not cover them. Ask a lawyer before you publish or sell cloned audio. The vendors' own rules are easier to read. OpenAI's pages repeat three of them, and they make a fair baseline for any clone: the speaker consents, nobody is impersonated, and listeners are told.

Under prompt-to-voice, OpenAI's responsible-use list says: use voices responsibly, do not impersonate others, use adult voices, and tell listeners that the speech is AI-generated while preserving any provenance metadata or watermarks. It also says voices are not guaranteed to be unique, since others may generate similar ones. The guide prints that list under described voices. For recorded voices it asks for the consent recording and points to the Text-to-Speech Supplemental Agreement for more terms. Holding a recorded voice to the whole list is this lesson's advice, not a line the guide prints there.

Checking a clip, and what a clean result means

OpenAI's Content Provenance API checks an image or audio file for its own provenance signals. Send the file to POST /v1/content_provenance_checks, or use the web tool at openai.com/verify. For audio, the signal is SynthID, a watermark embedded in the audio itself that may survive some edits. C2PA Content Credentials are signed metadata on images, and editing, converting or sharing a file can strip them.

A not_detected result means the tool found no supported signal. The clip can still come from OpenAI if its metadata was stripped, its watermark degraded, it came from a legacy model or it predates the signals. The tool does not detect other companies' models either, and the page says it is not a general-purpose AI detector. Chatterbox's own watermark has a separate check: its README shows a short script, perth.PerthImplicitWatermarker() followed by get_watermark, that returns 1.0 for watermarked audio and 0.0 for none.

Common mistakes

• Cloning a voice without the speaker's agreement because the tool did not ask for one.
• Reading the consent phrase from memory instead of from the script.
• Recording a sample in a noisy or echoing room, or in a different mood from the one you want, since the model copies both.
• Using F5-TTS's CC-BY-NC weights for paid work.
• Setting a reference clip in one language and the target text in another, then wondering about the accent.
• Treating not_detected as proof that a clip is human.
• Expecting a described voice to work in the Text to Speech API when the guide limits it to GPT-Live.

Try it yourself

• Write a description for a voice, using three of the nine qualities, and change one of them to see what moves.
• Read the F5-TTS command and the Chatterbox line above, and write down which of the two weights you could use in a paid project.
• Record a short clip of your own voice in a quiet room and clone it with an open model, then label the result as synthetic.
• Run a short audio file through the provenance web tool at openai.com/verify and note what a not_detected result does and does not tell you.

Next lesson: AI music

Voices say words. AI music turns a style and some lyrics into a whole song, with its own licensing questions about the models, the hosted tools and what you may release.

Frequently asked questions

Sources

1. OpenAI API docs: Custom voices (read November 6, 2026)https://developers.openai.com/api/docs/guides/custom-voices
2. OpenAI API docs: Content provenance (read November 6, 2026)https://developers.openai.com/api/docs/guides/content-provenance
3. Hugging Face: coqui/XTTS-v2 model card (read November 6, 2026)https://huggingface.co/coqui/XTTS-v2
4. GitHub: resemble-ai/chatterbox README (read November 6, 2026)https://github.com/resemble-ai/chatterbox
5. Hugging Face: ResembleAI/chatterbox model card (read November 6, 2026)https://huggingface.co/ResembleAI/chatterbox
6. GitHub: SWivid/F5-TTS README (read November 6, 2026)https://github.com/SWivid/F5-TTS
7. Hugging Face: SWivid/F5-TTS model card (read November 6, 2026)https://huggingface.co/SWivid/F5-TTS
8. Hugging Face: hexgrad/Kokoro-82M model card (read November 6, 2026)https://huggingface.co/hexgrad/Kokoro-82M