diff --git a/QA.md b/QA.md new file mode 100644 index 0000000..08473f5 --- /dev/null +++ b/QA.md @@ -0,0 +1,30 @@ +# Common Questions and Answers + +## General Comments + +**OpenVoice is a Technology, not a Product** + +Although it works on a majority of voices if used correctly, please do not expect it to work perfectly on every case, as it takes a lot of engineering effort to translate a technology to a stable product. The targeted users of this technology are developers and researchers, not end users. End users expects a perfect product. However, we are confident to say that OpenVoice is the state-of-the-art among the source-available voice cloning technologies. + +The contribution of OpenVoice is a versatile instant voice cloning technical approach, not a ready-to-use perfect voice cloning product. However, we firmly believe that by releasing OpenVoice, we can accelerate the open research community's progress on instant voice cloning, and someday in the future the free voice cloning methods will be as good as commercial ones. + +## Issues with Voice Quality + +**Accent and Emotion of the Generated Voice is not Similar to the Reference Voice** + +First of all, OpenVoice only clones the tone color of the reference speaker. It does NOT clone the accent or emotion. The accent and emotion is controlled by the base speaker TTS model, not cloned by the tone color converter (please refer to our [paper](https://arxiv.org/pdf/2312.01479.pdf) for technical details). If the user wants to change the accent or emotion of the output, they need to have a base speaker model with that accent. OpenVoice provides sufficient flexibility for users to integrate their own base speaker model into the framework by simply replacing the current base speaker we provided. + +**Bad Audio Quality of the Generated Speech** + +Please check the followings: +- Is your reference audio is clean enough without any background noise? +- Is your audio too short? +- Does your audio contain speech from more than one person? +- Does the reference audio contain long blank sections? +- Did you name the reference audio the same name you used before but forgot to delete the `processed` folder? + +## Issues with Languages + +**Support of Other Languages** + +For multi-lingual and cross-lingual usage, please refer to [`demo_part2.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part2.ipynb). OpenVoice supports any language as long as you have a base speaker in that language. The OpenVoice team already did the most difficult part (tone color converter training) for you. Base speaker TTS model is relatively easy to train, and multiple existing open-source repositories support it. If you don't want to train by yourself, simply use the OpenAI TTS model as the base speaker. diff --git a/README.md b/README.md index df618b7..6e7c3d3 100644 --- a/README.md +++ b/README.md @@ -43,14 +43,16 @@ OpenVoice has been powering the instant voice cloning capability of [myshell.ai] ## Live Demo
- +      - + +      +
-## Disclaimer +## Common Issues -This is a implementation that approximates the performance of the internal voice clone technology of [myshell.ai](https://app.myshell.ai/explore). The online version in myshell.ai has better 1) audio quality, 2) voice cloning similarity, 3) speech naturalness and 4) computational efficiency. +Please see [QnA](QA.md) for common questions and answers. We will regularly update the question and answer list. ## Installation Clone this repo, and run @@ -70,13 +72,12 @@ Please see [`demo_part1.ipynb`](demo_part1.ipynb) for an example usage of how Op **2. Cross-Lingual Voice Cloning.** Please see [`demo_part2.ipynb`](demo_part2.ipynb) for an example for languages seen or unseen in the MSML training set. -**3. Gradio Demo.** -Launch a local gradio demo with [`python -m openvoice_app --share`](openvoice_app.py). +**3. Gradio Demo.**. We provide a minimalist local gradio demo here. We strongly suggest the users to look into `demo_part1.ipynb`, `demo_part2.ipynb` and the [QnA](QA.md) if they run into issues with the gradio demo. Launch a local gradio demo with `python -m openvoice_app --share`. -**4. Advanced Usage.** +**3. Advanced Usage.** The base speaker model can be replaced with any model (in any language and style) that the user prefer. Please use the `se_extractor.get_se` function as demonstrated in the demo to extract the tone color embedding for the new base speaker. -**5. Tips to Generate Natural Speech.** +**4. Tips to Generate Natural Speech.** There are many single or multi-speaker TTS methods that can generate natural speech, and are readily available. By simply replacing the base speaker model with the model you prefer, you can push the speech naturalness to a level you desire. ## Roadmap @@ -100,7 +101,7 @@ There are many single or multi-speaker TTS methods that can generate natural spe ``` ## License -This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. **MyShell reserves the ability to detect whether an audio is generated by OpenVoice**, no matter whether the watermark is added or not. +This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. **This will be changed to a license that allows Free Commercial usage in the near future.** Stay tuned. For social responsibility and anti-misuse considerations, **MyShell reserves the ability to detect whether an audio is generated by OpenVoice**, no matter whether the watermark is added or not. ## Acknowledgements diff --git a/demo_part1.ipynb b/demo_part1.ipynb index 065c540..225b7d3 100644 --- a/demo_part1.ipynb +++ b/demo_part1.ipynb @@ -148,7 +148,7 @@ "save_path = f'{output_dir}/output_whispering.wav'\n", "\n", "# Run the base speaker tts\n", - "text = \"This audio is generated by OpenVoice with a half-performance model.\"\n", + "text = \"This audio is generated by OpenVoice.\"\n", "src_path = f'{output_dir}/tmp.wav'\n", "base_speaker_tts.tts(text, src_path, speaker='whispering', language='English', speed=0.9)\n", "\n", diff --git a/openvoice_app.py b/openvoice_app.py index 3922341..65d6854 100644 --- a/openvoice_app.py +++ b/openvoice_app.py @@ -34,7 +34,7 @@ zh_source_se = torch.load(f'{zh_ckpt_base}/zh_default_se.pth').to(device) # This online demo mainly supports English and Chinese supported_languages = ['zh', 'en'] -def predict(prompt, style, audio_file_pth, mic_file_path, use_mic, agree): +def predict(prompt, style, audio_file_pth, agree): # initialize a empty info text_hint = '' # agree with the terms @@ -92,22 +92,7 @@ def predict(prompt, style, audio_file_pth, mic_file_path, use_mic, agree): None, ) - if use_mic == True: - if mic_file_path is not None: - speaker_wav = mic_file_path - else: - text_hint += f"[ERROR] Please record your voice with Microphone, or uncheck Use Microphone to use reference audios\n" - gr.Warning( - "Please record your voice with Microphone, or uncheck Use Microphone to use reference audios" - ) - return ( - text_hint, - None, - None, - ) - - else: - speaker_wav = audio_file_pth + speaker_wav = audio_file_pth if len(prompt) < 2: text_hint += f"[ERROR] Please give a longer prompt text \n" @@ -197,7 +182,7 @@ markdown_table_v2 = """ """ content = """
- For multi-lingual & cross-lingual examples, please refer to this jupyter notebook. + If the generated voice does not sound like the reference voice, please refer to this QnA. For multi-lingual & cross-lingual examples, please refer to this jupyter notebook. This online demo mainly supports English. The default style also supports Chinese. But OpenVoice can adapt to any other language as long as a base speaker is provided.
""" @@ -208,24 +193,18 @@ examples = [ [ "今天天气真好，我们一起出去吃饭吧。", 'default', - "resources/demo_speaker0.mp3", - None, - False, + "resources/demo_speaker1.mp3", True, ],[ "This audio is generated by open voice with a half-performance model.", 'whispering', - "resources/demo_speaker1.mp3", - None, - False, + "resources/demo_speaker2.mp3", True, ], [ "He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered, flour-fattened sauce.", 'sad', - "resources/demo_speaker2.mp3", - None, - False, + "resources/demo_speaker0.mp3", True, ], ] @@ -268,18 +247,7 @@ with gr.Blocks(analytics_enabled=False) as demo: label="Reference Audio", info="Click on the ✎ button to upload your own target speaker audio", type="filepath", - value="resources/demo_speaker0.mp3", - ) - mic_gr = gr.Audio( - source="microphone", - type="filepath", - info="Use your microphone to record audio", - label="Use Microphone for Reference", - ) - use_mic_gr = gr.Checkbox( - label="Use Microphone", - value=False, - info="Notice: Microphone input may not work properly under traffic", + value="resources/demo_speaker2.mp3", ) tos_gr = gr.Checkbox( label="Agree", @@ -297,11 +265,11 @@ with gr.Blocks(analytics_enabled=False) as demo: gr.Examples(examples, label="Examples", - inputs=[input_text_gr, style_gr, ref_gr, mic_gr, use_mic_gr, tos_gr], + inputs=[input_text_gr, style_gr, ref_gr, tos_gr], outputs=[out_text_gr, audio_gr, ref_audio_gr], fn=predict, cache_examples=False,) - tts_button.click(predict, [input_text_gr, style_gr, ref_gr, mic_gr, use_mic_gr, tos_gr], outputs=[out_text_gr, audio_gr, ref_audio_gr]) + tts_button.click(predict, [input_text_gr, style_gr, ref_gr, tos_gr], outputs=[out_text_gr, audio_gr, ref_audio_gr]) demo.queue() -demo.launch(debug=True, show_api=True, share=args.share) \ No newline at end of file +demo.launch(debug=True, show_api=True, share=args.share) diff --git a/resources/framework.jpg b/resources/framework.jpg deleted file mode 100644 index 26fb4b9..0000000 Binary files a/resources/framework.jpg and /dev/null differ diff --git a/resources/huggingface.png b/resources/huggingface.png new file mode 100644 index 0000000..eb5f301 Binary files /dev/null and b/resources/huggingface.png differ diff --git a/resources/lepton-hd.png b/resources/lepton-hd.png new file mode 100644 index 0000000..c8e6c54 Binary files /dev/null and b/resources/lepton-hd.png differ diff --git a/resources/lepton.jpg b/resources/lepton.jpg deleted file mode 100644 index 5bd5601..0000000 Binary files a/resources/lepton.jpg and /dev/null differ diff --git a/resources/myshell-hd.png b/resources/myshell-hd.png new file mode 100644 index 0000000..f62bc37 Binary files /dev/null and b/resources/myshell-hd.png differ diff --git a/resources/myshell.jpg b/resources/myshell.jpg deleted file mode 100644 index 501d7ab..0000000 Binary files a/resources/myshell.jpg and /dev/null differ diff --git a/se_extractor.py b/se_extractor.py index 3ea2b9d..e7d897f 100644 --- a/se_extractor.py +++ b/se_extractor.py @@ -1,23 +1,28 @@ import os import glob import torch +import hashlib +import librosa +import base64 from glob import glob import numpy as np from pydub import AudioSegment from faster_whisper import WhisperModel +import hashlib +import base64 +import librosa from whisper_timestamped.transcribe import get_audio_tensor, get_vad_segments model_size = "medium" # Run on GPU with FP16 model = None -def split_audio_whisper(audio_path, target_dir='processed'): +def split_audio_whisper(audio_path, audio_name, target_dir='processed'): global model if model is None: model = WhisperModel(model_size, device="cuda", compute_type="float16") audio = AudioSegment.from_file(audio_path) max_len = len(audio) - audio_name = os.path.basename(audio_path).rsplit('.', 1)[0] target_folder = os.path.join(target_dir, audio_name) segments, info = model.transcribe(audio_path, beam_size=5, word_timestamps=True) @@ -69,7 +74,7 @@ def split_audio_whisper(audio_path, target_dir='processed'): return wavs_folder -def split_audio_vad(audio_path, target_dir, split_seconds=10.0): +def split_audio_vad(audio_path, audio_name, target_dir, split_seconds=10.0): SAMPLE_RATE = 16000 audio_vad = get_audio_tensor(audio_path) segments = get_vad_segments( @@ -90,7 +95,6 @@ def split_audio_vad(audio_path, target_dir, split_seconds=10.0): audio_dur = audio_active.duration_seconds print(f'after vad: dur = {audio_dur}') - audio_name = os.path.basename(audio_path).rsplit('.', 1)[0] target_folder = os.path.join(target_dir, audio_name) wavs_folder = os.path.join(target_folder, 'wavs') os.makedirs(wavs_folder, exist_ok=True) @@ -111,14 +115,21 @@ def split_audio_vad(audio_path, target_dir, split_seconds=10.0): count += 1 return wavs_folder - - - +def hash_numpy_array(audio_path): + array, _ = librosa.load(audio_path, sr=None, mono=True) + # Convert the array to bytes + array_bytes = array.tobytes() + # Calculate the hash of the array bytes + hash_object = hashlib.sha256(array_bytes) + hash_value = hash_object.digest() + # Convert the hash value to base64 + base64_value = base64.b64encode(hash_value) + return base64_value.decode('utf-8')[:16].replace('/', '_^') def get_se(audio_path, vc_model, target_dir='processed', vad=True): device = vc_model.device - audio_name = os.path.basename(audio_path).rsplit('.', 1)[0] + audio_name = f"{os.path.basename(audio_path).rsplit('.', 1)[0]}_{hash_numpy_array(audio_path)}" se_path = os.path.join(target_dir, audio_name, 'se.pth') if os.path.isfile(se_path): @@ -127,9 +138,9 @@ def get_se(audio_path, vc_model, target_dir='processed', vad=True): if os.path.isdir(audio_path): wavs_folder = audio_path elif vad: - wavs_folder = split_audio_vad(audio_path, target_dir) + wavs_folder = split_audio_vad(audio_path, target_dir=target_dir, audio_name=audio_name) else: - wavs_folder = split_audio_whisper(audio_path, target_dir) + wavs_folder = split_audio_whisper(audio_path, target_dir=target_dir, audio_name=audio_name) audio_segs = glob(f'{wavs_folder}/*.wav') if len(audio_segs) == 0: diff --git a/utils.py b/utils.py index 747a3b7..4e80909 100644 --- a/utils.py +++ b/utils.py @@ -128,7 +128,7 @@ def merge_short_sentences_latin(sens): """ sens_out = [] for s in sens: - # If the previous sentense is too short, merge them with + # If the previous sentence is too short, merge them with # the current sentence. if len(sens_out) > 0 and len(sens_out[-1].split(" ")) <= 2: sens_out[-1] = sens_out[-1] + " " + s