Merge branch 'main' of https://github.com/iamshreeram/OpenVoice

QA.md

@@ -0,0 +1,30 @@
+# Common Questions and Answers
+
+## General Comments
+
+**OpenVoice is a Technology, not a Product**
+
+Although it works on a majority of voices if used correctly, please do not expect it to work perfectly on every case, as it takes a lot of engineering effort to translate a technology to a stable product. The targeted users of this technology are developers and researchers, not end users. End users expects a perfect product. However, we are confident to say that OpenVoice is the state-of-the-art among the source-available voice cloning technologies.
+
+The contribution of OpenVoice is a versatile instant voice cloning technical approach, not a ready-to-use perfect voice cloning product. However, we firmly believe that by releasing OpenVoice, we can accelerate the open research community's progress on instant voice cloning, and someday in the future the free voice cloning methods will be as good as commercial ones.
+
+## Issues with Voice Quality
+
+**Accent and Emotion of the Generated Voice is not Similar to the Reference Voice**
+
+First of all, OpenVoice only clones the tone color of the reference speaker. It does NOT clone the accent or emotion. The accent and emotion is controlled by the base speaker TTS model, not cloned by the tone color converter (please refer to our [paper](https://arxiv.org/pdf/2312.01479.pdf) for technical details). If the user wants to change the accent or emotion of the output, they need to have a base speaker model with that accent. OpenVoice provides sufficient flexibility for users to integrate their own base speaker model into the framework by simply replacing the current base speaker we provided.
+
+**Bad Audio Quality of the Generated Speech** 
+
+Please check the followings:
+- Is your reference audio is clean enough without any background noise?
+- Is your audio too short?
+- Does your audio contain speech from more than one person?
+- Does the reference audio contain long blank sections?
+- Did you name the reference audio the same name you used before but forgot to delete the `processed` folder?
+
+## Issues with Languages
+
+**Support of Other Languages**
+
+For multi-lingual and cross-lingual usage, please refer to [`demo_part2.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part2.ipynb). OpenVoice supports any language as long as you have a base speaker in that language. The OpenVoice team already did the most difficult part (tone color converter training) for you. Base speaker TTS model is relatively easy to train, and multiple existing open-source repositories support it. If you don't want to train by yourself, simply use the OpenAI TTS model as the base speaker.

README.md

@@ -43,14 +43,16 @@ OpenVoice has been powering the instant voice cloning capability of [myshell.ai]
 ## Live Demo
 
 <div align="center">
-    <a href="https://www.lepton.ai/playground/openvoice"><img src="resources/lepton.jpg"></a>
+    <a href="https://www.lepton.ai/playground/openvoice"><img src="resources/lepton-hd.png" height="28"></a>
     &nbsp;&nbsp;&nbsp;&nbsp;
-    <a href="https://app.myshell.ai/bot/z6Bvua/1702636181"><img src="resources/myshell.jpg"></a>
+    <a href="https://app.myshell.ai/bot/z6Bvua/1702636181"><img src="resources/myshell-hd.png" height="28"></a>
+    &nbsp;&nbsp;&nbsp;&nbsp;
+    <a href="https://huggingface.co/spaces/myshell-ai/OpenVoice"><img src="resources/huggingface.png" height="32"></a>
 </div>
 
-## Disclaimer
+## Common Issues
 
-This is a implementation that approximates the performance of the internal voice clone technology of [myshell.ai](https://app.myshell.ai/explore). The online version in myshell.ai has better 1) audio quality, 2) voice cloning similarity, 3) speech naturalness and 4) computational efficiency.
+Please see [QnA](QA.md) for common questions and answers. We will regularly update the question and answer list.
 
 ## Installation
 Clone this repo, and run
@@ -70,13 +72,12 @@ Please see [`demo_part1.ipynb`](demo_part1.ipynb) for an example usage of how Op
 **2. Cross-Lingual Voice Cloning.**
 Please see [`demo_part2.ipynb`](demo_part2.ipynb) for an example for languages seen or unseen in the MSML training set.
 
-**3. Gradio Demo.**
-Launch a local gradio demo with [`python -m openvoice_app --share`](openvoice_app.py). 
+**3. Gradio Demo.**. We provide a minimalist local gradio demo here. We strongly suggest the users to look into `demo_part1.ipynb`, `demo_part2.ipynb` and the [QnA](QA.md) if they run into issues with the gradio demo. Launch a local gradio demo with `python -m openvoice_app --share`.
 
-**4. Advanced Usage.**
+**3. Advanced Usage.**
 The base speaker model can be replaced with any model (in any language and style) that the user prefer. Please use the `se_extractor.get_se` function as demonstrated in the demo to extract the tone color embedding for the new base speaker.
 
-**5. Tips to Generate Natural Speech.**
+**4. Tips to Generate Natural Speech.**
 There are many single or multi-speaker TTS methods that can generate natural speech, and are readily available. By simply replacing the base speaker model with the model you prefer, you can push the speech naturalness to a level you desire.
 
 ## Roadmap
@@ -100,7 +101,7 @@ There are many single or multi-speaker TTS methods that can generate natural spe
 ```
 
 ## License
-This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. **MyShell reserves the ability to detect whether an audio is generated by OpenVoice**, no matter whether the watermark is added or not.
+This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. **This will be changed to a license that allows Free Commercial usage in the near future.** Stay tuned. For social responsibility and anti-misuse considerations, **MyShell reserves the ability to detect whether an audio is generated by OpenVoice**, no matter whether the watermark is added or not.
 
 
 ## Acknowledgements

demo_part1.ipynb

@@ -148,7 +148,7 @@
     "save_path = f'{output_dir}/output_whispering.wav'\n",
     "\n",
     "# Run the base speaker tts\n",
-    "text = \"This audio is generated by OpenVoice with a half-performance model.\"\n",
+    "text = \"This audio is generated by OpenVoice.\"\n",
     "src_path = f'{output_dir}/tmp.wav'\n",
     "base_speaker_tts.tts(text, src_path, speaker='whispering', language='English', speed=0.9)\n",
     "\n",

openvoice_app.py

@@ -34,7 +34,7 @@ zh_source_se = torch.load(f'{zh_ckpt_base}/zh_default_se.pth').to(device)
 # This online demo mainly supports English and Chinese
 supported_languages = ['zh', 'en']
 
-def predict(prompt, style, audio_file_pth, mic_file_path, use_mic, agree):
+def predict(prompt, style, audio_file_pth, agree):
     # initialize a empty info
     text_hint = ''
     # agree with the terms
@@ -92,22 +92,7 @@ def predict(prompt, style, audio_file_pth, mic_file_path, use_mic, agree):
                 None,
             )
 
-    if use_mic == True:
-        if mic_file_path is not None:
-            speaker_wav = mic_file_path
-        else:
-            text_hint += f"[ERROR] Please record your voice with Microphone, or uncheck Use Microphone to use reference audios\n"
-            gr.Warning(
-                "Please record your voice with Microphone, or uncheck Use Microphone to use reference audios"
-            )
-            return (
-                text_hint,
-                None,
-                None,
-            )
-
-    else:
-        speaker_wav = audio_file_pth
+    speaker_wav = audio_file_pth
 
     if len(prompt) < 2:
         text_hint += f"[ERROR] Please give a longer prompt text \n"
@@ -197,7 +182,7 @@ markdown_table_v2 = """
 """
 content = """
 <div>
-  <strong>For multi-lingual & cross-lingual examples, please refer to <a href='https://github.com/myshell-ai/OpenVoice/blob/main/demo_part2.ipynb'>this jupyter notebook</a>.</strong>
+  <strong>If the generated voice does not sound like the reference voice, please refer to <a href='https://github.com/myshell-ai/OpenVoice/blob/main/QA.md'>this QnA</a>.</strong> <strong>For multi-lingual & cross-lingual examples, please refer to <a href='https://github.com/myshell-ai/OpenVoice/blob/main/demo_part2.ipynb'>this jupyter notebook</a>.</strong>
   This online demo mainly supports <strong>English</strong>. The <em>default</em> style also supports <strong>Chinese</strong>. But OpenVoice can adapt to any other language as long as a base speaker is provided.
 </div>
 """
@@ -208,24 +193,18 @@ examples = [
     [
         "今天天气真好，我们一起出去吃饭吧。",
         'default',
-        "resources/demo_speaker0.mp3",
-        None,
-        False,
+        "resources/demo_speaker1.mp3",
         True,
     ],[
         "This audio is generated by open voice with a half-performance model.",
         'whispering',
-        "resources/demo_speaker1.mp3",
-        None,
-        False,
+        "resources/demo_speaker2.mp3",
         True,
     ],
     [
         "He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered, flour-fattened sauce.",
         'sad',
-        "resources/demo_speaker2.mp3",
-        None,
-        False,
+        "resources/demo_speaker0.mp3",
         True,
     ],
 ]
@@ -268,18 +247,7 @@ with gr.Blocks(analytics_enabled=False) as demo:
                 label="Reference Audio",
                 info="Click on the ✎ button to upload your own target speaker audio",
                 type="filepath",
-                value="resources/demo_speaker0.mp3",
-            )
-            mic_gr = gr.Audio(
-                source="microphone",
-                type="filepath",
-                info="Use your microphone to record audio",
-                label="Use Microphone for Reference",
-            )
-            use_mic_gr = gr.Checkbox(
-                label="Use Microphone",
-                value=False,
-                info="Notice: Microphone input may not work properly under traffic",
+                value="resources/demo_speaker2.mp3",
             )
             tos_gr = gr.Checkbox(
                 label="Agree",
@@ -297,11 +265,11 @@ with gr.Blocks(analytics_enabled=False) as demo:
 
             gr.Examples(examples,
                         label="Examples",
-                        inputs=[input_text_gr, style_gr, ref_gr, mic_gr, use_mic_gr, tos_gr],
+                        inputs=[input_text_gr, style_gr, ref_gr, tos_gr],
                         outputs=[out_text_gr, audio_gr, ref_audio_gr],
                         fn=predict,
                         cache_examples=False,)
-            tts_button.click(predict, [input_text_gr, style_gr, ref_gr, mic_gr, use_mic_gr, tos_gr], outputs=[out_text_gr, audio_gr, ref_audio_gr])
+            tts_button.click(predict, [input_text_gr, style_gr, ref_gr, tos_gr], outputs=[out_text_gr, audio_gr, ref_audio_gr])
 
 demo.queue()  
-demo.launch(debug=True, show_api=True, share=args.share)
+demo.launch(debug=True, show_api=True, share=args.share)

se_extractor.py

@@ -1,23 +1,28 @@
 import os
 import glob
 import torch
+import hashlib
+import librosa
+import base64
 from glob import glob
 import numpy as np
 from pydub import AudioSegment
 from faster_whisper import WhisperModel
+import hashlib
+import base64
+import librosa
 from whisper_timestamped.transcribe import get_audio_tensor, get_vad_segments
 
 model_size = "medium"
 # Run on GPU with FP16
 model = None
-def split_audio_whisper(audio_path, target_dir='processed'):
+def split_audio_whisper(audio_path, audio_name, target_dir='processed'):
     global model
     if model is None:
         model = WhisperModel(model_size, device="cuda", compute_type="float16")
     audio = AudioSegment.from_file(audio_path)
     max_len = len(audio)
 
-    audio_name = os.path.basename(audio_path).rsplit('.', 1)[0]
     target_folder = os.path.join(target_dir, audio_name)
     
     segments, info = model.transcribe(audio_path, beam_size=5, word_timestamps=True)
@@ -69,7 +74,7 @@ def split_audio_whisper(audio_path, target_dir='processed'):
     return wavs_folder
 
 
-def split_audio_vad(audio_path, target_dir, split_seconds=10.0):
+def split_audio_vad(audio_path, audio_name, target_dir, split_seconds=10.0):
     SAMPLE_RATE = 16000
     audio_vad = get_audio_tensor(audio_path)
     segments = get_vad_segments(
@@ -90,7 +95,6 @@ def split_audio_vad(audio_path, target_dir, split_seconds=10.0):
     
     audio_dur = audio_active.duration_seconds
     print(f'after vad: dur = {audio_dur}')
-    audio_name = os.path.basename(audio_path).rsplit('.', 1)[0]
     target_folder = os.path.join(target_dir, audio_name)
     wavs_folder = os.path.join(target_folder, 'wavs')
     os.makedirs(wavs_folder, exist_ok=True)
@@ -111,14 +115,21 @@ def split_audio_vad(audio_path, target_dir, split_seconds=10.0):
         count += 1
     return wavs_folder
 
-
-    
-
+def hash_numpy_array(audio_path):
+    array, _ = librosa.load(audio_path, sr=None, mono=True)
+    # Convert the array to bytes
+    array_bytes = array.tobytes()
+    # Calculate the hash of the array bytes
+    hash_object = hashlib.sha256(array_bytes)
+    hash_value = hash_object.digest()
+    # Convert the hash value to base64
+    base64_value = base64.b64encode(hash_value)
+    return base64_value.decode('utf-8')[:16].replace('/', '_^')
 
 def get_se(audio_path, vc_model, target_dir='processed', vad=True):
     device = vc_model.device
 
-    audio_name = os.path.basename(audio_path).rsplit('.', 1)[0]
+    audio_name = f"{os.path.basename(audio_path).rsplit('.', 1)[0]}_{hash_numpy_array(audio_path)}"
     se_path = os.path.join(target_dir, audio_name, 'se.pth')
 
     if os.path.isfile(se_path):
@@ -127,9 +138,9 @@ def get_se(audio_path, vc_model, target_dir='processed', vad=True):
     if os.path.isdir(audio_path):
         wavs_folder = audio_path
     elif vad:
-        wavs_folder = split_audio_vad(audio_path, target_dir)
+        wavs_folder = split_audio_vad(audio_path, target_dir=target_dir, audio_name=audio_name)
     else:
-        wavs_folder = split_audio_whisper(audio_path, target_dir)
+        wavs_folder = split_audio_whisper(audio_path, target_dir=target_dir, audio_name=audio_name)
     
     audio_segs = glob(f'{wavs_folder}/*.wav')
     if len(audio_segs) == 0:

utils.py

@@ -128,7 +128,7 @@ def merge_short_sentences_latin(sens):
     """
     sens_out = []
     for s in sens:
-        # If the previous sentense is too short, merge them with
+        # If the previous sentence is too short, merge them with
         # the current sentence.
         if len(sens_out) > 0 and len(sens_out[-1].split(" ")) <= 2:
             sens_out[-1] = sens_out[-1] + " " + s