Real-Time Voice Cloning v2

What is this?

It is an improved version of Real-Time-Voice-Cloning. Our emotion voice cloning implementation is here!

Installation

1. Install ffmpeg. This is necessary for reading audio files.

2. Create a new conda environment with

conda create -n rtvc python=3.7.13

3. Install PyTorch. Pick the proposed CUDA version if you have a GPU, otherwise pick CPU. My torch version: torch=1.9.1+cu111 torchvision=0.10.1+cu111

4. Install the remaining requirements with

pip install -r requirements.txt

5. Install spaCy model en_core_web_sm by python -m spacy download en_core_web_sm

Training

Encoder

Download dataset：

1. LibriSpeech: train-other-500 for training, dev-other for validation (extract as <datasets_root>/LibriSpeech/<dataset_name>)

2. VoxCeleb1: Dev A - D for training, Test for validation as well as the metadata file vox1_meta.csv (extract as <datasets_root>/VoxCeleb1/ and <datasets_root>/VoxCeleb1/vox1_meta.csv)

3. VoxCeleb2: Dev A - H for training, Test for validation (extract as <datasets_root>/VoxCeleb2/)

Encoder preprocessing：

python encoder_preprocess.py <datasets_root>

Encoder training：

it is recommended to start visdom server for monitor training with

visdom

then start training with

python encoder_train.py <model_id> <datasets_root>/SV2TTS/encoder

Synthesizer

Download dataset：

1. LibriSpeech: train-clean-100 and train-clean-360 for training, dev-clean for validation (extract as <datasets_root>/LibriSpeech/<dataset_name>)
2. LibriSpeech alignments: merge the directory structure with the LibriSpeech datasets you have downloaded (do not take the alignments from the datasets you haven't downloaded else the scripts will think you have them)
3. VCTK: used for training and validation

Synthesizer preprocessing:

python synthesizer_preprocess_audio.py <datasets_root>
python synthesizer_preprocess_embeds.py <datasets_root>/SV2TTS/synthesizer

Synthesizer training:

python synthesizer_train.py <model_id> <datasets_root>/SV2TTS/synthesizer --use_tb

if you want to monitor the training progress, run

tensorboard --logdir log/synthesizer --host localhost --port 8088

Vocoder

Download dataset：

The same as synthesizer. You can skip this if you already download synthesizer training dataset.

Vocoder preprocessing:

python vocoder_preprocess.py <datasets_root>

Vocoder training:

python vocoder_train.py <model_id> <datasets_root> --use_tb

if you want to monitor the training progress, run

tensorboard --logdir log/vocoder --host localhost --port 8080

Note:

Training breakpoints are saved periodically, so you can run the training command and resume training when the breakpoint exists.

Inference

Terminal:

python demo_cli.py

First input the number of audios, then input the audio file paths, then input the text message. The attention alignments and mel spectrogram are stored in syn_results/. The generated audio is stored in out_audios/.

GUI demo:

python demo_toolbox.py

Dimension reduction visualization

Download dataset:

LibriSpeech: test-other (extract as <datasets_root>/LibriSpeech/<dataset_name>)

Preprocessing:

python encoder_test_preprocess.py <datasets_root>

Visualization:

python encoder_test_visualization.py <model_id> <datasets_root>

The results are saved in dim_reduction_results/.

Pretrained models

You can download the pretrained model from this and extract as saved_models/20230609

Demo results

Reference Audio	Input Text	Synthetic Audio
	Life was like a box of chocolates, you never know what you're gonna get.
	In 2014, P&G recorded $83.1 billion in sales. On August 1, 2014, P&G announced it was streamlining the company, dropping and selling off around 100 brands from its product portfolio in order to focus on the remaining 65 brands, which produced 95% of the company's profits.
	Mechanics is a branch of physics that deals with the behavior of physical bodies under the influence of various forces. The study of mechanics is important in understanding the behavior of machines, the motion of objects, and the principles of engineering. Mechanics has been an essential part of physics since ancient times and has continued to evolve with advancements in science and technology. This paper will discuss the principles of mechanics, the laws of motion, and the applications of mechanics in engineering and technology.
	Life was like a box of chocolates, you never know what you're gonna get.
	In 2014, P&G recorded $83.1 billion in sales. On August 1, 2014, P&G announced it was streamlining the company, dropping and selling off around 100 brands from its product portfolio in order to focus on the remaining 65 brands, which produced 95% of the company's profits.
	Mechanics is a branch of physics that deals with the behavior of physical bodies under the influence of various forces. The study of mechanics is important in understanding the behavior of machines, the motion of objects, and the principles of engineering. Mechanics has been an essential part of physics since ancient times and has continued to evolve with advancements in science and technology. This paper will discuss the principles of mechanics, the laws of motion, and the applications of mechanics in engineering and technology.
	Life was like a box of chocolates, you never know what you're gonna get.
	In 2014, P&G recorded $83.1 billion in sales. On August 1, 2014, P&G announced it was streamlining the company, dropping and selling off around 100 brands from its product portfolio in order to focus on the remaining 65 brands, which produced 95% of the company's profits.
	Mechanics is a branch of physics that deals with the behavior of physical bodies under the influence of various forces. The study of mechanics is important in understanding the behavior of machines, the motion of objects, and the principles of engineering. Mechanics has been an essential part of physics since ancient times and has continued to evolve with advancements in science and technology. This paper will discuss the principles of mechanics, the laws of motion, and the applications of mechanics in engineering and technology.
	Life was like a box of chocolates, you never know what you're gonna get.
	In 2014, P&G recorded $83.1 billion in sales. On August 1, 2014, P&G announced it was streamlining the company, dropping and selling off around 100 brands from its product portfolio in order to focus on the remaining 65 brands, which produced 95% of the company's profits.
	Mechanics is a branch of physics that deals with the behavior of physical bodies under the influence of various forces. The study of mechanics is important in understanding the behavior of machines, the motion of objects, and the principles of engineering. Mechanics has been an essential part of physics since ancient times and has continued to evolve with advancements in science and technology. This paper will discuss the principles of mechanics, the laws of motion, and the applications of mechanics in engineering and technology.