VoiceCraft/voicecraft.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyPEhMt0mIcJv2BbaCwogF07",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Sewlell/VoiceCraft-gradio-colab/blob/master/voicecraft.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y87ixxsUVIhM"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/Sewlell/VoiceCraft-gradio-colab"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorboard\n",
        "!pip install phonemizer\n",
        "!pip install datasets\n",
        "!pip install torchmetrics\n",
        "\n",
        "!apt-get install -y espeak espeak-data libespeak1 libespeak-dev\n",
        "!apt-get install -y festival*\n",
        "!apt-get install -y build-essential\n",
        "!apt-get install -y flac libasound2-dev libsndfile1-dev vorbis-tools\n",
        "!apt-get install -y libxml2-dev libxslt-dev zlib1g-dev\n",
        "\n",
        "!pip install -e git+https://github.com/facebookresearch/audiocraft.git@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft\n",
        "\n",
        "!pip install -r \"/content/VoiceCraft-gradio-colab/gradio_requirements.txt\""
      ],
      "metadata": {
        "id": "-w3USR91XdxY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Let it restarted, it won't let your entire installation be gone."
      ],
      "metadata": {
        "id": "jNuzjrtmv2n1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Note before launching the `gradio_app.py`\n",
        "\n",
        "***You will get JSON warning if you move anything beside `sample_batch_size`, `stop_repetition` and `seed`.*** Which for most advanced setting, `kvache` and `temperature` unable to set in different value.\n",
        "\n",
        "You will get fp16 compatibility issue if you set `whisper backend` to `whisperX`, for whatever reason, setting `forced alignment model` to `whisperX` doesn't do anything.\n",
        "\n",
        "You will download a .file File when you download the Output Audio for some reason. You will need to **convert the file from .snd to .wav/.mp3 manually**. Or if you enable showing file type in the name in Windows or wherever you are, change the file name to \"xxx.wav\" or \"xxx.mp3\". (know the solution? pull request my repository)\n",
        "\n",
        "Frequency of VRAM spikes no longer exist as well in April 5 Update.\n",
        "\n",
        "# **To those who want to voice cloning**\n",
        "![Screenshot (438).png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABUoAAAMSCAYAAAC8sv40AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAP+lSURBVHhe7N0JQBXV/gfwL/u+IzsiICICgiiLoiKoiKi455aWlWU9/5X1qme99+pVr15vaU8rM9PKLM3MPdz3DXFXEEFF9n3fl/85c+fCvXCBC1xA4fepkTvLneXMmXNmfnfmjJqBiW09CCGEEEIecl5+QcjISBP72qe+vh62tvYd/j4hhDzMbGzshL9UBhJC+iJeBnau/KuHuromDA2NUFZagoK8POHcknQdNTU1mJqbQ9/AECUlxairq+FDJSM7SV38SwghhBBCCCGEEEIIaRc11NXWoKysFHr6+jC37AdtbW1xHFE1nrY8jXla8zTnaa+qIClHgVJCCCGEEEIIIYQQQjpKTQ011VUoLy+HlrY2LCz7CXc86urpQUNDQ5iE32VKXfs7jqchT0uepjxteRrztOZpztNelejRe0IIIYT0CvToPSGEdAw9ek8I6cs6/+i9DHZOqa6hCS0tTWioa/IBrFNtIK/vkqRlbV0NqqtrJHeSqjhIytEdpYQQQgghhBBCCCGEdJaamtBeZkVFBSoqK1BdUy0E9urqa6nrRCcER1la8jTlaSu0SdoFQVKO7iglhBBCSK9Ad5QSQkjH0B2lhJC+TKV3lMqpl9wEKflEOqEhJCp86No7dOmOUkIIIYQQQgghhBBCVEqN/S/p+Fvaqet4x/6RdF0cJOUoUEoIIYQQQgghhBBCCOnzKFBKCCGEEMLwx+/V1enUiBBCCCGEkL6KrgYIIYQQ0ufxh3hqamqgpaUlGUAIIYQQQgjpcyhQSgghhBDC8DdoamvriH2EEEIIIYSQvoYCpYQQQgghakB5eZkQKNXU1BQHEkIIIYQQQvoSCpQSQgghhEANtbU1qKqqgr6+vjiMEEIIIYQQ0pdQoJQQQgghhFFTA0pLS9hfdejpGYhDCSGEEEIIIX0FBUoJIYQQQgRqqKutQVlZqfBSJwMDQ3oMnxBCCCGEkD6EAqWEEEIIIVJqaqiuqkJ5eSn7qAZDQyPh7lJtbW2oq9NpEyGEEEIIIb2ZmoGJbb34mRBCCCHkoeXlF4SMjDSxr5Pq66GuoSncWcrvKtXQ0BACp4QQ0hvp6UnaZuYvtSOEkL6Gl4FU/hEpCpQSQgghpFdQaaBUUM/jpdDQ0ISmJg+UqvMbTgkhhBBCCCG9FAVKCSGEENIrqD5QKsVOlcSzJTppIoQQQgghpPeiQCkhhBBCeoWuC5QSQgghhBBC+gJ6KwEhhBBCCCGEEEIIIaTPo0ApIYQQQnoH3qAoIYQQQgghhHQQBUoJIYQQ0ivU1tZCXZ1ObQghhBBCCCEdQ1cThBBCCOkVKivLoaWlJfYRQgghhBBCSPtQoJQQQgghvUJJUQG0tbXFPkIIIYQQQghpHwqUEkIIIaRXKMjLgZaWNjQ1NcUhhBBCCCGEEKI8CpQSQgghpFeoqqxEcWEe9PX1xSGEEEIIIYQQojwKlBJCCCGk18jNykBNdTX09ChYSgghhBBCCGkfCpQSQgghpNfgd5XmZKQAdXUwMDCkx/AJIYQQQgghSqNAKSGEEEJ6lbLSUuRmpaKqvBQG+gbC3aX8JU/q6nTaQwghhBBCCGmZmoGJbb34mRBCCCGk19DW0YGeECg1hLaeHjQ1taBGwVJCCCGEEEJICyhQSgghhJBejQdMtbR1hMfw1dQoUEoIIYQQQghRjAKlhBBCCCGEEEIIIYSQPo9uqyCEEEIIIYQQQgghhPR5FCglhBBCCCGEEEIIIYT0eRQoJYQQQgghhBBCCCGE9HkUKCWEEEIIIYQQQgghhPR5FCglhBBCCCGEEEIIIYT0eRQoJYQQQgghhBBCCCGE9HkUKCWEEEIIIYQQQgghhPR5FCglhBBCCCGEEEIIIYT0eRQoJYQQQgghhBBCCCGE9HkUKCWEEEIIIYQQQgghhPR5FCglhBBCCCGEEEIIIYT0eRQoJYQQQgghhBBCCCGE9HkUKCWEEEIIIYQQQgghhPR5FCglhBBCCCGEEEIIIYT0eRQoJYQQQgghhBBCCCGE9HkUKCWEEEIIIYQQQgghhPR5FCglhBBCCCGEEEIIIYT0eRQoJYQQQgghhBBCCCGE9HlqBia29eLnLmVgaAQ9fUNoaetATV0N6uoUoyWEEEJ6g9qaWlRXVaK8rASlJcXi0J4hnG8YGEGbnW+oa9C5BiGEEEIIIb1JXW0dqvi1R2lxl1x7dHmglF+wGBqboa6+Hux/gHXsE9TU1CQTkIeCjZ09CgsLxD5CCCFEilfuauy/etTV1aG2uhpFBbndHjDl5xsmZpbQ0NKEmrqGeL5BCCGEEEII6U2EaCL7p76ull171KAwP0el1x5dGig1tegHHV091NYBFeXlqK2pES6i6oWIKXmYDPIYgoyMVPaJAtyEEELk8R8/+ZMiGhoa0NLSFk5aigryUJCbLU7RtcwsrGBkasbWQQNV1VWora2l8w1CCCGEEEJ6IdlrD2127VHHrj2KC/KRn5slTtE5XfZMmiRIaoDysgqUFBWhqrJSuHChi5aHGO06QgghCvC6ndfxVVVVKC8vQ21dHUzNLYVzga7Gg6SmFpbCMsvYsvk60PkGIYQQQgghvZPstQc//xeuPdj1AL8uUIUuCZTyx9/0DY1RVlaGiooK4a4O8vCjS05CCCFt4XV+ZWUlqqqrYW5pJZwTdBU+bzPLfqhkJ0l8mXS+QQghhBBCSN8hvfbg1wP8ukAV1x5dEig1t7RBWWmZsLKEEEII6Xv4L7w1tXXoZ20vDlG9fjYOwjL4sgghhBBCCCF9U8O1B7s+6CyVB0p59Jbfz1FeXs7vh5UMJIQQQkifU1FRDi0d3S65q5TPU5vNmy+DEEIIIYQQ0rfx6wJ+fdDZaw+VB0pNzCxQWlqKenr8jRBCCOnT+KMw/OVK/NxA1UzM+6Gqih63J4QQQgghhIjXHuz6gF8ndIbKA6W8bVL+4iZCCCGEEH6yYmBoIvapjqGRMSrZvAkhhBBCCCGE49cH/DqhM1QeKNXW1kFtbY3YRwghhJC+rKamBto6OmKf6mix8w0+b0IIIYQQQgjh+PUBv07oDJUHStXU1VFf1/m2STU1NWBhbgYX5wEY5jsUUyZPhJ2tjTiWEEIIIQ+D+vp64dxA1dTU1IR5E0IIIYQQQggnXHuw64TOUDMwsVXpVYbnsEAkxN0U+9rHyNAQI4NGQF9fDwbsc3VllfCK/6qqajg5OeDu3WTsiz4kTk0ajJuDlV7Apc+34jBM4TN+NOyKT2DvuQJxgs4b5DEE6empnc5whBDSl+npsfrNwAA6OjodLk955V9ZWSm0By68OLENqlgm197lyrKxscO12DNin2p4+QUhIyNN7COEEEIIIYSQzl97PFCBUvdBAzElYiLOXbiIiooKdiFW0fB3xIhh0NRQx7btu8WplTECj708AVYZB/CfH2PEYV1tAMZMGwzc2IfjieIgRcY9ilVjHFBxczM+2npXHNhBr36FM1OB7WOfwb8QhX9tWgiP1F8x/ZUt4gSdR4FSQgjpHBMTExgY6KMoPw8V5eWoZ/91hBr7T1dPD8Zm5igtLUNhYaE4pjlVLZNrz3KbokApIYQQQgghpDt09tpD9c/CdQK/mCstK0PMhUtCd/1GHBKT7iItPQNlbHh1dXvbInNBcGQkpgW7iP3dYTimP/oIIr3E3haET52JSWFjMW/6JLiKwzpM3wCGRgbQF3pu4czJMzgR27FgNSGEENXjd3UaGxsh7f49pKfeR2Z6KjLTOtix7/J58HnxefJ5K6LSZfJOyeUSQgghhBBCyMPqgQqUcvyxPn4XqWqoQ0NLHVqa0s20gbvPYLiaA3ZBEZg3ZwbmTRkBO3EsYApXLy+427PxfuMwi4+fPg4+bHopM5fB8Bks21Yqn6fkOzAfAE+fQRjkYgd7Oy/4iMtqbgKmDzNC1tm7qHYJwFTZSKn9QPh4
        "\n",
        "Don't make your input audio too long like in the screenshot, 20s-30s is fine. Or else it will raise up JSON issue. This one is due to how VoiceCraft worked so probably unfixable. It will add those text you want to get audio from at the end of the input audio transcript. It was way too much word for application or code to handle as it added up with original transcript. So please keep it short."
      ],
      "metadata": {
        "id": "qQ-And_w2vJV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "My tip on voice cloning is just \"get a good dataset\" that contain plausible amount of variety of tones. I guess that's it, you could always try experimenting with other voice that are hard to be clone.\n",
        "\n",
        "The inference speed is much stable. With sample text, T4 (Free Tier Colab GPU) can do 6-10s on 6s-8s `prompt end time`.\n",
        "\n",
        "I haven't test the Edit mode yet as those are not my focus, but you can try it."
      ],
      "metadata": {
        "id": "nnu2cY4t8P6X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python \"/content/VoiceCraft-gradio-colab/gradio_app.py\""
      ],
      "metadata": {
        "id": "NDt4r4DiXAwG"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}