Refactor formatting, add setup steps to OAI Speech section, add troubleshooting steps for Open WebUI & OAI Speech

2025-12-16 11:37:45 +01:00 · 2024-05-22 14:25:04 -05:00
parent 441d15cbcc
commit bf5b316463
1 changed files with 245 additions and 167 deletions
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # Local LLaMA Server Setup Documentation

-_TL;DR_: A guide to setting up a server for running local language models using [`ollama`](https://ollama.com).
+_TL;DR_: A guide to setting up a fully local and private language model server using [`ollama`](https://ollama.com).

 ## Table of Contents

@@ -11,12 +11,21 @@ _TL;DR_: A guide to setting up a server for running local language models using
  - [Prerequisites](#prerequisites)
  - [Essential Setup](#essential-setup)
  - [Additional Setup](#additional-setup)
+    - [SSH](#ssh)
+    - [Firewall](#firewall)
+    - [Open WebUI](#open-webui)
+      - [Docker](#docker)
+      - [Open WebUI](#open-webui-1)
+    - [OpenedAI Speech](#openedai-speech)
+      - [Open WebUI Integration](#open-webui-integration)
+      - [Downloading Voices](#downloading-voices)
  - [Accessing Ollama](#accessing-ollama)
  - [Troubleshooting](#troubleshooting)
    - [Nvidia drivers](#nvidia-drivers)
    - [`ollama`](#ollama)
-    - [`ssh`](#ssh)
-    - [OpenedAI Speech](#openedai-speech)
+    - [`ssh`](#ssh-1)
+    - [Open WebUI](#open-webui-2)
+    - [OpenedAI Speech](#openedai-speech-1)
  - [Monitoring](#monitoring)
  - [Notes](#notes)
  - [References](#references)
@@ -203,7 +212,7 @@ I also recommend installing a lightweight desktop environment like XFCE for ease

 ## Additional Setup

- ### SSH
+### SSH

 Enabling SSH allows you to connect to the server remotely. After configuring SSH, you can connect to the server from another device on the same network using an SSH client like PuTTY or the terminal. This lets you run your server headlessly without needing a monitor, keyboard, or mouse after the initial setup.

@@ -234,7 +243,7 @@ I also recommend installing a lightweight desktop environment like XFCE for ease

 If you expect to tunnel into your server often, I highly recommend following [this guide](https://www.raspberrypi.com/documentation/computers/remote-access.html#configure-ssh-without-a-password) to enable passwordless SSH using `ssh-keygen` and `ssh-copy-id`. It worked perfectly on my Debian system despite having been written for Raspberry Pi OS.

- ### Firewall
+### Firewall

 Setting up a firewall is essential for securing your server. The Uncomplicated Firewall (UFW) is a simple and easy-to-use firewall for Linux. You can use UFW to allow or deny incoming and outgoing traffic to and from your server.

@@ -258,7 +267,7 @@ I also recommend installing a lightweight desktop environment like XFCE for ease

 Refer to [this guide](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-debian-10) for more information on setting up UFW.

- ### Open WebUI
+### Open WebUI

 In this step, we'll install Docker and Open WebUI. Docker is a containerization platform that allows you to run applications in isolated environments. Open WebUI is a web-based interface for managing Ollama models and chats, and provides a beautiful, performant UI for communicating with your models.

@@ -305,7 +314,7 @@ I also recommend installing a lightweight desktop environment like XFCE for ease
 sudo docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda
 ```

-    You can access it by navigating to `http://localhost:3000` in your browser or `http://(server IP):3000` from another device on the same network. There's no need to add this to the `init.bash` script as Open WebUI will start automatically at boot via Docker Engine.
+You can access it by navigating to `http://localhost:3000` in your browser or `http://(server_IP):3000` from another device on the same network. There's no need to add this to the `init.bash` script as Open WebUI will start automatically at boot via Docker Engine.

 To update Open WebUI once, run the following command:
 ```
@@ -319,36 +328,40 @@ I also recommend installing a lightweight desktop environment like XFCE for ease

 Read more about Open WebUI [here](https://github.com/open-webui/open-webui).

- ### OpenedAI Speech
+### OpenedAI Speech

 OpenedAI Speech is a text-to-speech server that wraps [Piper TTS](https://github.com/rhasspy/piper) and [Coqui XTTS v2](https://docs.coqui.ai/en/latest/models/xtts.html) in an OpenAI-compatible API. This is great because it plugs in easily to the Open WebUI interface, giving your models the ability to speak their responses.

-    > Piper TTS is a more lightweight, less performant model that is great for quick responses - it can also run CPU-only inference, which may be a better fit for systems that need to reserve as much VRAM for language models as possible. However, XTTS v2 is a much more realistic and expressive model. If you have the resources, XTTS is the way to go.
+> WARNING: The documentation for this project leaves a little to be desired. I've run into a plethora of issues trying to get it to work and almost all of it has been trial and error by going through the code. I've done my best to provide a comprehensive guide here but be prepared to troubleshoot in case things have changed when you download it.
+
+Piper TTS is a more lightweight, less performant model that is great for quick responses - it can also run CPU-only inference, which may be a better fit for systems that need to reserve as much VRAM for language models as possible. XTTS is a more performant model that requires a GPU for inference. Piper is:
+
+1) generally easier to setup with out-of-the-box CUDA acceleration, and,
+2) has a plethora of voices that can be found [here](https://rhasspy.github.io/piper-samples/), so it's what I would suggest you start with.

 - To install OpenedAI Speech, first clone the repository and navigate to the directory:
    ```
    git clone https://github.com/matatonic/openedai-speech
    cd openedai-speech
    ```
+- Create a new virtual environment named `speech` and activate it. Then, install `piper-tts`:
+    ```
+    python3 -m venv speech
+    source speech/bin/activate
+    pip install piper-tts
+    ```
+    This is a minimal virtual environment that is only required to run the script that downloads voices.
 - Download the relevant voices. If you intend to use both Piper and XTTS, download both. Otherwise, download the specific voices you need:
    ```
    # for tts-1 / piper
    bash download_voices_tts-1.sh
-        # and for tts-1-hd / xtts
+    # for tts-1-hd / xtts
    bash download_voices_tts-1-hd.sh
    ```
 - Copy the `sample.env` file to `speech.env`:
    ```
    cp sample.env speech.env
    ```
-    - For Nvidia GPUs, edit the `docker-compose.yml` file to use the Nvidia runtime:
-        ```
-        sudo nano docker-compose.yml
-        ```
-        Find and uncomment the following line:
-        ```
-        #runtime: nvidia
-        ```
 - To use the container as a service, also uncomment the following line:
    ```
    #restart: unless-stopped
@@ -363,6 +376,11 @@ I also recommend installing a lightweight desktop environment like XFCE for ease
        #CLI_COMMAND="python speech.py --xtts_device none"
        ```

+        To use CUDA acceleration with Piper, add the following `CLI_COMMAND` line:
+        ```
+        CLI_COMMAND="python speech.py --piper_cuda"
+        ```
+
    - #### Using XTTS
        
        Uncomment the `PRELOAD_MODEL` and `CLI_COMMAND` lines:
@@ -375,7 +393,9 @@ I also recommend installing a lightweight desktop environment like XFCE for ease
    docker compose up -d
    ```

-    OpenedAI Speech runs on `0.0.0.0:8000` by default. You can access it by navigating to `http://localhost:8000` in your browser or `http://(server IP):8000` from another device on the same network without any additional changes.
+OpenedAI Speech runs on `0.0.0.0:8000` by default. You can access it by navigating to `http://localhost:8000` in your browser or `http://(server_IP):8000` from another device on the same network without any additional changes.
+
+#### Open WebUI Integration

 To integrate your OpenedAI Speech server with Open WebUI, navigate to the `Audio` tab under `Settings` in Open WebUI and set the following values:
 - Text-to-Speech Engine: `OpenAI`
@@ -387,6 +407,35 @@ I also recommend installing a lightweight desktop environment like XFCE for ease

 > The TTS engine is set to `OpenAI` because OpenedAI Speech is OpenAI-compatible. There is no data transfer between OpenAI and OpenedAI Speech - the API is simply a wrapper around Piper and XTTS.

+#### Downloading Voices
+
+We'll use Piper here because I haven't found any good resources for high quality .wav files for XTTS. The process is the same for both models, just replace `tts-1` with `tts-1-hd` in the following commands. We'll download the `en_GB-alba-medium` voice as an example.
+```
+bash download_voices_tts-1.sh en_GB-alba-medium
+```
+
+Update the `voice_to_speaker.yaml` file to include the voice you downloaded. This file maps the voice to a speaker name that can be used in the Open WebUI interface. For example, to map the `en_GB-alba-medium` voice to the speaker name `alba`, add the following lines to the file:
+```
+alba:
+    model: voices/en_GB-alba-medium.onnx
+    speaker: # default speaker
+```
+Run the following command:
+```
+sudo docker ps -a
+```
+Identify the container IDs of
+1) OpenedAI Speech
+2) Open WebUI
+
+Restart both containers:
+```
+sudo docker restart (openedai_speech_container_ID)
+sudo docker restart (open_webui_container_ID)
+```
+
+> Replace `(openedai_speech_container_ID)` and `(open_webui_container_ID)` with the container IDs you identified.
+
 ## Accessing Ollama

 This section deals with accessing `ollama` from an extension/application/plugin where you have to specify the base URL to access your `ollama` models. Accessing `ollama` on the server itself is trivial. To test your endpoint, simply run:
@@ -404,7 +453,7 @@ Refer to [Ollama's REST API docs](https://github.com/ollama/ollama/blob/main/doc

 ## Troubleshooting

-For any service running in a container, you can check the logs by running `docker logs -f (container ID)`. If you're having trouble with a service, this is a good place to start.
+For any service running in a container, you can check the logs by running `sudo docker logs -f (container_ID)`. If you're having trouble with a service, this is a good place to start.

 ### Nvidia drivers
 - Disable Secure Boot in the BIOS if you're having trouble with the Nvidia drivers not working. For me, all packages were at the latest versions and `nvidia-detect` was able to find my GPU correctly, but `nvidia-smi` kept returning the `NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver` error. [Disabling Secure Boot](https://askubuntu.com/a/927470) fixed this for me. Better practice than disabling Secure Boot is to sign the Nvidia drivers yourself but I didn't want to go through that process for a non-critical server that can afford to have Secure Boot disabled.
@@ -422,8 +471,34 @@ For any service running in a container, you can check the logs by running `docke
 ### `ssh`
 - If you encounter an issue using `ssh-copy-id` to set up passwordless SSH, try running `ssh-keygen -t rsa` on the client before running `ssh-copy-id`. This generates the RSA key pair that `ssh-copy-id` needs to copy to the server.

+### Open WebUI
+- If you encounter `Ollama: llama runner process has terminated: signal: killed`, check your `Advanced Parameters`, under `Settings > General > Advanced Parameters`. For me, bumping the context length past what certain models could handle was breaking the `ollama` server. Leave it to the default (or higher, but make sure it's still under the limit for the model you're using) to fix this issue.
+
 ### OpenedAI Speech
 - If you encounter `docker: Error response from daemon: Unknown runtime specified nvidia.` when running `docker compose up -d`, ensure that you have `nvidia-container-toolkit` installed (this was previously `nvidia-docker2`, which is now deprecated). If not, installation instructions can be found [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Make sure to reboot the server after installing the toolkit. If you still encounter issues, ensure that your system has a valid CUDA installation by running `nvcc --version`.
+- If `nvcc --version` doesn't return a valid response despite following Nvidia's installation guide, the issue is likely that CUDA is not in your PATH variable.
+    Run the following command to edit your `.bashrc` file:
+    ```
+    sudo nano /home/(username)/.bashrc
+    ```
+    > Replace `(username)` with your username.
+    Add the following to your `.bashrc` file:
+    ```
+    export PATH="/usr/local/(cuda_version)/bin:$PATH"
+    export LD_LIBRARY_PATH="/usr/local/(cuda_version)/lib64:$LD_LIBRARY_PATH"
+    ```
+    > Replace `(cuda_version)` with your installation's version. If you're unsure of which version, run `ls /usr/local` to find the CUDA directory. It is the directory with the `cuda` prefix, followed by the version number. 
+    
+    Save and exit the file, then run `source /home/(username)/.bashrc` to apply the changes (or close the current terminal and open a new one). Run `nvcc --version` again to verify that CUDA is now in your PATH. You should see something like the following:
+    ```
+    nvcc: NVIDIA (R) Cuda compiler driver
+    Copyright (c) 2005-2024 NVIDIA Corporation
+    Built on Thu_Mar_28_02:18:24_PDT_2024
+    Cuda compilation tools, release 12.4, V12.4.131
+    Build cuda_12.4.r12.4/compiler.34097967_0
+    ```
+    If you see this, CUDA is now in your PATH and you can run `docker compose up -d` again.
+- If you run into a `VoiceNotFoundError`, you may either need to download the voices again or the voices may not be compatible with the model you're using. Make sure to check your `speech.env` file to ensure that the `PRELOAD_MODEL` and `CLI_COMMAND` lines are configured correctly.

 ## Monitoring

@@ -439,7 +514,7 @@ For any service running in a container, you can check the logs by running `docke
 - I chose Debian because it is, apparently, one of the most stable Linux distros. I also went with an XFCE desktop environment because it is lightweight and I wasn't yet comfortable going full command line.
 - The power draw of my EVGA FTW3 Ultra RTX 3090 was 350W at stock settings. I set the power limit to 250W and the performance decrease was negligible for my use case, which is primarily code completion in VS Code and the Q&A via chat.
 - Use a user for auto-login, don't log in as root unless for a specific reason.
- If something using a Docker container doesn't work, try running `sudo docker ps -a` to see if the container is running. If it isn't, try running `sudo docker compose up -d` again. If it is and isn't working, try running `sudo docker restart (container ID)` to restart the container.
+- If something using a Docker container doesn't work, try running `sudo docker ps -a` to see if the container is running. If it isn't, try running `sudo docker compose up -d` again. If it is and isn't working, try running `sudo docker restart (container_ID)` to restart the container.
 - If something isn't working no matter what you do, try rebooting the server. It's a common solution to many problems. Try this before spending hours troubleshooting. Sigh.

 ## References
@@ -475,6 +550,9 @@ Firewall:
 Passwordless `ssh`:
 - https://www.raspberrypi.com/documentation/computers/remote-access.html#configure-ssh-without-a-password

+Adding CUDA to PATH:
+- https://askubuntu.com/questions/885610/nvcc-version-command-says-nvcc-is-not-installed
+
 Docs:

 - [Debian](https://www.debian.org/releases/buster/amd64/)