talemate/docs/chromadb.md

## ChromaDB

If you want chromaDB to use the more accurate (but much slower) instructor embeddings add the following to `config.yaml`:

```yaml
chromadb:
    embeddings: instructor
    instructor_device: cpu
    instructor_model: hkunlp/instructor-xl
```

You will need to restart the backend for this change to take effect.

**NOTE** - The first time you do this it will need to download the instructor model you selected. This may take a while, and the talemate backend will be un-responsive during that time.

Once the download is finished, if talemate is still un-responsive, try reloading the front-end to reconnect. When all fails just restart the backend as well.

### GPU support

If you want to use the instructor embeddings with GPU support, you will need to install pytorch with CUDA support. 

To do this on windows, run `install-pytorch-cuda.bat` from the project root. Then change your device in the config to `cuda`:

```yaml
chromadb:
    embeddings: instructor
    instructor_device: cuda
    instructor_model: hkunlp/instructor-xl
```

Instructor embedding models:

- `hkunlp/instructor-base` (smallest / fastest)
- `hkunlp/instructor-large` 
- `hkunlp/instructor-xl` (largest / slowest) - requires about 5GB of GPU memory