Instructions: Running on AMD 6800XT Radeon using ROCm within a docker container

[milksteakjellybeans]

I had a hard time getting tortoise running with my Radeon 6800xt. I'm sharing these instructions to help whoever else might be struggling. I am not a Docker master, nor have I had any experience with tortoise or rocm before I tried this so keep that in mind, I know there are places that could be simplified. I tried a bunch of different combinations of version of rocm/torch and this is the first I could get to work after trying all day. But the below instructions do work for me and if you've found them while trying to get tortoise working on an AMD card, I hope they are helpful and work for your too.

Prepare Host - ROCm

The host must have rocm installed. Use AMD's instructions:

Prequisites:
https://rocm.docs.amd.com/en/latest/deploy/linux/prerequisites.html

Note: I installed the kernel headers and dev packages. And followed the instructions for group permissions. I expect both are unnecessary for this use case.

And instructions:
https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html

Note: Because we will be using docker, only steps (1) and (2) are required on the host to get the kernel-mode driver installed.

Follow the steps to verify the Kernel-mode Driver and ROCm Installation.

Note: At the time of writing this, I used ROCm 5.7.1 on the host running Ubuntu 20.04.6 LTS

Prepare Host - Tortoise-TTS

Install git if needed.

Make a directory to work out of and cd into it.

Run

git clone https://github.com/neonbjb/tortoise-tts.git

change ./tortoise-tts/requirements.txt as follows

"tokenizers" -> "tokenizers==0.13.3"

otherwise you will get this:

ERROR: transformers 4.31.0 has requirement tokenizers!=0.11.3,<0.14,>=0.11.1, but you'll have tokenizers 0.14.1 which is incompatible.

Docker

Add this "Dockerfile"

FROM ubuntu:20.04

RUN apt update
RUN apt install -y python3 python3-pip libsndfile1 wget

# From https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html  - step 1
RUN mkdir --parents --mode=0755 /etc/apt/keyrings
RUN wget https://repo.radeon.com/rocm/rocm.gpg.key -O /tmp/key
RUN gpg --dearmor /tmp/key
RUN cp /tmp/key.gpg /etc/apt/keyrings/rocm.gpg

# From https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html  - step 3 - Note using 5.6.1 here
RUN echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/5.6.1 focal main" | tee --append /etc/apt/sources.list.d/rocm.list
RUN echo "Package: *" | tee /etc/apt/preferences.d/rocm-pin-600
RUN echo "Pin: release o=repo.radeon.com" | tee --append /etc/apt/preferences.d/rocm-pin-600
RUN echo "Pin-Priority: 600" | tee --append /etc/apt/preferences.d/rocm-pin-600
RUN apt update

# From https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html  - step 4
RUN apt install -y rocm-hip-sdk

COPY ./tortoise-tts /app

# These are probably not needed
ENV HOME "/root"
#ENV CONDA_DIR "${HOME}/miniconda"
#ENV PATH="$CONDA_DIR/bin":$PATH
#ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PIP_DOWNLOAD_CACHE="$HOME/.pip/cache"
ENV TORTOISE_MODELS_DIR="$HOME/tortoise-tts/build/lib/tortoise/models"

WORKDIR /app

RUN python3 -m pip install -r ./requirements.txt

# I could probalby change the requirements.txt to install these in one go, but this works
RUN pip3 uninstall torch torchaudio -y
RUN pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.1.1
RUN python3 setup.py install

Build with something like

docker build . --network=host -t tts-rocm

From scratch, the build took more than 10 mins, and generated build artifacts totaling ~32GB.

Then run it with something like

docker run --rm \
    --device /dev/kfd --device /dev/dri \
    --network=host \
    -e TORTOISE_MODELS_DIR=/models \
    -v ./models:/models \
    -v ./results:/results \
    -v ./huggingface:/root/.cache/huggingface \
    -v /root:/work \
    -it tts-rocm

Test

Test you get "True" back from this call, with no errors:
python3 -c "import torch; print(torch.cuda.is_available());torch.zeros(1).cuda()"

And then try it out:
python3 tortoise/do_tts.py --output_path /results --preset ultra_fast --voice angie --text "hello world"
I got 10-20x speed up.

Caveat:
I do have this warning while running. Quick look seems to indicate it's a pytorch issue I didn't bother going further.

MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx1030_36.kdb Performance may degrade. Please follow instructions to install: https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package

Bonus - here are some of the versions of rocm torch I tried and the outcome

pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.1.1 - Warning above
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.1.3 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.2 - Warning above
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.2.0 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.2.1 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.2.3 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.3 - RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.3.0 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.3.2 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.3.3 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.4 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.0 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.1 - DNE
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2 - RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)
pip3 install torch torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.3 - DNE

[binarydepth]

Can I just change the requirement.txt file and run the usage instructions?

[milksteakjellybeans]

In addition to the change in requirements.txt, you need a version of PyTorch that works with the version of rocm installed. From the last section of the initial post you can see some combinations I tried, only two of which would work to generate audio.

[FWDekker]

The link https://rocm.docs.amd.com/en/latest/deploy/linux/prerequisites.html has died. There's an archived version at https://web.archive.org/web/20230926163713/https://rocm.docs.amd.com/en/latest/deploy/linux/prerequisites.html. This version is for ROCm Platform 5.7.0, which I suppose is close enough to your version, which you note is 5.7.1.

[milksteakjellybeans]

Looks like it got moved to here:
https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/prerequisites.html

[mrllama123]

So from reading the docs above there is already a rocm base image so using the Dockerfile above its can be simplified to:

FROM rocm/dev-ubuntu-22.04:latest

RUN apt update
RUN apt install -y python3 python3-pip libsndfile1 wget

COPY . /app

# These are probably not needed
ENV HOME "/root"
#ENV CONDA_DIR "${HOME}/miniconda"
#ENV PATH="$CONDA_DIR/bin":$PATH
#ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PIP_DOWNLOAD_CACHE="$HOME/.pip/cache"
ENV TORTOISE_MODELS_DIR="$HOME/tortoise-tts/build/lib/tortoise/models"

WORKDIR /app

RUN python3 -m pip install -r ./requirements.txt

# I could probalby change the requirements.txt to install these in one go, but this works
RUN pip3 uninstall torch torchaudio -y
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.7
RUN python3 setup.py install

tested that locally with an AMD Radeon RX 6800 XT

related doc page: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html#using-the-pytorch-rocm-base-docker-image

They also have a pytorch base image as well https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html#using-the-pytorch-rocm-base-docker-image so might be worth trying out

[vicbois390]

@mrllama123 Yes! It works without docker, starting with Ubuntu 22.04, installing ROCm per AMD's instructions, then manually running the components of your dockerfile I am up and running on my 7900XT.

[TriedAllDay]

I got it working on my RX 6800 without docker, but I cannot get deepspeed to work. I get this error
RuntimeError: Error building extension 'transformer_inference'