AMD 7900xt GPU (gfx1100 docker image buid) workaround to run SD

[Nikoos]

Hello Team,

After purchasing my first AMD GPU (7900xt) I hit a big stop when I tried to run SD on my new config. After some troubleshooting/reading lot of github issues, I was able to build an unstable (really ! very unstable) docker image with 7900xt (gfx1100 GPU) support for ROCM and Pytorch.

If you want to give a try, here is the actual workflow to build this docker image.

Note: you'll need to have at least up to 40GB of freespace.
Note 2: I am no docker specialist, nor pytorch. This workflow can be simplified, however as it's a community discussion, I hope someone more knowledgeable than me could simplified it

I prefer to re-write: at the moment it's very buggy and unstable so be prepare to have issues.

If you want to try by yourself, here is how I build my image:

1. Git clone pytorch repository:
   git clone --recursive https://github.com/pytorch/pytorch
2. In the .ci/docker/ there is a file named build.sh, at line 203, you'll have the following esac statement:
   Just after the ;; Add the following block:

  pytorch-linux-focal-rocm543-n-py3)
    ANACONDA_PYTHON_VERSION=3.10
    GCC_VERSION=9
    PROTOBUF=yes
    DB=yes
    VISION=yes
    ROCM_VERSION=5.4.3
    NINJA_VERSION=1.9.0
    CONDA_CMAKE=yes
    PYTORCH_ROCM_ARCH=gfx1100
    IMAGE_NAME=pytorch-linux-focal-rocm543-py3.8

So the complete output starting at line 194 is:

  pytorch-linux-focal-rocm-n-py3)
    ANACONDA_PYTHON_VERSION=3.8
    GCC_VERSION=9
    PROTOBUF=yes
    DB=yes
    VISION=yes
    ROCM_VERSION=5.4.3
    NINJA_VERSION=1.9.0
    CONDA_CMAKE=yes
    ;;
  pytorch-linux-focal-rocm543-n-py3)
    ANACONDA_PYTHON_VERSION=3.10
    GCC_VERSION=9
    PROTOBUF=yes
    DB=yes
    VISION=yes
    ROCM_VERSION=5.4.3
    NINJA_VERSION=1.9.0
    CONDA_CMAKE=yes
    PYTORCH_ROCM_ARCH=gfx1100
    IMAGE_NAME=pytorch-linux-focal-rocm543-py3.8
    ;;

Save your file.
3. Now in the uubntu-rocm subfolder of .ci/docker/ you'll have the Dockerfile that will be used to build your image. You'll have to add three lines at the bottom of this file:

WORKDIR /tmp
RUN git clone --recursive https://github.com/pytorch/pytorch.git && cd pytorch && .ci/pytorch/build.sh
RUN git clone https://github.com/pytorch/vision.git && cd vision && FORCE_CUDA=1 python setup.py install

Save your file.

I think that part can be optimized to reduce docker image size
4. Now go back to your previous folder and launch the following command (I assume your user is in the docker group):

./build.sh pytorch-linux-focal-rocm543-n-py3

The build process will be launched (it takes up to 40 minutes on my computer).
5. If the build process finished correctly, you'll have a huge image:

$ docker images
REPOSITORY             TAG       IMAGE ID       CREATED              SIZE
tmp.7iqw2ylj9q         latest    005ef5546f67   About a minute ago   35.9GB
rocm/pytorch-nightly   latest    2cfd54c9e428   2 days ago           38.2GB

Time to tag the image named tmp.7iqw2ylj9q with a more human readable name:

docker image tag 005ef5546f67 localbuild-rocm:latest

6. Once done, you can run your container with a mounted folder that target an empty folder (we will recreate from scratch the python venv) on your host (replace /home/YOURUSER/stable-diffusion-webui/ by your username):

docker run -it --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -p 127.0.0.1:7860:7860 -v /home/YOURUSER/stable-diffusion-webui/:/var/lib/jenkins/stable-diffusion-webui/ localbuild-rocm:latest

7. Once connected in your container, let's reassign group ownership on some important devices:

sudo chgrp video /dev/kfd

8. Now, there is an additional device that could differs from my environment, so you'll have to replace the /dev/dri/renderXXX by the one mounted in your docker container.

sudo chgrp video /dev/dri/renderXXX

9. Now, if you run rocminfo you must see your device detected, if not, no need to continue, you won't be able to run SD.
   Example:

jenkins@f56a43abba09:~/stable-diffusion-webui$ rocminfo
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
                      

[...]

*******                  
Agent 2                  
*******                  
  Name:                    gfx1100                            
  Uuid:                    GPU-XX                             
  Marketing Name:                                             
  Vendor Name:             AMD   

10. If step 9 is OK, let's test with pytorch to confirm that it's also working, run the python3 interpreter and type the following code:

import torch

It will load torch python module, then to check if torch is able to work your GPU, you can use the following command (note: the >>> should not be typed):

>>> torch.cuda.is_available()
True
>>> torch.cuda.current_device()
0
>>> torch.cuda.get_device_name(0)
'Radeon RX 7900 XT'

If your GPU appears, you're good to go to the next step.
11. Let's git clone stable diffusion repo in your $HOMEDIR (in my case /var/lib/jenkins) :

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git stable-diffusion-webui/

12. Once clone, we will build SD venv based on the compiled version from this docker image:

python3 -m venv --copies --system-site-packages venv

13. To check if everything is correct, if you try to launch:

pip list

You'll have a very long list of python packages, that list, should be identical in SD's venv, to confirm that, source SD's venv

source venv/bin/activate

Then, if you use the same command as above pip list you must have the same content, if it's identical, you can go to next and final step : updating sd to listen on 0.0.0.0
14. Edit webui-user.sh and replace the following line (line 13):

#export COMMANDLINE_ARGS=""

by

export COMMANDLINE_ARGS="--listen"

(note: if you want to install extension, I found that option --enable-insecure-extension-access should be added to this command line too).
15. All modifications are done, now it's time to start SD with ./webui.sh

Regards,

Nikos

[cmlibo]

晚上回去试试这个方案，19:30到货的7900xt，折腾到凌晨3点，都没有运行起来，要么找不到GPU，找到GPU了，又出现其他错误。都有点后悔为什么不买4080

[Boom-Hacker]

n卡驱动不开源，不可信

[Boom-Hacker]

当然我知道没什么人在乎开源信仰但是少一个风险也是少

[cmlibo]

rocm5.5发布后，目前7900xt可以使用SD了，本机还是docker，自己编译的版本都可以成功出图，就是早没有当初的干劲了，出完测试图，就切到Windows玩游戏去了