- Create an NFS share on synology for storage
- Add NFS storage pool to proxmox pointed to the NFS share
- Passthrough GPU to VM
- Make sure that the processor type is
x86-64-v3
or some type that supportsavx
andavx512
otherwise ollama will disable gpu support.x86-64-v2-AES
which is usually the default does not support these features
- Make sure that the processor type is
- Install & Setup Ubuntu Server
- Assign a static IP address to the VM
- Install QEMU guest agent
sudo apt install qemu-guest-agent
- Install Nvidia Drivers
sudo ubuntu-drivers install
- Setup NFS stored disk
- Find the disk
lsblk
- Create partition
Whilw in
sudo fdisk /dev/sdb
fsidk
use the following keystrokes
n
>p
>defaults for the rest
>p
to confirm settings >w
to write - format disk
sudo mkfs.ext4 /dev/sdb1
- mount disk
mkdir /mnt/llm-data
sudo mount /dev/sdb1 /mnt/llm-data
chown ollama:ollama /mnt/llm-data
sudo nano /etc/fstab /dev/sdb1 /mnt/llm-data ext4 defaults 0 1
- Find the disk
-
Install Ollama
https://ollama.com/
curl -fsSL https://ollama.com/install.sh | sh
-
Modify Ollama systemd configuration
chown ollama:ollama /mnt/llm-data
sudo systemctl edit ollama.service
[Service] Environment="OLLAMA_HOST=0.0.0.0" Environment="OLLAMA_MODELS=/mnt/llm-data"
optional environment variables for context quantization
Environment="OLLAMA_FLASH_ATTENTION=true" Environment="OLLAMA_KV_CACHE_TYPE=f16"
sudo systemctl daemon-reload
sudo systemctl restart ollama
-
Install a model to use
ollama pull llama3
-
Test Ollama is using GPU
watch -n 1 nvidia-smi
ollama run llama3
-
Install Docker
# Add Docker's official GPG key: sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
-
Install OpenWebUI
sudo docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \ --name open-webui --restart always ghcr.io/open-webui/open-webui:main
-
Use OpenWebUI Open web ui can be loaded by going to
http://localhost:8080
-
Update OpenWebUI
sudo docker pull ghcr.io/open-webui/open-webui:main docker stop open-webui docker rm open-webui
Run the same
Install OpenWebUI
command from above
- Install and Setup
pyenv
sudo apt install -y make build-essential libssl-dev zlib1g-dev \ libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \ libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git
curl https://pyenv.run | bash
export PYENV_ROOT="$HOME/.pyenv" [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init -)"
source .bashrc
- Install Python
pyenv install 3.10
pyenv global 3.10
- Install Automatic1111
mkdir stablediff && cd stablediff
wget -q https://raw.githubusercontent.com/AUTOMATIC1111/stable-diffusion-webui/master/webui.sh
chmod +x webui.sh
./webui.sh --listen --api
- Setup Automatic1111 as a service
File:
sudo vim /etc/systemd/system/automatic1111.service
[Unit]
Description=Stable Diffusion AUTOMATIC1111 Web UI service
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=user
ExecStart=/usr/bin/env bash /home/user/stablediff/webui.sh --api --api-auth username:password
WorkingDirectory=/home/user/stablediff
StandardOutput=append:/var/log/sdwebui.log
StandardError=append:/var/log/sdwebui.log
[Install]
WantedBy=multi-user.target
Note: In my case the user account is simply "user" and path is '/home/user/stablediff'. Please adjust this based on your setup.
Add export PYTHONUNBUFFERED="1"
to /home/user/stablediff/stable-diffusion-webui/webui-user.sh
Then create the /var/log/sdwebui.log
file.
sudo touch /var/log/sdwebui.log
Finally enable the service.
sudo systemctl enable automatic1111.service
Go to models location, remove the existing Stable-diffusion
folder, symlink to your new location.
You can do this for all the other model types as needed
sudo mkdir /mnt/llm-data/sd
cd /home/user/stablediff/stable-diffusion-webui/models
rm -r Stable-diffusion
ln -s /mnt/llm-data/sd Stable-diffusion
sudo systemctl restart automatic1111
Unlink by
cd /home/user/stablediff/stable-diffusion-webui/models
unlink Stable-diffusion
- Ollama: 11434
- OpenWebUI: 8080
- Automatic1111: 7680
Reference List docker volumes
sudo docker volume ls
To get docker volume details and thus their mount points use the name of the volume with the inspect
sub command
sudo docker volume inspect open-webui