LABCRAFT

Files for homelab provisioning and maintenance operations of my personal proxmox cluster for self-hosted services, application deployment environment and playhouse :)

ARCHITECTURE

The machine runs proxmox cluster with vms. The main purpose of the server is to expose web interfaces of docker containers for some services that i use every day

---
title: torterra
---
flowchart LR
subgraph web_services
direction TB
A[(wailord)]
B{staraptor}
B --http requests--> A
end
subgraph dns_servers
direction TB
C[espeon]
D[umbreon]
C ~~~ D
end
web_services --dns queries--> dns_servers

Loading

NETWORKING

some services are exposed to the internet via HTTPS reverse proxy with nginx

flowchart LR
A((Internet))
B{staraptor}
C[nextcloud]
D[...]
C & D --> B
B --> A

Loading

some other services are exposed through port forwarding on the router

flowchart LR
A((Internet))
B{router\n port forwarding}
C[wireguard]
C --> B
B --> A

Loading

---

DISKS MANAGEMENT

Containers and virtual machines's rootfs disk is located in the local-lvm volume on the nvme disk. all the volumes are backuped in the other hard drive from pbs

flowchart
	subgraph data disks
		direction TB
		subgraph nvme
				A[container rootfs]
		end
	end
	subgraph backupdisks
		direction TB
		subgraph HD2
			direction LR
			C[backup volume]
		end
	end
	A -- backup on --> C

Loading

BACKUPS MANAGEMENT

This infrastructure manages all of my backups, the backup centralizer is an lxc container with an external volume mounted with data inside

flowchart
    subgraph ditto
		subgraph main-lvm-storage
        A[rootfs]
		end
		subgraph secondary-storage
        B["/mnt/datastore"]
		end
    end

Loading

all of my personal pc use borg for managing backup locally and then copy content to the centralizer machine using rsync, backup is achieved trough a script that runs as a systemd timer

sequenceDiagram
participant laptop
participant ditto
laptop ->> laptop: creates backup
laptop ->> ditto: sync changes
Note over laptop,ditto: connection secured trough vpn

Loading

vms and containers backups are managed trough proxmox backup server installed on the centralizer

proxmox host backups are done trough the use of borg and a cronjob script, after running the configure_proxmox.yml playbook configure ssh authentication to storage box

INSTALLATION

• clone repository inside the proxmox host

cd /usr/local
git clone https://github.com/carnivuth/labcraft

• create venv and install dependencies

cd labcraft
python -m venv env
source env/bin/activate
pip install -r requirements.txt

• install ansible collections and roles

source env/bin/activate
ansible-galaxy collection install -r collections/requirements.yml
ansible-galaxy role install -r roles/requirements.yml

• add secrets folowing this guide

• create terraform vars file following the vars declaration in terraform/variables.tf

• create a proxmox admin token for terraform

• create templates for vms and containers following this

• run terraform to deploy vms and add one of the dns servers to /etc/hosts

• run preflight playbook for provisioning

ansible-playbook -i inventory/prod.proxmox.yml carnivuth.labcraft.preflight

HANDLE SECRETS

Sensitive informations are stored inside an encrypted vault file generated with ansible-vault, in order to create it do the following:

• create a sample with the following command:

grep -e 'vault_[a-z_]*' playbooks/group_vars/all/vars.yml inventory/inventory.proxmox.yml  -ho > sample.yml

• create a file to store the vault password

pwgen -N 1 64 > passfile && chmod 600 passfile

• set vault pass file in ansible.cfg

[defaults]
host_key_checking = False
vault_password_file=/usr/local/labcraft/passfile

• add variables and encrypt the file with ansible vault

ansible-vault encrypt sample.yml

• move the file to the group_vars folder

mv sample.yml playbooks/group_vars/all/vault.yml

UPDATE MANAGEMENT

To avoid having to run ansible manually every time there is an update do the following

• add the scripts/update_labcraft.sh to cron:

* * * * * /usr/local/labcraft/update_labcraft.sh > /dev/null 2>&1

Then link workflows/middleware.sh to the git hooks dir (more on the topic here) as follows

cd .git/hooks
ln -fs ../../workflows/middleware.sh post-merge

So every time a commit is pushed to remote cron will pull the repo and the hook will run ansible

---
title: UPDATE WORKFLOW
---
sequenceDiagram
participant dev_machine
participant github_repo
participant torterra

dev_machine ->> github_repo: push chainges
loop every x minutes
torterra ->> github_repo: fetch changes
alt changes
torterra ->> torterra: run ansible
end
end

Loading