hetzner-k3s-flux-template - An opionated template for a K3s cluster implementing GitOps through Flux on Hetzner Cloud

Flux is an amazing tool for managing cluster resources, Hetzner Cloud is a great platform for creating cost effective hobby or production Kuberenetes clusters. This repository attempts to create a solid starting point for creating a fully fledged Flux managed K3s cluster on Hetzner Cloud.

Table of contents

• Overview
• Quickstart
• Considerations
• Contributing
• License

Overview

This template is built on my general flux-template. It is generally the same except for some minor changes in the config.yaml file. The values key for a controller is replaced with a values_override key when we offer an optimized default configuration. If you wish to not use our default options just fill in values_override with whatever configuration you want and the default options will be ignored.

Stack

All core components are optional, you can choose to only use the repository structure as a starting point.

Name	Description	Optimized defaults
Flux	Automated Kubernetes cluster updates using GitOps	N/A
SOPS	Secret management	N/A
Longhorn	Cloud native distributed block storage for Kubernetes	✅
hcloud-cloud-controller-manager	Kubernetes Cloud Controller Manager for Hetzner Cloud	✅
Cilium	eBPF-based Networking, Observability, Security	✅
cert-manager	X.509 certificate management	✅
ExternalDNS	Synchronize exposed Kubernetes Services and Ingresses with DNS providers	❌
Traefik	Edge Router / Ingress	✅
Capacitor	General purpose UI for Flux	❌
kube-prometheus	Monitoring stack, including Prometheus, Grafana, and Alertmanager	❌
Loki	Log aggregation	❌
Promtail	Log discovery	❌
Kured	Reboot Daemon	❌
System Upgrade Controller	Automated node updates	✅
Reloader	Auto-reload Kubernetes resources based on ConfigMap/Secret changes	❌

Flux Kustomization Reconciliation Flowchart

flowchart TD;
    id1[apps] -->|dependsOn| id2[configs]
    %% https://github.com/mermaid-js/mermaid/issues/2977
    subgraph "`**infrastructure**`"
        id2[configs] -->|dependsOn| id3[controllers]
        %% https://github.com/mermaid-js/mermaid/issues/2977
        subgraph "`**controllers**`"
            id3[controllers] -->|dependsOn| id4[monitoring]
            id4[upgrade] -->|dependsOn| id5[monitoring]
            id5[monitoring] -->|dependsOn| id6[network]
        end
        id6[network] -->|dependsOn| id7[image-automations]
        id7[image-automations] -->|dependsOn| id8[sources]
        id8[sources] -->|dependsOn| id9[notifications]
        id9[notifications]
    end

Loading

A fully set up repository structure is as follows:

Top directories

• apps: user related deployments (e.g. webapps, game servers)
• cluster: Flux configuration
• infrastructure: common infrastructure components (e.g. monitoring, network)
• templates: yaml template files
• tools: workspace and template rendering

Flux is set up to only look at apps, cluster and infrastructure:

├── 📁 apps
│   └── kustomization.yaml
│  
├── 📁 cluster
│   ├── 📁 flux-system
│   ├── apps.yaml
│   └── infrastructure.yaml
|
└── 📁 infrastructure
    ├── 📁 configs
    ├── 📁 controllers
    ├── 📁 image-automations
    ├── 📁 notifications
    └── 📁 sources

Applications

The apps directory is very straightforward, you can structure it however you want as long as you mention the resource in the kustomization.yaml file.

Cluster

The cluster directory holds all the files necessary for Flux to work. The flux-system directory is generated after bootstrapping, the apps.yaml and infrastructure.yaml hold the Flux Kustomization definitions.

Infrastructure

The infrastructure directory is structured into individual files and 5 predetermined sub directories:

• configs: Kubernetes custom resources such as cert issuers and networks policies
• controllers: namespaces and Helm release definitions for Kubernetes controllers
• image-automations: Image reflector and automation controllers
• notifications: Notification Controllers
• sources: Source Controllers

The configs, image-automations, notifications directories have no resources by default. For brevity, they are omitted from the following view:

📁 infrastructure
├── 📁 controllers
│   ├── 📁 monitoring
|   ├── 📁 network
|   ├── 📁 upgrade
|   └── kustomization.yaml
│  
└── 📁 sources
    ├── 📁 bucket
    ├── 📁 git
    ├── 📁 oci
    ├── 📁 helmrepos
    └── kustomization.yaml

The controllers and sources directories have the following sub directories:

controllers

• monitoring: Monitoring controllers
• network: Network controllers
• upgrade: Upgrade controllers

sources

• bucket: Buckets
• git: GitRepositories
• oci: OCIRepositories
• helmrepos: HelmRepositories

Quickstart

Getting started is easy, the following guide uses GitHub as a Git server.

Prerequisites

• A (preferably clean) K3s cluster on Hetzner Cloud that does not have Flux installed.
• age for secret keys.
• sops for encrypting secrets.
• just for commands.
• Docker for a containerized work environment.

1. Create a new Git repository by using this one as a template.

2. Create an age key that will be used for encrypting cluster based secrets:

age-keygen -o flux.agekey

Add the age secret key string (identity) AGE-SECRET-KEY-1... found in the file to the default identities file ~/.config/sops/age/keys.txt.

3. Fill config.yaml with your age public key (recipient) and your controller values.

4. Add and encrypt the kubeconfig file:

You can choose to use a different age key to encrypt the kubeconfig (this key will not be pushed to the cluster). Make sure that the identity also exists in the default identities file (~/.config/sops/age/keys.txt).

sops -e -i --age age... --encrypted-regex '^client-key-data$' --input-type yaml --output-type yaml kubeconfig
config

5. Build workspace container:

just build

6. Run workspace container:

just tools

7. Render templates:

just render

8. Create a flux-system namespace and create the sops-age secret:

Create flux-system namespace:

kubectl create ns flux-system

Create sops-age secret:

cat flux.agekey | kubectl create secret generic sops-age \
   --namespace=flux-system \
   --from-file=age.agekey=/dev/stdin

9. Bootstrap Flux:

Generate a GitHub PAT that can create repositories by checking all permissions under repo.

Export values:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<your-repo>

Bootstrap:

flux bootstrap github \
 --components-extra=image-reflector-controller,image-automation-controller \
 --owner=$GITHUB_USER \
 --repository=$GITHUB_REPO \
 --branch=main \
 --path=cluster \
 --read-write-key \
 --personal

If you wish to create a public repository, add the --private=false flag.

10. Done!

You now have a GitOps compliant K3s cluster hosted on Hetzner Cloud that is fully managed through a GitHub repository, the sky's the limit 🚀!

Considerations

This looks interesting but complicated, how can I understand this template better?

I've tried to make everything as intuitive as possible. Looking at the individual files and cross-referencing values should help get a rough view of how things work. I also recommend you to carefully look over the Flux documentation whenever you don't understand the contents of a file.

Why use Hetzner Cloud?

Hetzner Cloud has a good price to performance ratio. On top of that they also offer outstanding service and support.

Why use K3s?

K3s is perfect for Hetzner Cloud as it has a minimal footprint. Meaning you can create a capable HA (High Availability) cluster for cheap.

Why not a monorepo approach?

A monorepo is great in theory, dev, staging and prod in one repository with the ability for overlays to minimize duplicated resource declaration. In practice it gets tangled quick, there are a lot of nuances between different environments that can result in more complexity than structure.

Why not use X for Y?

That's for YOU to decide, this is only meant as a foundation for your own cluster/templates. You can choose to not install any core component and roll with your own.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

Acknowledgements

This "solution" could not have been without the following OSS:

• kronform
• home-ops