A lot of Flux template repositories expect you have already decided what type of Kubernetes distribution you want to use (i.e. the same one they are using). It works when you're following their design but it can be a pain when you even have a slighly different setup. This repository proposes the opposite approach, you know what core applications you want to run on your cluster, but you're not yet sure about the cluster infrastructure.
hetzner-k3s-flux-template is a real-world example I made using this template.
This Flux template was created with a holistic but flexible view in mind. The repository structure is straightforward, easy to understand and easy to use. Still, every aspect is explained below:
All core components are optional, you can choose to only use the repository structure as a starting point.
| Name | Description |
|---|---|
| Flux | Automated Kubernetes cluster updates using GitOps |
| SOPS | Secret management |
| Cilium | eBPF-based Networking, Observability, Security |
| cert-manager | X.509 certificate management |
| ExternalDNS | Synchronize exposed Kubernetes Services and Ingresses with DNS providers |
| Traefik | Edge Router / Ingress |
| Capacitor | General purpose UI for Flux |
| kube-prometheus | Monitoring stack, including Prometheus, Grafana, and Alertmanager |
| Loki | Log aggregation |
| Promtail | Log discovery |
| Kured | Reboot Daemon |
| System Upgrade Controller | Automated node updates |
| Reloader | Auto-reload Kubernetes resources based on ConfigMap/Secret changes |
flowchart TD;
id1[apps] -->|dependsOn| id2[configs]
%% https://github.com/mermaid-js/mermaid/issues/2977
subgraph "` **infrastructure**                          `"
id2[configs] -->|dependsOn| id3[controllers]
%% https://github.com/mermaid-js/mermaid/issues/2977
subgraph "` **controllers**                     `"
id3[controllers] -->|dependsOn| id4[monitoring]
id4[upgrade] -->|dependsOn| id5[monitoring]
id5[monitoring] -->|dependsOn| id6[network]
end
id6[network] -->|dependsOn| id7[image-automations]
id7[image-automations] -->|dependsOn| id8[sources]
id8[sources] -->|dependsOn| id9[notifications]
id9[notifications]
end
A fully set up repository structure is as follows:
apps: user related deployments (e.g. webapps, game servers)cluster: Flux configurationinfrastructure: common infrastructure components (e.g. monitoring, network)templates: yaml template filestools: workspace and template rendering
Flux is set up to only look at apps, cluster and infrastructure:
├── 📁 apps
│ └── kustomization.yaml
│
├── 📁 cluster
│ ├── 📁 flux-system
│ ├── apps.yaml
│ └── infrastructure.yaml
|
└── 📁 infrastructure
├── 📁 configs
├── 📁 controllers
├── 📁 image-automations
├── 📁 notifications
└── 📁 sources
The apps directory is very straightforward, you can structure it however you want as long as you mention the resource in the kustomization.yaml file.
The cluster directory holds all the files necessary for Flux to work. The flux-system directory is generated after bootstrapping, the apps.yaml and infrastructure.yaml hold the Flux Kustomization definitions.
The infrastructure directory is structured into individual files and 5 predetermined sub directories:
configs: Kubernetes custom resources such as cert issuers and networks policiescontrollers: namespaces and Helm release definitions for Kubernetes controllersimage-automations: Image reflector and automation controllersnotifications: Notification Controllerssources: Source Controllers
The configs, image-automations, notifications directories have no resources by default. For brevity, they are omitted from the following view:
📁 infrastructure
├── 📁 controllers
│ ├── 📁 monitoring
| ├── 📁 network
| ├── 📁 upgrade
| └── kustomization.yaml
│
└── 📁 sources
├── 📁 bucket
├── 📁 git
├── 📁 oci
├── 📁 helmrepos
└── kustomization.yaml
The controllers and sources directories have the following sub directories:
monitoring: Monitoring controllersnetwork: Network controllersupgrade: Upgrade controllers
bucket: Bucketsgit: GitRepositoriesoci: OCIRepositorieshelmrepos: HelmRepositories
Getting started is easy, the following guide uses GitHub as a Git server.
- A (preferably clean) Kubernetes cluster that does not have Flux installed, can be any distribution. For local testing you can use something like minikube, kind, microk8s, etc.
- age for secret keys.
- sops for encrypting secrets.
- just for commands.
- Docker for a containerized work environment.
-
Create a new Git repository by using this one as a template.
-
Create an age key that will be used for encrypting cluster based secrets:
age-keygen -o flux.agekeyAdd the age secret key string (identity) AGE-SECRET-KEY-1... found in the file to the default identities file ~/.config/sops/age/keys.txt.
-
Fill
config.yamlwith your age public key (recipient) and your controller values. -
Add and encrypt the kubeconfig file:
You can choose to use a different age key to encrypt the kubeconfig (this key will not be pushed to the cluster). Make sure that the identity also exists in the default identities file (~/.config/sops/age/keys.txt).
sops -e -i --age age... --encrypted-regex '^client-key-data$' --input-type yaml --output-type yaml kubeconfig
config- Build workspace container:
just build- Run workspace container:
just tools- Render templates:
just render- Create a
flux-systemnamespace and create the sops-age secret:
kubectl create ns flux-systemcat flux.agekey | kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin- Bootstrap Flux:
Generate a GitHub PAT that can create repositories by checking all permissions under repo.
export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<your-repo>flux bootstrap github \
--components-extra=image-reflector-controller,image-automation-controller \
--owner=$GITHUB_USER \
--repository=$GITHUB_REPO \
--branch=main \
--path=cluster \
--read-write-key \
--personalIf you wish to create a public repository, add the --private=false flag.
- Done!
You now have a GitOps compliant cluster fully managed through a Git repository, the sky's the limit 🚀!
This looks interesting but complicated, how can I understand this template better?
I've tried to make everything as intuitive as possible. Looking at the individual files and cross-referencing values should help get a rough view of how things work. I also recommend you to carefully look over the Flux documentation whenever you don't understand the contents of a file.
Why not a monorepo approach?
A monorepo is great in theory, dev, staging and prod in one repository with the ability for overlays to minimize duplicated resource declaration. In practice it gets tangled quick, there are a lot of nuances between different environments that can result in more complexity than structure.
Why not use X for Y?
That's for YOU to decide, this is only meant as a foundation for your own cluster/templates. You can choose to not install any core component and roll with your own.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
This "solution" could not have been without the following OSS: