ZFS ZVOL Enhancements and Documentation Updates

BREAKING CHANGES:

• 0a58066 - TCP/IP tweaks moved from defaults/main.yml to sysctl_k3s-network-tweaks.conf
• 111b3b8 - Redesigned Upgrade-Controller Kustomization layout

Improvements:

• 87eaf53 - NEW: Support for ZFS ZVOL Native Encryption
• 826e6e2 - NEW: Enabled ZVOL sync=always by default
• 96d69c1 - Parameterized system-upgrade namespace value
• Many additional fixes per LINTER

README.md / Document Updates:

• 7ba1554 - DOC: Added Uptime Kuma Monitors
• 3c10b64 - DOC: Updated 30 Day Service List linked to Uptime Kuma
• c847f7d - DOC: Updated format to use tables; added ZFS ZVOL Encryption
• 7f1c828 - DOC: Updated link to new cluster dashboard
• f8ff54b - DOC: Updated Home Cluster Hardware section
• d6d203c - DOC: Fixed broken links

Bug Squashes and General Improvements

BREAKING CHANGES:

• N/A

Improvements:

• 9be9b02 - Don't fail on k3s volume mount check in check_mode
• d11266f - Added networking sysctl settings
• 333ddd3 - Redesign Grafana dashboard directory structure
• 986b4e6 - External Cluster Dashboards referenced directly from Project Repo
• 34890e1 - Add SYS_TIME to Capabilities to Kube-VIP
• 8d23e24 - Removed unneeded labels from Kube-VIP
• 4220077 - Added nfs-common packages to Democratic CSI NFS install
• 98e1fa5 - Moved Node ready check delegation to control-plane
• a1cbca0 - Bump K3s default installation version
• 3484d45 - Use nodeSelector over affinity for Kube-VIP
• 31db84c - Added usbutils packages for lsusb command support
• 2956820 - Include volumeSnapshotClass for Democratic CSI iSCSI
• c2a858e - Change Grafana install from Deployment to StatefulSet
• 2e21eb2 - Update deprecated Traefik API CRD names

README.md / Document Updates:

• 9730e20 - Several document typos
• a1bd636 - Included kubeseal namespace flag in examples

Simplified Configuration with K3S Embedded ContainerD

BREAKING CHANGES:

• External cAdvisor as DaemonSet removed
• External ContainerD / RunC removed
• ZFS Snapshotter removed
• Replaced by 30GB (default/adjustable) ZFS ZVOL for K3s (/var/lib/rancher) as XFS file system
  • Still ZFS backed supports mirrors, compression, encryption, etc.
  • Allows K3s to use internal ContainerD, RunC and cAdvisor with overlay filesystem
  • Resolves several kludges, simplifies installation and solves all missing / incorrect metrics

If you want to convert your installation like this, contact me. I have written steps I used to convert my 5 nodes (masters & workers) without downtime or re-installation.

Improvements:

• _extra_apps moved to own repository as these do not require any Ansible
• Configuration File for K3S generated during Prometheus Installation format has been updated per linter
• Democratic CSI PVC example files updated per linter

README.md / Document Updates:

• Documentation updates to support removal of external containerd
• New documentation to support K3S embedded containerd configuration
• Sealed Secrets - show example of extracting private keys needed to perform cluster restore / re-install
• OS Settings - Documented changes applied to sysctl entries

_Extra_Apps Refinements and Tweaks

BREAKING CHANGES: (Coming Next Release)

• External cAdvisor as DaemonSet will be removed.
• External ContainerD / RunC will be removed.
• ZFS Snapshotter will be removed.
• Replaced by 30GB (adjustable) ZFS ZVOL for K3s (/var/lib/rancher) as XFS file system.
  • Still ZFS backed supports mirrors, compression, encryption, etc.
  • Allows K3s to use internal ContainerD, RunC and cAdvisor with overlay filesystem
  • Resolves several kludges, simplifies installation and solves all missing / incorrect metrics

If you want to convert your installation like this, contact me. I have written steps I used to convert my 5 nodes (masters & workers) without downtime or re-installation. My testing based on corrected metrics being reported by internal cAdvisor generated most of the resource corrections listed below.

Improvements:

• Metric relabeling adjustments for cAdvisor.
• Renovate configuration "renovate.json" updated to reduce Renovate upgrades.
  • Renovate can have multiple Patch and Minor upgrades released per day!
  • Each update can be over 1GB in size. Now only Pull-Requests for Major upgrades are created (reduce PR SPAM).

README.md / Document Updates:

• Added overview of my home cluster hardware to introduction.
• Updated Grafana Dashboard "Global View" to use updated cAdvisor metrics per upstream project.
• Grafana.ini adjustments to resolve random session timeouts.

Changes to _extra_apps:

• NEW: Added Syncthing / Helm ArgoCD based deployment.
• NEW: Added Gitea / Kustomize ArgoCD based deployment.
• Added Kustomize yaml-language-server hints for syntax validation:
  • External cAdvisor DaemonSet
  • Custom Alerts
  • Mosquitto
  • Mosquitto Exporter
  • Grafana Dashboards
  • Kube VIP
  • Kube VIP Cloud Provider
  • Unpoller Exporter
  • Upgrade Controller
• Reduced cAdvisor polling intervals to reduce CPU overhead.
• Rook-Ceph Cluster Storage changes:
  • Reduced Log Collector resources
  • Reduced Crash Collector resources
  • Bumped Manger resources
  • Bumped Manager SideCar (watch-active) resources
  • Bumped Monitor resources
  • Bumped OSD resources
  • Bumped Ceph File System resources
• K3s Update Controller - added comment reference to K3S stable channel
• Bumped Apt-Cacher NG suggested resources
• Bumped Unpoller Exporter suggested resources
• Bumped Trilium Notes suggested resources
• Set "revisionHistoryLimit: 3" to reduce deployment/statefulset history clutter on:
  • Kube VIP
  • Kube VIP Cloud Provider
  • Mosquitto Exporter
  • Unpoller Exporter

External cAdvisor DaemonSet / Updated ZFS Exporter / Updated Dashboards

BREAKING CHANGES:

• Workaround for Kublet cAdvisor issues with containerd removed.
• Older established nodes should have kubelet cAdvisor disabled after external cAdvisor DaemonSet deployed.
• Dashboards should be updated to use standard labels vs previous workaround labels.

Improvements:

• NEW: External cAdvisor deployed as DaemonSet to support containerd with ZFS Snapshotter.
• NEW: Kublet internal cAdvisor disabled, does not scrape containerd correctly.
• Updated many Dashboards to support standard labels used by External cAdvisor metrics.
• Added support for worker nodes (non control-plane)
• Moved ZFS Exporter Configuration from Secret to Helm Values section.
• FIX: ZFS Exporter now drops metrics unlikely to be needed,
• FIX: ZFS Exporter Ignore containerd datasets no longer assumes just rpool ZFS pool.
• FIX: Disable ectd Service Monitors when only one node.

README.md / Document Updates:

• Fixed worker node references.
• Added doc on checking / resolving missing wildcard cert on new install.
• Updated example Cluster Dashboard images to reflect recent updates.

Changes to _extra_apps:

• NEW: Added Apt-Cacher NG ArgoCD / Helm deployment.
• Removed Unifi Controller Kustomized Version.
• Updated to Home Assistant README.md description.

Worker Nodes Tested / Updated per YAML Lint

BREAKING CHANGES:

• N/A

Improvements:

• Added support for worker nodes (non control-plane)
• Added validation check that ZFS containerd pool already exists
• All pages updated per YAML linter recommendations
• Tested with Kubernetes 1.25
• Tested with Ubuntu 22.04
• Updated base install versions of all referenced applications

README.md / Document Updates:

• Updated docs to reference updated base version numbers

Changes to _extra_apps:

• N/A

Two New Apps Added to "Extra Apps" List

BREAKING CHANGES:

• N/A

Improvements:

• N/A

README.md / Document Updates:

• N/A

Changes to _extra_apps:

• NEW: Trilium Notes added
  • Hierarchical note taking application with focus on building large personal knowledge bases
• NEW: Rook-Ceph Cluster Storage added
  • Rook operator to automate Ceph storage management
  • This would be a replacement for Longhorn storage (no reason to use both)
  • Resources optimized for small 3 node K3s cluster with each node having dedicated storage device or partition
  • Not recommended for smaller clusters, 5 node or larger would be ideal
• CHG: Custom-Alerts for Traefik / Prometheus Alerts
  • Commented out alerts for HTTP code 401
  • This code is expected during authentication attempts, login pages generated too many false alerts
• CHG: Added Trilium Notes and Rook-Ceph to Application List README.md
• FIX: multiple typos in Home Assistant with an ArgoCD README.md
• FIX: corrected links in Application List README.md for Mosquitto MQTT Broker

Stability and Performance Improvements

BREAKING CHANGES:

• ArgoCD Helm Package 5.5 moved "extraArgs" to "configs.params" section

Improvements:

• NEW: Initial Release of "cri-prurge.sh" v0.03 script to intelligently clean up unused container images
• Added flags to Ansible k3s_exec_options:
  • Added file system garbage collection flags for kubelet (clean up disk space sooner)
  • Added flag to K3S that ZFS is the snapshotter
• FIX: Prometheus Operator disable podSecurityPolicy
  • podSecurityPolicy is deprecated and removed in Kubernetes 1.25
• NEW: Enabled ArgoCD Container Terminal Access by Default

README.md / Document Updates:

• Added reference to K3S Upgrade Controller Version Skew Policy

Changes to _extra_apps:

• Updated Home Assistant with an ArgoCD / Helm implementation
  • Added comments on how to migrate data to new installation
• NEW: Added Pod Restart Info Collector
  • Provides detailed information about why a pod has restarted
• FIX: Changed Zigbee2MQTT PVC Storage class to iSCSI by default
  • Previously used Longhorn storage is not appropriate for storage solutions that are write heavy

Improved ArgoCD organization with Projects, Easier Prometheus resource finding, and Home Assistant

BREAKING CHANGES:

• N/A

Improvements:

• FIX: Moved ArgoCD Projects application from a sub-directory under Applications to its own application named Projects
  • Projects allow for grouping applications, added security, restrict source repos and namespaces
  • ArgoCD WebUI allows apps displayed by filtering by Projects
• FIX: Removed Ansible Notify reference on include_task to restart K3s service (not supported)
• NEW: Kube Prometheus Stack no longer requires a release label to find resources in other namespaces
  • Kube Prometheus Stack will now default to finding resources across namespaces, no labels needed.

README.md / Document Updates:

• Updated ArgoCD docs to show separated Projects application
• NEW: Added MQTT Zero Publishing Rate alert example
• NEW: Added Home Assistant configuration docs
  • Added hints on data migration and HACS configuration

Changes to _extra_apps:

• The k8s-at-home repository has been archived switched to read-only mode and will no longer provide updates
  • Switched to hosting self-copy of manifests for future updates/changes
  • All references to this repository changed to self-copy of it
• Removed Prometheus release labels not needed from resources and cleaned up Kustomizations to not try to apply them
• FIX: Custom-Alerts - MQTT Zero Publishing Rate alert to use rate expression (0 messages for 1 min)
• FIX: zigbee2mqtt ArgoCD / Helm version, added startup probe to allow time for container to start responding
  • This resolved unnecessary repetitive container restarts
• NEW: Added Home Assistant with an ArgoCD / Helm implementation
  • Statefulset, PVC storage, Traefik Ingress, CodeServer sidecar with Traefik Authentication for YAML file editing & terminal access

Incremental Improvements

BREAKING CHANGES:

• N/A

Improvements:

• FIX: Added K3S Restart after Prometheus Deployment
• FIX: Traefik defaults to more secure TLS 1.2 (1.0/1.1 disabled)
• FIX: Added default blacklist of multipath names with Longhorn and Democratic-CSI iSCSI storage
• FIX: Enabled Renovate Dependency Dashboard by default

README.md / Document Updates:

• Respective updates to _extras_apps documentation

Changes to _extra_apps:

• ADD: ArgoCD Helm Chart version of Mosquitto Message Broker
• ADD: ArgoCD Helm Chart version of Zigbee2MQTT Bridge
• ADD: ArgoCD Helm Chart version of Unifi Network Controller
• NEW: Redesigned Mosquitto MQTT Dashboard
• FIX: Added Renovate Update Hint for RegExManager
• FIX: Replaced some node-exporter references with mosquitto-exporter
• NEW: Custom-Alerts - Added MQTT Zero Publishing Rate alert
• NEW: Custom-Alerts - Added POD Restarted Alert
• FIX: Custom-Alerts - Reformatted ZFS Pool state change alert
• FIX: Corrected line wraps in multiple alerts