v1.0.0: Official release of ClusterFactory

It's been exactly a year since the last version of ClusterFactory was released. We've worked closely with DeepSquare and other companies to correct the imperfections. DeepSquare has now acquired ClusterFactory for closer collaborative development.

The product in itself is still "rough" but does not have any critical issues that block the deployment of the services.

Here's the main changes since v0.8.0:

• Compute Node OS images are now stable, named SquareOS, running on Rockylinux 9.2 and can be build using the packer-recipes directory.
• xCat has been replaced with Grendel, a lighter provisioner and GitOps-friendly.
• OpenLDAP has been replaced with 389ds, a production-ready LDAP.
• Removed Helm extensions from cfctl.yaml.
• Added DeepSquare solutions in the ArgoCD examples (ldap-connector and supervisor).
• Added Harbor, an container image registry, in the ArgoCD examples.
• Added MariaDB in the ArgoCD examples.
• Added Rook Ceph Cluster, a solution for Object Storage, in the ArgoCD examples.
• Added Keycloak, an Identity Provider with LDAP connector, in the ArgoCD examples.
• Added Knative, a solution for serverless applications, in the ArgoCD examples.
• Added "dynamic-nfs" in the ArgoCD examples.
• Added terraform examples (which are still shaky...).
• Added smoke-tests.
• Migrated from Squarefactory to deepsquare-io.
• Many documentation updates.
• All dependencies have been updated:
  • K0s from 1.25.3+k0s.0 to 1.28.3+k0s.0
  • cfctl from v0.13.2+9001 to v0.16.0+9001, this includes a non-breaking change in the management of secrets, which allows an "offline" encryption of secrets, permitting a 100% zero-trust infrastructure.
  • Traefik from v20.2.0 to v25.0.0, this includes a breaking change in the values.yaml and the migration from traefik.container.us to traefik.io.
  • ArgoCD from v5.13.8 to v5.51.1
  • SealedSecrets from v2.7.0 to v2.13.2 and Kubeseal from 0.19.1 to 0.24.3
  • KubeVirt from v0.59.0-alpha.0 to v1.1.0
  • CoreDNS from v1.10.0 to v1.11.1
  • Helm from v3.10.2 to v3.13.2
  • etcdctl from v3.5.5 to v3.5.10
  • Local path provisioner from v0.0.23 to v0.0.24
  • Kube Prometheus Stack from v41.7.4 to v52.1.0
  • cert-manager from v1.11.0-alpha.0 to v1.13.2
  • csi-driver-nfs from v4.1.0 to v4.5.0

Full Changelog: v0.8.0...v1.0.0

v0.9.0-beta.2

Many deprecations

Changelog

• Updated dependencies
  • ArgoCD: 5.29.1
  • sealed-secrets: 2.8.2
  • KubeVirt: v0.60.0-alpha.0
  • CoreDNS: 1.10.1
  • cfctl: v0.15.0+9003
  • kubeseal: 0.20.5
  • Helm: v3.11.3
  • etcdctl: v3.5.8
  • local path provisioner: v0.0.24
  • Kube Prometheus Stack: 45.10.1
  • K0s: 1.26.3+k0s.0
  • MetalLB: 4.3.0
  • Traefik: 22.1.0
  • Cert-manager: v1.12.0-alpha.2
  • CSI NFS Driver: v4.2.0
• Various documentation fixes
• Added harbor, a selfhosted container images registry
• Added 389ds, a replacement for OpenLDAP
• Added Keycloak, a identity and access management solution
• Various fixes with the Packer recipes. Deleted rocky9 as it is unstable. Offering the SquareOS recipe.
• Various Terraform fixes.

Deprecations

• OpenLDAP deprecated for 389ds
• xCAT deprected for Grendel

Full Changelog: https://github.com/SquareFactory/ClusterFactory/compare/v0.9.0-beta.1..v0.9.0-beta.2

v0.9.0-beta.1

Breaking change involved at core level, please read the migration guide!

Changelog

• Updated dependencies
  • ArgoCD: 5.16.1
  • sealed-secrets: 2.7.1
  • KubeVirt: v0.59.0-alpha.1
  • cfctl: v0.15.0-rc.1+9001
  • kubeseal: 0.19.2
  • etcdctl: v3.5.6
  • Kube Prometheus Stack: 42.2.0
  • K0s: 1.25.4+k0s.0
  • MetalLB: 4.1.12
  • Traefik: 20.6.0
• BREAKING: Core: Removed extensions from cfctl.yaml
• Core: fixed setup-env script for darwin and arm64
• Tests: Added smoke tests
• Terraform: Implemented Terraform for OVH and Exoscale
• Slurm: Cloud-burster updated for OVH and Exoscale
• Others: Removed "CE" from the project name

Breaking changes

Removed extensions from cfctl.yaml

We decided to not use the extensions field from cfctl.yaml because of instabilities. The migration is the following:

1. Migrate the values of your extensions to the core directory when merging upstream/main.

cert-manager: no values file
csi-driver-nfs: core/csi-driver-nfs/values.yaml
metallb: no values file
traefik: core/traefik/values.yaml

2. Remove the finalizer helm.k0sproject.io/uninstall-helm-release from the Chart, this avoid the delete of the Helm release.

kubectl patch Chart k0s-addon-chart-cert-manager -n kube-system  \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
kubectl patch Chart k0s-addon-chart-csi-driver-nfs -n kube-system  \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
kubectl patch Chart k0s-addon-chart-metallb -n kube-system  \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
kubectl patch Chart k0s-addon-chart-traefik -n kube-system  \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

3. Delete the Chart from kubernetes

kubectl delete Chart k0s-addon-chart-cert-manager -n kube-system 
kubectl delete Chart k0s-addon-chart-csi-driver-nfs -n kube-system
kubectl delete Chart k0s-addon-chart-metallb -n kube-system
kubectl delete Chart k0s-addon-chart-traefik -n kube-system

4. Delete the addons in the /var/lib/k0s/manifests/helm/ directory on the controller nodes.

rm /var/lib/k0s/manifests/helm/addon*

5. Redeploy everything using the deploy-core script or by using the install.sh script in the app directories inside core.

You can also let the extensions field untouched, but we won't support this feature anymore.

Full Changelog: v0.8.0...v0.9.0-beta.1

v0.8.0: Cloud-burster

Drain the nodes before applying the deploy core script

---

Changelog

• Updated dependencies
  • ArgoCD: 5.13.8
  • Multus: 4.0.0-unstable (not yet tracked by version tracker)
  • K0s: 1.25.3+k0s.0
  • MetalLB: 4.1.11 (#96)
  • Kube Prometheus Stack: 41.7.4
  • csi-driver-nfs: v4.1.0 (downgrade)
  • Sealed Secrets: 2.7.0
  • KubeVirt: v0.59.0-alpha.0
  • CoreDNS: 1.10.0
  • cfctl: v0.13.2+9001
  • Helm: v3.10.2
  • kubeseal: 0.19.1
  • etcdctl: v3.5.5
  • Local Path Provisioner: v0.0.23
  • Traefik: 20.2.0
  • cert-manager: v1.11.0-alpha.0
• Slurm: Added Cloud-Burster to slurm (3d2f092)
• Slurm: Slurm now supports metrics (3dcc4ab)
• Core: "Deploy core" script deploys CoreDNS
• Core: "Deploy core" script only waits for specific deployments instead of all
• Core: Uncoupled CoreDNS from initial K0s deployment
• Core: CoreDNS as a DaemonSet
• Core: Enable HTTP/3 on Traefik by default
• Helm apps: Added 398ds to Helm directory
• Helm apps: Various fixes on Squid Proxy
• Helm apps: Various fixes on CVMFS Service
• Helm apps: Various fixes on OpenLDAP
• Helm apps: Supports OpenOnDemand with Dex image and without.
• Packer: New DeepSquare Yum repository path
• Packer: Initial support for Rockylinux 9, support for Rockylinux 8.6
• Documentation updates on cfctl and cfctl.yaml

Breaking changes

The new major version of Multus CNI introduces heavy changes (k8snetworkplumbingwg/multus-cni#893)

The migration is seamless; but needs attention.

Multus CNI 4.0 thick daemonset introduces a new architecture which is the server/client architecture. As there is a server that handles all the network attachments, the process is quite slow.

As soon as you apply the Multus CNI, each pod will be killed to reattach the networks. Be aware that the process is REALLY slow. You might see some Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "<id>": plugin type="multus-shim" name="multus-cni-network" failed (add): CNI request failed with status 400: <...>. This means that the multus-shim is working hard.

To accelerate the process and to be safe, drain the nodes.

Rebooting a node won't change anything and may break your setup.

The new major version of ArgoCD introduces breaking changes (https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#520)

The ArgoCD helm chart now handles CRDs

• To adopt the new CRD management style, runs this script:

  YOUR_ARGOCD_NAMESPACE="argocd"
  YOUR_ARGOCD_RELEASENAME="argocd"
  
  for crd in "applications.argoproj.io" "applicationsets.argoproj.io" "argocdextensions.argoproj.io" "appprojects.argoproj.io"; do
    kubectl label --overwrite crd $crd app.kubernetes.io/managed-by=Helm
    kubectl annotate --overwrite crd $crd meta.helm.sh/release-namespace="$YOUR_ARGOCD_NAMESPACE"
    kubectl annotate --overwrite crd $crd meta.helm.sh/release-name="$YOUR_ARGOCD_RELEASENAME"
  done

• To NOT adopt the new CRD management style, add in the values file:

  crds:
    install: false

  You will have to update the CRDs using: kubectl apply -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=<appVersion>"

Deprecated configs.repositoryCredentials, server.additionalApplications and server.additionalProjects.

See: https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#500

The new major version of MetaLB introduces breaking changes (#96)

MetalLB 0.13.0 is now available, with its chart bitnami/metallb 4.0.0.

configInline is now deprecated and CRDs are now preferred.

This major release includes the changes and features available in MetalLB from version 0.13.0. Those changes include the deprecation of configmaps for configuring the service and using CRDs instead. If you are upgrading from a previous version, you can follow the official documentation on how to migrate the configuration from a configMap to CRDs.

Migration instructions:

1. Fetch the MetalLB configMap and store it inside a file config.yaml
2. Run the conversion utility using docker:

docker run -it --rm -v $(pwd):/var/input quay.io/metallb/configmaptocrs -source config.yaml

3. Remove configInline from the Helm values inside cfctl.yaml and redeploy with cfctl. If there are issues with updating a Helm extensions, see the documentation

4. Apply the CRDs

kubectl apply -f .

Full Changelog: v0.7.0...v0.8.0

v0.8.0-beta.3

Drain the nodes before applying the deploy core script

---

Changelog

• Updated dependencies
  • Multus: 4.0.0-unstable (not yet tracked by version tracker)
  • ArgoCD: 5.3.6
• "Deploy core" script deploys CoreDNS
• "Deploy core" script only waits for specific deployments instead of all

Breaking changes

The new major version of Multus CNI introduces heavy changes (k8snetworkplumbingwg/multus-cni#893)

The migration is seamless; but needs attention.

Multus CNI 4.0 thick daemonset introduces a new architecture which is the server/client architecture. As there is a server that handles all the network attachments, the process is quite slow.

As soon as you apply the Multus CNI, each pod will be killed to reattach the networks. Be aware that the process is REALLY slow. You might see some Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "<id>": plugin type="multus-shim" name="multus-cni-network" failed (add): CNI request failed with status 400: <...>. This means that the multus-shim is working hard.

To accelerate the process and to be safe, drain the nodes.

Rebooting a node won't change anything and may break your setup.

Full Changelog: v0.8.0-beta.2...v0.8.0-beta.3

v0.8.0-beta.2

Changelog

• Updated dependencies
  • ArgoCD: 5.3.4
  • MetalLB: 4.1.1
  • Kube Prometheus Stack: 39.9.0
  • K0s: 1.24.4+k0s.0
  • Sealed Secrets: 2.6.1
  • KubeVirt: v0.56.0-rc.1
  • cfctl: v0.13.2+9001
  • kubeseal: 0.18.2
  • Helm: v3.9.4
• CoreDNS as a DaemonSet
• Supports OpenOnDemand with Dex image and without.
• New DeepSquare Yum repository path
• Initial supports for Rockylinux 9

Breaking changes

The new major version of ArgoCD introduces breaking changes (https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#520)

The ArgoCD helm chart now handles CRDs

• To adopt the new CRD management style, runs this script:

  YOUR_ARGOCD_NAMESPACE="argocd"
  YOUR_ARGOCD_RELEASENAME="argocd"
  
  for crd in "applications.argoproj.io" "applicationsets.argoproj.io" "argocdextensions.argoproj.io" "appprojects.argoproj.io"; do
    kubectl label --overwrite crd $crd app.kubernetes.io/managed-by=Helm
    kubectl annotate --overwrite crd $crd meta.helm.sh/release-namespace="$YOUR_ARGOCD_NAMESPACE"
    kubectl annotate --overwrite crd $crd meta.helm.sh/release-name="$YOUR_ARGOCD_RELEASENAME"
  done

• To NOT adopt the new CRD management style, add in the values file:

  crds:
    install: false

  You will have to update the CRDs using: kubectl apply -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=<appVersion>"

Deprecated configs.repositoryCredentials, server.additionalApplications and server.additionalProjects.

See: https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#500

Full Changelog: v0.8.0-beta.1...v0.8.0-beta.2

v0.8.0-beta.1

Changelog

• Updated dependencies
  • MetalLB: 4.0.2 (#96)
  • Kube Prometheus Stack: 39.5.0
  • K0s: 1.24.3+k0s.0
  • csi-driver-nfs: v4.1.0 (downgrade)
  • ArgoCD: 4.10.5
  • Sealed Secrets: 2.6.0
  • KubeVirt: v0.56.0-rc.0
  • cfctl: v0.13.1+9001
  • kubeseal: 0.18.1
  • Helm: v3.9.2
  • cert-manager: v1.9.1
• Uncoupled CoreDNS from initial K0s deployment
• Added 398ds to Helm directory
• Enable HTTP/3 on Traefik by default
• Slurm now supports metrics (3dcc4ab)
• Various fixes on Squid Proxy
• Various fixes on CVMFS Service
• Various fixes on OpenLDAP
• Documentation updates on cfctl and cfctl.yaml

Breaking changes

The new major version of MetaLB introduces breaking changes (#96)

MetalLB 0.13.0 is now available, with its chart bitnami/metallb 4.0.0.

configInline is now deprecated and CRDs are now preferred.

This major release includes the changes and features available in MetalLB from version 0.13.0. Those changes include the deprecation of configmaps for configuring the service and using CRDs instead. If you are upgrading from a previous version, you can follow the official documentation on how to migrate the configuration from a configMap to CRDs.

Migration instructions:

1. Fetch the MetalLB configMap and store it inside a file config.yaml
2. Run the conversion utility using docker:

docker run -it --rm -v $(pwd):/var/input quay.io/metallb/configmaptocrs -source config.yaml

3. Remove configInline from the Helm values inside cfctl.yaml and redeploy with cfctl. If there are issues with updating a Helm extensions, see the documentation

4. Apply the CRDs

kubectl apply -f .

Full Changelog: v0.7.0...v0.8.0-beta.1

v0.7.0

Changelog

• Updated dependencies
  • Argo CD 4.9.11
  • Sealed Secrets 2.3.0
  • Kube Prometheus Stack 36.2.1
  • MetalLB 3.0.10
  • Traefik 10.24.0
  • cert-manager 1.9.0-beta.0
  • CSI NFS driver v4.2.0
  • cfctl 0.13.0+9004
• Use Helm subcharts to avoid hard-coding inside Argo CD applications
• Scripts has been moved, renamed or included direcly inside cfctl
• Various Packer fixes
• Documentation: added pages about the cli (#41)
• Documentation: added pages about setting a repository for GitOps
• Documentation: added Algolia

Breaking Changes

cfctl replaces k0sctl (#35)

To migrate:

• Delete and never use k0sctl inside the ./bin/ directory. To be safe delete the bin directory inside the cluster factory directory.
• Call . ./scripts/common.sh to install the dependencies.
• Rename k0sctl.yaml to cfctl.yaml.
• Inside the cfctl.yaml, change k0sctl.k0sproject.io/v1beta1 to cfctl.clusterfactory.io/v1beta1.
• Inside the cfctl.yaml, delete the part about Install CNI plugins and Fix Kubelet directory.
• Use cfctl for deployment. The documentation is written here: cfctl.

Helm values are no more hard-coded inside Argo CD applications (#57)

To migrate:

1. Move the values from the Argo CD application to the helm/ or helm-subcharts/ directories by create a values file (like values-production.yaml.
2. Change the repoURL to your fork. Example:

source:
  # You should have forked this repo. Change the URL to your fork.
  repoURL: [email protected]:<your account>/cluster-factory-ce-configs.git
  targetRevision: HEAD
  path: helm-subcharts/kube-prometheus-stack
  helm:
    releaseName: prometheus

    skipCrds: true

    # If the values file is not `values.yaml`:
    valueFiles:
      - values-production.yaml

3. Commit and push the values-production.yaml file. In the example, it would be helm-subcharts/kube-prometheus-stack/values-production.yaml.

4. Delete the application and recreate it.

If there are no subchart, create a subchart inside the helm-subcharts/ directory:

apiVersion: v2
name: kube-prometheus-stack-subchart
description: Kube Prometheus Stack subchart
type: application
version: 36.2.1
appVersion: '36.2.1'

dependencies:
  - name: kube-prometheus-stack
    version: 36.2.1
    repository: https://prometheus-community.github.io/helm-charts

Full Changelog: v0.6.1...v0.7.0

v0.7.0-beta.3

Changelog

• Updated dependencies
  • Argo CD 4.9.11
  • Sealed Secrets 2.3.0
  • Kube Prometheus Stack 36.2.1
  • MetalLB 3.0.10
  • Traefik 10.24.0
  • cert-manager 1.9.0-beta.0
  • CSI NFS driver v4.2.0
• Use Helm subcharts to avoid hard-coding inside Argo CD applications
• Various documentation modifications
• Various Packer fixes

Breaking Changes

ClusterFactory is now using the helm subcharts to avoid hard-coding values inside Argo CD applications.

To migrate:

1. Move the values from the Argo CD application to the helm/ or helm-subcharts/ directories by create a values file (like values-production.yaml.
2. Change the repoURL to your fork. Example:

source:
  # You should have forked this repo. Change the URL to your fork.
  repoURL: [email protected]:<your account>/cluster-factory-ce-configs.git
  targetRevision: HEAD
  path: helm-subcharts/kube-prometheus-stack
  helm:
    releaseName: prometheus

    skipCrds: true

    # If the values file is not `values.yaml`:
    valueFiles:
      - values-production.yaml

3. Commit and push the values-production.yaml file. In the example, it would be helm-subcharts/kube-prometheus-stack/values-production.yaml.

4. Delete the application and recreate it.

If there are no subchart, create a subchart inside the helm-subcharts/ directory:

apiVersion: v2
name: kube-prometheus-stack-subchart
description: Kube Prometheus Stack subchart
type: application
version: 36.2.1
appVersion: '36.2.1'

dependencies:
  - name: kube-prometheus-stack
    version: 36.2.1
    repository: https://prometheus-community.github.io/helm-charts

Full Changelog: v0.7.0-beta.2...v0.7.0-beta.3

v0.7.0-beta.2

Changelog

• Updated dependencies
  • Argo CD 4.9.8
  • cfctl 0.13.0+9002
  • Traefik 10.23.0
  • cert-manager v1.9.0-alpha.0
• cfctl.yaml: fixed setting up SELinux
• Documentation: added pages about setting a repository for GitOps

Full Changelog: v0.7.0-beta.1...v0.7.0-beta.2