Refactor to explicit imports and rename holos Projects to Stacks #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix the following legitamate error with evalv3:
Component: field not allowed:
./schemas/kargo/project_types.cue:80:70
This patch begins refactoring the domain model into packages. The goal is to establish a platform package with types in pkg/types/platform and concrete configuration values in pkg/config/platform. The platform spec is in the main package, also named platform in ./platform Components will import the main platform package from example.com/holos/platform to get a handle on aspects of the Platform spec. The main platform package imports the platform configuration from example.com/holos/pkg/config/platform, iterating over platform.Stacks to assemble the Platform.spec.components field. It may be worth defining a #PlatformBuilder which takes in parameters and assembles a Platform resource, but we'll see how this comes together. It may also be worth separating out the #FooBuilder definitions from the types, the way things are assembled is more specific to the organization than the definitions of the things being assembled. Result: At the end of this change list, the deploy directory should be able to be removed and re-created as it was prior with `holos render platform`. This is intended to be a true refactor. Once complete, we'll add additional functionality. If additional functionality is needed while refactoring, record it as a follow up task. The change list will work through components in the order listed in scripts/apply
The argocd crds are the first component to manage. Establish the argocd Stack to manage the component. We'll need to manage the argocd namespace. The security stack contains the namespaces component. The current behavior is for this component to manage all namespaces for all components in a single BuildPlan. We iterate over all stacks and manage the namespaces for each stack from the namespaces field. The Namespace kubernetes resource is largely passed as-is all the way from the type definition. This allows composition at each step along the way. Follow up work. This patch relocates the namespaces component output directory to the desired location of stacks/security/components/namespaces. We'll remove this in a follow up patch to preserve the integrity of the refactor, currently components/namespaces.
We want components built with the stack builder to consistently write output into a stack scoped path. Revert this patch after the refactor completes to get this behavior.
Refactor the argocd-crds component. Note evalv3 fails to produce the
same BuildPlan as evalv2. Create a test case for this as a follow up.
Added a test case to catch the failure:
go test -timeout 30s -run ^TestUnity/evalv3
Result:
The platform spec:
holos show platform
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
components:
- annotations:
description: argocd custom resource definitions
labels:
holos.run/stack.name: argocd
name: argocd-crds
path: stacks/argocd/components/argocd-crds
- annotations:
description: configures namespaces for all stacks
labels:
holos.run/stack.name: security
name: namespaces
path: stacks/security/components/namespaces
The platform config package:
CUE_EXPERIMENT=embed cue export --out yaml ./pkg/config/platform
stacks:
argocd:
metadata:
name: argocd
components:
stacks:argocd:components:argocd-crds:
name: argocd-crds
path: stacks/argocd/components/argocd-crds
annotations:
description: argocd custom resource definitions
labels:
holos.run/stack.name: argocd
namespaces:
argocd:
metadata:
name: argocd
labels:
kubernetes.io/metadata.name: argocd
kind: Namespace
apiVersion: v1
security:
metadata:
name: security
components:
stacks:security:components:namespaces:
name: namespaces
path: stacks/security/components/namespaces
annotations:
description: configures namespaces for all stacks
labels:
holos.run/stack.name: security
This patch composes the argocd Application resources into the
deploy/gitops/ directory.
holos show platform
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
components:
- annotations:
description: argocd custom resource definitions
labels:
holos.run/stack.name: argocd
name: argocd-crds
parameters:
stack: argocd
path: stacks/argocd/components/argocd-crds
- annotations:
description: configures namespaces for all stacks
labels:
holos.run/stack.name: security
name: namespaces
parameters:
stack: security
path: stacks/security/components/namespaces
holos show buildplans --selector holos.run/stack.name=argocd
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: argocd-crds
labels:
holos.run/stack.name: argocd
annotations:
description: argocd custom resource definitions
spec:
artifacts:
- artifact: components/argocd-crds/argocd-crds.gen.yaml
generators:
- kind: Resources
output: resources.gen.yaml
- kind: File
output: argocd-crds.2.13.2.yaml
file:
source: argocd-crds.2.13.2.yaml
transformers:
- kind: Kustomize
inputs:
- resources.gen.yaml
- argocd-crds.2.13.2.yaml
output: components/argocd-crds/argocd-crds.gen.yaml
kustomize:
kustomization:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- includeSelectors: false
pairs:
argocd.argoproj.io/instance: argocd-argocd-crds
resources:
- resources.gen.yaml
- argocd-crds.2.13.2.yaml
- artifact: gitops/argocd-crds.application.gen.yaml
generators:
- kind: Resources
output: gitops/argocd-crds.application.gen.yaml
resources:
Application:
argocd-argocd-crds:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
labels: {}
name: argocd-argocd-crds
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: argocd
source:
path: deploy/components/argocd-crds
repoURL: https://github.com/holos-run/kargo-demo.git
targetRevision: main
holos show buildplans --selector holos.run/component.name=gateway-api
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: gateway-api
labels:
holos.run/component.name: gateway-api
holos.run/stack.name: network
annotations:
description: gateway api custom resource definitions
spec:
artifacts:
- artifact: components/gateway-api/gateway-api.gen.yaml
generators:
- kind: Resources
output: resources.gen.yaml
- kind: File
output: standard/gateway.networking.k8s.io_gatewayclasses.yaml
file:
source: standard/gateway.networking.k8s.io_gatewayclasses.yaml
- kind: File
output: standard/gateway.networking.k8s.io_gateways.yaml
file:
source: standard/gateway.networking.k8s.io_gateways.yaml
- kind: File
output: standard/gateway.networking.k8s.io_grpcroutes.yaml
file:
source: standard/gateway.networking.k8s.io_grpcroutes.yaml
- kind: File
output: standard/gateway.networking.k8s.io_httproutes.yaml
file:
source: standard/gateway.networking.k8s.io_httproutes.yaml
- kind: File
output: standard/gateway.networking.k8s.io_referencegrants.yaml
file:
source: standard/gateway.networking.k8s.io_referencegrants.yaml
transformers:
- kind: Kustomize
inputs:
- resources.gen.yaml
- standard/gateway.networking.k8s.io_gatewayclasses.yaml
- standard/gateway.networking.k8s.io_gateways.yaml
- standard/gateway.networking.k8s.io_grpcroutes.yaml
- standard/gateway.networking.k8s.io_httproutes.yaml
- standard/gateway.networking.k8s.io_referencegrants.yaml
output: components/gateway-api/gateway-api.gen.yaml
kustomize:
kustomization:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- includeSelectors: false
pairs:
argocd.argoproj.io/instance: network-gateway-api
resources:
- resources.gen.yaml
- standard/gateway.networking.k8s.io_gatewayclasses.yaml
- standard/gateway.networking.k8s.io_gateways.yaml
- standard/gateway.networking.k8s.io_grpcroutes.yaml
- standard/gateway.networking.k8s.io_httproutes.yaml
- standard/gateway.networking.k8s.io_referencegrants.yaml
- artifact: gitops/gateway-api.application.gen.yaml
generators:
- kind: Resources
output: gitops/gateway-api.application.gen.yaml
resources:
Application:
network-gateway-api:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
labels: {}
name: network-gateway-api
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: network
source:
path: deploy/components/gateway-api
repoURL: https://github.com/holos-run/kargo-demo.git
targetRevision: main
holos show buildplans --selector holos.run/component.name=external-secrets-crds
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: external-secrets-crds
labels:
holos.run/component.name: external-secrets-crds
holos.run/stack.name: security
annotations:
description: external secrets custom resource definitions
spec:
artifacts:
- artifact: components/external-secrets-crds/external-secrets-crds.gen.yaml
generators:
- kind: Resources
output: resources.gen.yaml
- kind: File
output: bundle.0.10.7.yaml
file:
source: bundle.0.10.7.yaml
transformers:
- kind: Kustomize
inputs:
- resources.gen.yaml
- bundle.0.10.7.yaml
output: components/external-secrets-crds/external-secrets-crds.gen.yaml
kustomize:
kustomization:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- includeSelectors: false
pairs:
argocd.argoproj.io/instance: security-external-secrets-crds
patches:
- patch: |
- op: replace
path: /spec/conversion/webhook/clientConfig/service/name
value: external-secrets-webhook
- op: replace
path: /spec/conversion/webhook/clientConfig/service/namespace
value: external-secrets
target:
group: apiextensions.k8s.io
kind: CustomResourceDefinition
name: clustersecretstores.external-secrets.io
version: v1
- patch: |
- op: replace
path: /spec/conversion/webhook/clientConfig/service/name
value: external-secrets-webhook
- op: replace
path: /spec/conversion/webhook/clientConfig/service/namespace
value: external-secrets
target:
group: apiextensions.k8s.io
kind: CustomResourceDefinition
name: externalsecrets.external-secrets.io
version: v1
- patch: |
- op: replace
path: /spec/conversion/webhook/clientConfig/service/name
value: external-secrets-webhook
- op: replace
path: /spec/conversion/webhook/clientConfig/service/namespace
value: external-secrets
target:
group: apiextensions.k8s.io
kind: CustomResourceDefinition
name: secretstores.external-secrets.io
version: v1
resources:
- resources.gen.yaml
- bundle.0.10.7.yaml
- artifact: gitops/external-secrets-crds.application.gen.yaml
generators:
- kind: Resources
output: gitops/external-secrets-crds.application.gen.yaml
resources:
Application:
security-external-secrets-crds:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
labels: {}
name: security-external-secrets-crds
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: security
source:
path: deploy/components/external-secrets-crds
repoURL: https://github.com/holos-run/kargo-demo.git
targetRevision: main
We don't add this to the StackBuilder because it would manage a namespace named after the stack unconditionally. We follow the previous behavior of allowing a stack to selectively opt-into a Kargo project by labeling a namespace with kargo.akuity.io/project: "true"
Passing the version as a common parameter to both components.
This was previously managed as a Kargo Project prior to the refactor, integrated with Kargo and loading data through CUE embed. We preserve this behavior in the refactored structure, the yaml file is in pkg/config/certmanager/cert-manager.yaml
Configures the localhost certificate authory from mkcert.
Pretty much a straight copy.
Previously the component refered to a root level AppProjects structu built from all Projects. Now the component imports the platform config and builds AppProjects in the component definition at the leaf of the tree, which does keep the configuration in one file and is a bit more readable.
Straight forward migration from the old component. Only the namespace was referenced at the root of the component definition main package.
In this refactor we start mixing in additional custom resource types
using resources.config.cue
We need to keep the #Resources definition closed so we get good type
check errors, but we need each embedded definition open so they compose
with other embedded definitions. This is how we achieve those two
goals:
#Resources: {
{
kargo.#Resources
...
}
{
externalsecrets.#Resources
...
}
}
Fairly straight forward, imports common configuration from kargo.config.
Istio is effectively a sub-stack in and of itself, but there's no need to define a separate collection for it. Configuration values used by multiple components are imported from example.com/holos/pkg/config/istio and accessible as istio.config The istio Kargo Project will be handled separately, but the istio config package should support this use case just fine.
Straight-forward copy where the component imports the istio config package for config values shared with other components.
Straight-forward copy where the component imports the istio config package for config values shared with other components.
Straight-forward copy where the component imports the istio config package for config values shared with other components.
Straight-forward copy where the component imports the istio config package for config values shared with other components.
The addon-promoter component is intended to be shared by multiple components in multiple stacks. It lives next to namespaces in stacks/shared/components as a result. The component is configured primarily through input parameters passed from the Platform spec. Shared configuration values for istio are in the istio config package.
Refactor the httproutes component to the new import structure. Noticeably slower at 5s. Before the refactor it took 1.5s, which was also considered slow. Ideally it takes as much time as other components, about 300-500ms.
This patch is a major re-working of the previous KargoProjectBuilder to
a platform.#ProjectBuilder definition. We define a "project" in the
context of a holos platform as a Kargo Project specifically. It's a
special kind of Stack where a single input component definition is
expanded to multiple component build plans varying by the kargo project
promotion stages.
This could use some serious cleaning up and futher refactoring but it
does work and produces the same output manifests.
Note the httproutes component is much slower now, taking 35 seconds.
This patch leaves the slow down in so we can troubleshoot. It seems
related to additional for loops used in the project builder.
Avoid httproutes with a selector.
holos render platform --selector holos.run/component.name!=httproutes
Note the httproutes component is much slower again, taking close to 4 minutes. See previous commit for more information.
Manage kargo deployment stages for the podinfo and httpbin projects. ❯ time holos render platform --selector holos.run/component.name!=httproutes rendered kargo-secrets in 211.590625ms rendered kargo-project in 216.36375ms rendered kargo-project in 274.204834ms rendered kargo-stages in 277.49475ms rendered istiod in 317.303292ms rendered rollouts in 332.3255ms rendered kargo-stages in 374.860209ms rendered kargo in 394.55475ms rendered app-projects in 410.735417ms rendered istio-cni in 206.816292ms rendered argocd in 432.260708ms rendered gateway-api in 435.248584ms rendered istio-ztunnel in 219.499ms rendered istio-promoter in 246.363667ms rendered istio-gateway in 259.632667ms rendered local-ca in 168.3855ms rendered external-secrets in 234.405541ms rendered dev-httpbin in 213.351542ms rendered test-httpbin in 210.537375ms rendered istio-base in 677.776167ms rendered prod-us-east-httpbin in 243.859916ms rendered uat-httpbin in 276.014792ms rendered prod-us-west-httpbin in 220.047208ms rendered dev-podinfo in 237.91125ms rendered rollouts-crds in 826.557667ms rendered prod-us-central-httpbin in 309.637ms rendered test-podinfo in 225.151625ms rendered namespaces in 518.684833ms rendered uat-podinfo in 212.26825ms rendered argocd-crds in 846.892125ms rendered prod-us-central-podinfo in 196.1665ms rendered prod-us-east-podinfo in 234.80125ms rendered external-secrets-crds in 556.598125ms rendered prod-us-west-podinfo in 215.065459ms rendered cert-manager in 523.673167ms rendered platform in 918.445792ms holos render platform --selector holos.run/component.name!=httproutes 9.20s user 2.17s system 1101% cpu 1.032 total
Wasn't yet migrated. With this patch nothing remains but the httproutes component, which works but very slow at 4 minutes.
Uses the plain v1.#HTTPRoute definition imported with Timoni. The contstraints seem to be the issue, they're brutal on cue memory usage.
Switching away from the for loops appear cause evalv3=1 to produce the same output data as evalv3=0, only with a different field sort.
|
Potentially related to the 4 minute httproutes evaluation noted: |
jeffmccune
changed the title
jeffmccune
changed the title
Follows closely `cue mod init` which creates a module named `cue.example` with no path part. We create `holos.example` for a similar example module starting point. It's also fairly easy to search and replace `holos.example` across all *.cue files to change the module name. If the module name changes again, support this use case.
Avoid mucking up the repo root with go.mod and go.sum, they're needed only for the tests. Keep the Makefile at the root though since that's where it's expected to be.
Remove old cue configs no longer needed after the refactor.
pkg directory is an unnecessary go style convention. It only becomes necessary if we want a way to mix definitions into both config and types, but we don't. We use explicit imports instead.
Not used right now due to performance issue with a for loop in the httproutes component.
Previously when I worked I pushed to my own fork. Users are expected to do the same, reconfiguring all of the GitOps pieces using `holos render platform -t $USER` This patch adds back an example file included with a cue build tag to make this work as expected.
jeffmccune
force-pushed
the
jeff/reorg
branch
3 times, most recently
from
d738800 to
383fb38
Compare
Verified these steps by working through them locally.
Previously the reset-cluster script was hard-coded to check for kargo credentials. Other examples may use other credentials, so the script changes to apply all manifests in $(mkcert -CAROOT)/manifests/ immediately after a reset. The kargo-git-creds similarly changes to write the credentials to a file unique to the github app installation id so it doesn't write over other manifests in the manifests directory.
jeffmccune
force-pushed
the
jeff/reorg
branch
from
05974d1 to
0e143be
Compare
jeffmccune
force-pushed
the
jeff/reorg
branch
2 times, most recently
from
fe6d4fc to
931ba31
Compare
Use a yaml datafile so it's easier to integrate with Kargo.
In the demo we apply the istio-promoter with argocd. It would be nice to browse directly to the resources created, so we add argocd annotations to get the links in the web ui.
Without this patch we get permission denied errors trying to view resources in the ArgoCD web UI. Links also do not show up. This patch fixes RBAC so the admin role is granted by default.
jeffmccune
force-pushed
the
jeff/reorg
branch
from
931ba31 to
aab60a1
Compare
There's a race between the argocd controller and redis against the auth password. When the cluster is reset, the argocd web ui often returns 401 NOAUTH errors from redis. This patch manages the redis auth secret prior to managing the ArgoCD component. This avoids the race, both redis and argocd come up configured with the correct password.
jeffmccune
force-pushed
the
jeff/reorg
branch
from
2be22d4 to
d4ac36f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor the code from user feedback:
To address 1 we refactor the code to use explicit imports. The configuration previously at the module root moves into the package
holos.example/config/platform. Theholos render platformentrypoint changes meaning in this example to, "Platform.spec.components is a flattened list of all components configured in each stack of platform.stacks` where platform is the imported config package previously mentioned.To address 2 we rename a "Holos Project" to a "Holos Stack" to disambiguate it from other project concepts in the domain. There are Kargo Projects, ArgoCD Projects, ZITADEL Projects, Google Cloud Projects, etc...
Note this change list is intended as a true refactor, the configuration in the deploy directory should remain unchanged if possible.
Note this change list also runs into multiple performance issues with the way we handle
httproutes. At some points evaluation of this single component takes upwards of 4 minutes for a relatively small number of httproutes. This may be related to these two issues, or it may not be. I need to investigate further, the v1.#HTTPRoutes definition is notable for how heavily it uses field constraints.