Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

selectively disable addons #275

Merged
merged 1 commit into from
Aug 19, 2022
Merged

selectively disable addons #275

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 29 additions & 15 deletions infrastructure/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ All other components are able to be installed via Flux resources.

### Components:

KIT Infrastructure creates a base K8s cluster with a few add-ons. Add-ons include permissions scoped to the pod using IAM Roles for Service Accounts (IRSA).
KIT Infrastructure creates a base K8s cluster with below add-ons by default but also provides an ability to not install some of these addons optionally through CDK context. Add-ons include permissions scoped to the pod using IAM Roles for Service Accounts (IRSA).

- EKS Cluster (host cluster)
- EKS Managed Node Group (for critical add-ons mentioned below)
- EBS CSI Driver
- EBS CSI Driver (optional)
- AWS Load Balancer Controller
- Karpenter
- Karpenter (optional)
- Flux v2
- Kubernetes Iteration Toolkit (KIT) Operator
- Kubernetes Iteration Toolkit (KIT) Operator (optional)

Flux is setup, by deafult, to monitor the KIT git repo path `./infrastructure/k8s-config/clusters/kit-infrastructure`, which includes other add-ons that do not require AWS credentials such as tekton, prometheus, grafana, and the metrics-server.

Expand All @@ -43,16 +43,30 @@ cdk deploy KITInfrastructure --no-rollback \
-c TestServiceAccount="karpenter-tests"
```

As an example, below are the parmeters used if you want to selectively disable some addons like Karpenter, EBSCSIDriver, KIT.
```shell
cdk bootstrap
cdk deploy KITInfrastructure --no-rollback \
-c TestNamespace="tekton-pipelines" \
-c TestServiceAccount="tekton-pipelines-executor" \
-c AWSEBSCSIDriverAddon=false \
-c KarpenterAddon=false \
-c KITAddon=false \
```

### Context Parameters:

| Context Param | Description | Default | | |
|--------------------|--------------------------------------------------------------------------------------------|---------------------------------------------------------|---|---|
| FluxRepoURL | Flux Source git repo URL to synchronize KIT infrastructure like Tekton | https://github.com/awslabs/kubernetes-iteration-toolkit | | |
| FluxRepoBranch | Flux Source git repo branch to synchronize KIT infrastructure | main | | |
| FluxRepoPath | Flux Source git repo path to Kubernetes resources | ./infrastructure/k8s-config/clusters/kit-infrastructure | | |
| TestFluxRepoName | Flux Source git repo name to synchronize application tests like Tekton Tasks and Pipelines | | | |
| TestFluxRepoURL | Flux Source git repo URL to synchronize application tests | | | |
| TestFluxRepoBranch | Flux Source git repo branch to synchronize application tests | | | |
| TestFluxRepoPath | Flux Source git repo path to Kubernetes resources | | | |
| TestNamespace | Namespace for application tests to run in | | | |
| TestServiceAccount | Service Account for application tests to run with | | | |
| Context Param | Description | Default | | |
|-------------------- |--------------------------------------------------------------------------------------------|---------------------------------------------------------|---|---|
| FluxRepoURL | Flux Source git repo URL to synchronize KIT infrastructure like Tekton | https://github.com/awslabs/kubernetes-iteration-toolkit | | |
| FluxRepoBranch | Flux Source git repo branch to synchronize KIT infrastructure | main | | |
| FluxRepoPath | Flux Source git repo path to Kubernetes resources | ./infrastructure/k8s-config/clusters/kit-infrastructure | | |
| TestFluxRepoName | Flux Source git repo name to synchronize application tests like Tekton Tasks and Pipelines | | | |
| TestFluxRepoURL | Flux Source git repo URL to synchronize application tests | | | |
| TestFluxRepoBranch | Flux Source git repo branch to synchronize application tests | | | |
| TestFluxRepoPath | Flux Source git repo path to Kubernetes resources | | | |
| TestNamespace | Namespace for application tests to run in | | | |
| TestServiceAccount | Service Account for application tests to run with | | | |
| KITAddon | KIT CRD addon that gets installed on KIT Infrastructure by default | true | | |
| KarpenterAddon | Karpenter CRD addon that gets installed on KIT Infrastructure by default | true | | |
| AWSEBSCSIDriverAddon| AWSEBSCSIDriver CRD addon that gets installed on KIT Infrastructure by default | true | | |
12 changes: 12 additions & 0 deletions infrastructure/k8s-config/clusters/kit-infrastructure/tekton-pipelines/tekton.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,18 @@ spec:
data:
default-task-run-workspace-binding: |
emptyDir: {}
- target:
kind: Service
name: tekton-dashboard
namespace: tekton-pipelines
patch: |-
apiVersion: v1
kind: Service
metadata:
name: tekton-dashboard
namespace: tekton-pipelines
spec:
type: NodePort
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
Expand Down
42 changes: 25 additions & 17 deletions infrastructure/lib/kit-infrastructure.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ export class KITInfrastructure extends Stack {
const repoUrl = this.getContextOrDefault('FluxRepoURL', "https://github.com/awslabs/kubernetes-iteration-toolkit")
const repoBranch = this.getContextOrDefault('FluxRepoBranch', 'main')
const repoPath = this.getContextOrDefault('FluxRepoPath', './infrastructure/k8s-config/clusters/kit-infrastructure')
const installEBSCSIDriverAddon = this.getContextOrDefault("AWSEBSCSIDriverAddon", "true")
const installKarpenterAddon = this.getContextOrDefault('KarpenterAddon', "true")
const installKitAddon = this.getContextOrDefault("KITAddon", "true")

const testRepoName = this.node.tryGetContext('TestFluxRepoName')
const testRepoUrl = this.node.tryGetContext('TestFluxRepoURL')
Expand Down Expand Up @@ -145,13 +148,14 @@ export class KITInfrastructure extends Stack {
})

// Install cluster add-ons for the host cluster

new AWSEBSCSIDriver(this, 'AWSEBSCSIDriver', {
cluster: cluster,
namespace: 'aws-ebs-csi-driver',
version: 'v1.9.0',
chartVersion: 'v2.8.1',
}).node.addDependency(cluster);
if (installEBSCSIDriverAddon == "true") {
new AWSEBSCSIDriver(this, 'AWSEBSCSIDriver', {
cluster: cluster,
namespace: 'aws-ebs-csi-driver',
version: 'v1.9.0',
chartVersion: 'v2.8.1',
}).node.addDependency(cluster);
}

new FluxV2(this, 'Flux', {
cluster: cluster,
Expand All @@ -172,17 +176,21 @@ export class KITInfrastructure extends Stack {
version: 'v2.4.2',
}).node.addDependency(cluster);

new KIT(this, 'KIT', {
cluster: cluster,
namespace: 'kit',
version: 'v0.0.18',
}).node.addDependency(cluster);
if(installKitAddon == "true"){
new KIT(this, 'KIT', {
cluster: cluster,
namespace: 'kit',
version: 'v0.0.18',
}).node.addDependency(cluster);
}

new Karpenter(this, 'KarpenterController', {
cluster: cluster,
namespace: 'karpenter',
nodeRoleName: workerRole.roleName,
}).node.addDependency(cluster);
if(installKarpenterAddon == "true") {
new Karpenter(this, 'KarpenterController', {
cluster: cluster,
namespace: 'karpenter',
nodeRoleName: workerRole.roleName,
}).node.addDependency(cluster);
}
}

private getContextOrDefault(key: string, def: string | null): any {
Expand Down
Loading