[ci.jenkins.io] Move ephemeral Linux containers to AWS

[dduportal]

Requires #4319

• Set up the 2 Kubernetes clouds using the 2 distinct credentials created by the 2 helmfile releases of the chart jenkins-kubernetes-agents for agents and bom agents (see current setup in https://github.com/jenkins-infra/kubernetes-management/blob/main/clusters/cijioagents1.yaml#L35-L44 and https://github.com/jenkins-infra/jenkins-infra/blob/bb9e757b20dcec1da678948ef8b453d6d2cac928/hieradata/clients/controller.sponsorship.ci.jenkins.io.yaml#L116)
• Test container templates!

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 76% [ci.jenkins.io] Move ephemeral VM agents to AWS #4316

[dduportal]

Update: see the first bullet of #4319 (comment)

• Setup Jenkins namespace + SVC account + RBAC, and set up the controller VM IAM identity to map to this SVC account in order to have a credential-less setup (like it has been been with EC2 agents)

  • Don't forget to remove the kubeconfig's output from terraform once setup. Implies eventually updating the helm chart to NOT create a token (unneeded anymore)

=> all requirements from #4319 have been met (only leftover is related to the EC2 agents to ACP)

@smerle33 did already started to work on this issue: #4319 (comment), to ensure that ci.jenkins.io may access the EKS cluster cijenkinsio-agents2 without credentials using instance identity AND with the proper IAM and RBAC permissions (requires proper namespace and quotas):

• feat(eks_ci_jio_agents2) adding an access entries for cijio to use within eks terraform-aws-sponsorship#76
  • Fixup: fix(eks-cijio-agents2) principal arn to the role of ci.jio terraform-aws-sponsorship#77
• feat(jenkins-kubernetes-agents) allow specifying groups bound to the RBAC role used by Jenkins helm-charts#1510
• feat(cijenkinsio-agents-2) adding the jenkins agents for bom and others builds kubernetes-management#6026

[smerle33]

The aws jenkins controller can now connect to the cijenkinsio-agents2 kubernetes cluster without credential but using AWS cli and the .kube/config file providing the cluster CA cert, server url and a aws call for the user.

we need to provide the AWS cli in the controller container and the kube/config file

[dduportal]

Cherry picking comment by @smerle33: #4319 (comment):

for part 1 (jenkins namespace, service account, rbac and iam link with VM iam identity and kubernetes service account):

namespace, service account and rbac are dealt with the helm chart: jenkins-infra/helm-charts@main/charts/jenkins-kubernetes-agents

following this step, we need to provide aws cli within the container jenkins and a specific kubeconfig to handle the credential less authentication from ci.jenkins.io in aws.

• install aws cli from puppet within the host ci.jenkins.io - feat(aws.ci.jenkins.io + jenkinscontroller) optionally install aws CLI in the Jenkins container + enable it on aws.ci jenkins-infra#3831
• share the aws cli install folder in the container - feat(aws.ci.jenkins.io + jenkinscontroller) optionally install aws CLI in the Jenkins container + enable it on aws.ci jenkins-infra#3831
• copy the kubeconfig file from puppet into a folder shared in the container (jenkins_user_home/.kube/config) - feat(aws.ci.jenkins.io) provide kubeconfig for ci.jenkins.io-agents-2 EKS cluster in the container jenkins-infra#3833

[dduportal]

Update:

• We need to set up the Karpenter Node Pools to handle the "normal" builds and the BOM build
• I'm taking over the task above from Stephane as he's ill for 1 week (and must rest!)

[dduportal]

Update:

• I'm taking over the task above from Stephane as he's ill for 1 week (and must rest!)

• The aws CLI and the KUBECONFIG are provided in aws.ci. Creating an empty cloud with only the agents namespace provided is successful to authenticate using the EC2 Instance metadata!
  • feat(aws.ci.jenkins.io + jenkinscontroller) optionally install aws CLI in the Jenkins container + enable it on aws.ci jenkins-infra#3831
  • feat(aws.ci.jenkins.io) provide kubeconfig for ci.jenkins.io-agents-2 EKS cluster in the container jenkins-infra#3833
• The 2 new Kubernetes Clouds have been created in JCasC (feat(aws.ci.jenkins.io) adding the 2 clouds kub ci.jenkins.io-agents-2 and -bom jenkins-infra#3824) and are both working as expected (e.g. the "Test Connection" button in each Cloud Parameters page returns "1.29.10-eks;xxxx" with success.

Next steps:

• Set up the Karpenter Node groups and Node Class to support both clouds (normal linux amd64 and BOM linux amd64), using spot instance by default (and fallback to on-demand)
  • Note: we have to anticipate [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS #4318 and Eventual Linux arm64 for both "normal" and "BOM"
  • Let's check that Jenkins can schedule pods which will scale/downscale using Karpenter. Already tried manually: let's set it up as code properly
• We have to fix the domain name setup of aws.ci.jenkins.io if using "Direct Connection" for Kubernetes pods: we might see pods scheduled and starting, but the agent will most probably fail to start due to certificate issue. See [ci.jenkins.io] Move controller (VM) to AWS #4315 (comment)

[dduportal]

• Set up the Karpenter Node groups and Node Class to support both clouds (normal linux amd64 and BOM linux amd64), using spot instance by default (and fallback to on-demand)
  • Note: we have to anticipate [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS #4318 and Eventual Linux arm64 for both "normal" and "BOM"
  • Let's check that Jenkins can schedule pods which will scale/downscale using Karpenter. Already tried manually: let's set it up as code properly

Update: jenkins-infra/terraform-aws-sponsorship#105 has been deployed.

Tested with the test job that using Node labels maven-21 and maven-bom are both scheduling pods and nodes on the correct node pools with success.

Next step: moving effort to as the agents in the pod are failing to start.

[dduportal]

Update:

• Controller now has a valid public hostname with HTTPS enabled: [ci.jenkins.io] Move controller (VM) to AWS #4315 (comment)
• Container agents are failing to start "Connection Error" hinting that there is a network restriction forbidding the access, despite the 2 following SG rules:
  • Pod agent egress: https://github.com/jenkins-infra/terraform-aws-sponsorship/blob/79b85e4a67fb2436abeea5f1244c7631d90cabe1/eks-cijenkinsio-agents-2.tf#L138-L146
  • Controller ingress from all private subnets: https://github.com/jenkins-infra/terraform-aws-sponsorship/blob/79b85e4a67fb2436abeea5f1244c7631d90cabe1/network-security.tf#L555-L565

[dduportal]

Update:

• Security group errors fixed with feat(ci.jenkins.io) allow inbound JNLP from public NAT gateways IPs instead of private subnet CIDRs as we use public DNS terraform-aws-sponsorship#110
• https://aws.ci.jenkins.io/job/test/71/ did succeed to spin up an agent

Next steps (wip):

• Testing a build of https://github.com/jenkinsci/jenkins-infra-test-plugin
• Testing a build of https://github.com/jenkinsci/bom

[dduportal]

Update: first results

• Testing a build of https://github.com/jenkinsci/jenkins-infra-test-plugin

This test went well on the Linux part. The Windows label was blocked as we have not applied the "label" trick from #4490 which is currently in place on (azure) ci.jenkins.io.

Ran 2 builds which were successful on both ACP part (we see a faster build 2nd time due to faster dependency resolution) and Linux 21 build.

Pod is allocated immediately (time for the node to start).

See https://aws.ci.jenkins.io/job/jenkins-infra-test-plugin/job/master/1/ and https://aws.ci.jenkins.io/job/jenkins-infra-test-plugin/job/master/2/

• Testing a build of https://github.com/jenkinsci/bom

Initial build https://aws.ci.jenkins.io/job/bom/job/master/2/ shows:

• The initial prep stage took around 30 min (time to load the ACP cache for the first time)
• Karpenter did quickly find a set of available instances. Mostly Spot instances, and it packed up the agents with success
• Problem 1 (Node Disks): The parallel stages all failed due to "Disk Insufficient Storage" errors:

• Problem 2 (Subnets): the EKS Cluster Health Advisory complained about subnets being too small top handle the load:

• NOTE: our monitoring did alert us about disk usage. Both iNodes and available space. Which is good: it means the datadog agent is working as expected!

[dduportal]

• Problem 1 (Node Disks): The parallel stages all failed due to "Disk Insufficient Storage" errors:

Update: the node disk should now be 300 Gb since jenkins-infra/terraform-aws-sponsorship#111

[dduportal]

• Problem 2 (Subnets): the EKS Cluster Health Advisory complained about subnets being too small top handle the load:

• Increased the cluster capacity (increase subnet sie + another subnet): feat(eks/ci.jenkins.io-agents-2) extend private IPv4 capacity terraform-aws-sponsorship#112

  • Required a hotfix for output: hotfix(outputs) report the correct amount of EKS subnets terraform-aws-sponsorship#113

• ACP LB recreation (see feat(eks/ci.jenkins.io-agents-2) extend private IPv4 capacity terraform-aws-sponsorship#112 for why)

  • Track config from reports: chore(updatecli) track the ACP AWS LB configuration from Terraform Infra reports kubernetes-management#6144
  • Automatic PR: Update configuration of the Artifact Caching Proxy for AWS kubernetes-management#6145
    • Which failed to apply as the LB needs only 1 subnet (and IPv4) per availability zone
  • Hotfix to ensure 1 subnet and IP per AZ only: jenkins-infra/kubernetes-management@7a7a836

• "quick" re-test to ensure we can allocate pod agent and ACP is still available: https://aws.ci.jenkins.io/job/jenkins-infra-test-plugin/job/master/3/

• New BOM build triggered: https://aws.ci.jenkins.io/job/bom/job/master/3/ to study the new behavior