Merge remote-tracking branch 'origin/master' into timmy/windows-compo…

…nents-json
Azure · Feb 3, 2025 · a8d3e43 · a8d3e43
2 parents 7b3d25e + b95d1c7
commit a8d3e43
Show file tree

Hide file tree

Showing 706 changed files with 142,194 additions and 20,638 deletions.
diff --git a/.github/README-RENOVATE.md b/.github/README-RENOVATE.md
@@ -7,7 +7,7 @@
   - [Package rules](#package-rules)
     - [Disable `minor` update](#disable-minor-update)
     - [Enable `patch`, `pin` and `digest` update](#enable-patch-pin-and-digest-update)
-    - [(Optional context) Why not updating minor?](#optional-context-why-not-updating-minor)
+    - [(Optional context) To update minor or not](#optional-context-to-update-minor-or-not)
     - [Assigning specific component to specific person](#assigning-specific-component-to-specific-person)
     - [Additional string operation to specific component](#additional-string-operation-to-specific-component)
   - [Custom managers](#custom-managers)
@@ -73,26 +73,38 @@ In summary, this package rule is saying it will apply auto-update without `autom
 
 Combining these 2 package rules together is actually asking Renovate not to update `major` and `minor`, but just `patch`, `pin` and `digest`.
 
+We configured auto-merge patch version for components `moby-runc` and `moby-containerd`. Please search `"matchPackageNames": ["moby-runc", "moby-containerd"]` in `renovate.json` for an example.
+
+As of 01/23/2025, the PR merging policy is as follows.
+| Components       | Major  | Minor  | Patch  |
+| --------         | ------ | ------ | ------ |
+| Runc, Containerd | Manual | Manual | Auto   |
+| Others           | Manual | Manual | Manual |
+
+The update of `Runc` and `Containerd` is owned by Node SIG and we have sufficient confidence to auto-merge it with our tests and PR gates. Thus it's set to `auto-merge`.
+For other components, we are still relying on the owner teams to approve and merge. If there is a need to auto-merge a component, it's always configurable.
+
+---
+
 For more context to anyone who is interested, let's walk through a real example. Feel free to skip reading this if it has nothing to do with your task.
-### (Optional context) Why not updating minor?
+
+### (Optional context) To update minor or not
 Using azure-cni as an example, if we enable auto updating `minor`, we will see the following PRs created by Renovate automatically at of Sep 12, 2024.
 - PR1: containernetworking/azure-cni minor v1.5.32 -> v1.6.6
 - PR2: containernetworking/azure-cni patch v1.6.3 -> v1.6.6
 - PR3: containernetworking/azure-cni patch v1.5.32 -> v1.5.36
 
 PR2 and PR3 are what we need because we want to auto-update the patch version.
 By enabling the `minor` package rule, PR1 will also be created.
-This will be a noise PR to the assignee because now he/she needs to manually check if the 2 latest versions of v1.5.x are still kept in the components.json if this PR is merged. And usually it's not. Thus the assignee will cancel this PR manually every time.
-If a new minor version needs to be added, the owner should update the components.json.
+Depending on use case, for some components this could be a helpful reminder that there are updated versions but for others this could become noisy if it's intended not to update minor version but only patch version.
 
-If there is only 1 component then it should be fine as it won't have too many noisy PRs.
-But in components.json, it's managing 50+ components. So with `minor` package rule enabled, it will look like this screenshot.
+In components.json, it's managing 50+ components. So with `minor` package rule enabled, it will look like this screenshot.
 ![Renovate Minor Enabled](./images/Renovate_minor_enabled.png)
-On the left side, there is no minor being updated. On the right side, it added many PRs for updating the minor but most of them should be just noise.
+On the left side, there is no `minor` being updated. On the right side, it added many PRs for updating the `minor` too.
 
-That's why we ended up disabling `minor` auto-update to avoid the noisy PRs.
+We enabled auto-updating `minor` versions as default configuration for now we can revise if it ends up being too noisy. We can also turn on/off updating `minor` version for specific components.
 
-p.s. To allow disable `minor` update but enable `patch`, `pin`, `digest` update, at root level `separateMinorPatch` needs to be `true`.
+p.s. To allow disable `minor` update but enable `patch` update, at root level `separateMinorPatch` needs to be `true`, though it's no harm to turn it on even we are also updating `minor`.
 
 ### Assigning specific component to specific person
 We can use `matchPackageNames` to achieve this purpose.
@@ -101,9 +113,10 @@ For example,
     {
       "matchPackageNames": ["moby-runc", "moby-containerd"],
       "assignees": ["devinwong", "anujmaheshwari1", "cameronmeissner", "AlisonB319", "lilypan26", "djsly", "jason1028kr", "UtheMan", "zachary-bailey", "ganeshkumarashok"]
+      "reviewers": ["devinwong", "anujmaheshwari1", "cameronmeissner", "AlisonB319", "lilypan26", "djsly", "jason1028kr", "UtheMan", "zachary-bailey", "ganeshkumarashok"]
     },
 ```
-In this block, it is saying that if the package name, that a PR is updating, is one of the defined values, then assign this PR to these Github IDs.
+In this block, it is saying that if the package name, that a PR is updating, is one of the defined values, then assign this PR to these Github IDs. The values in `reviewers` are the same group of people to allow them to self-approve the PR. Unfortunately JSON doesn't support variable in value so we have to provide the value strings twice for both `assignees` and `reviewers`
 
 ### Additional string operation to specific component
 ```
@@ -293,7 +306,7 @@ Depending on what kind of component you are going to onboard.
       ]
     }
   ```
-  Please make sure you set the `renovateTag` correctly, where `registry` is always `https://mcr.microsoft.com` now, and the `name` doesn't have a leading slash `/`. As of Sept 2024, The container Images in `components.json` are all hosted in MCR and MCR is the only registry enabled in the current Renovate configuration file `renovate.json`. If there is demand for other container images registry, it will be necessary to double check if it will just work.
+  Please make sure you set the `renovateTag` correctly, where `registry` is always `https://mcr.microsoft.com` now, and the `name` doesn't have a leading slash `/`. As of Jan 2025, The container Images in `components.json` are all hosted in MCR and MCR is the only registry enabled in the current Renovate configuration file `renovate.json`. If there is demand for other container images registry, it will be necessary to double check if it will just work.
 
   Fore more details, you can refer to Readme-components linked at the beginning of this document.
 
@@ -324,9 +337,10 @@ There is an example for packages `moby-runc` and `moby-containred`
     {
       "matchPackageNames": ["moby-runc", "moby-containerd"],
       "assignees": ["devinwong"]
+      "reviewers": ["devinwong"]
     },
 ```
-You can follow this example to create a block and fill in the matchPackageNames with your **GitHub ID** to assign to yourself, assuming you are the owner. Note that the packageName here must be the exact name that you can find in your datasource. For example, in the datasource PMC which hosts `moby-runc` and `moby-containerd`, we are running `apt-get install moby-runc moby-containerd`. So this is the correct package name.
+You can follow this example to create a block and fill in the matchPackageNames with your **GitHub ID** to assign and set reviewer to yourself, assuming you are the owner. Note that the packageName here must be the exact name that you can find in your datasource. For example, in the datasource PMC which hosts `moby-runc` and `moby-containerd`, we are running `apt-get install moby-runc moby-containerd`. So this is the correct package name.
 
 Another example is for a container image `mcr.microsoft.com/oss/kubernetes/kube-proxy`. In this case you should fill in the matchPackageNames with packageName `oss/kubernetes/kube-proxy`. Note there is no leading slash `/`.
 
@@ -343,9 +357,9 @@ If your GitHub ID is placed in the `assignees` array, you are responsible for th
 ## What components are onboarded to Renovate for auto-update and what are not yet?
 In general, if a component has the `"renovateTag": "<DO_NOT_UPDATE>"`, it means it's not monitored by Renovate and won't be updated automatically.
 
-As of 11/12/2024,
+As of 01/23/2025,
 - All the container images are onboarded to Renovate for auto-update.
-- PMC hosted packages, namely `runc` and `containerd`, are onboarded for auto-update.
+- PMC hosted packages, namely `runc` and `containerd`, are configured as auto-merge patch version.
 - OCI artifacts hosted on MAR(aka MCR) such as `kubernetes-binaries`, `azure-acr-credential-provider` and `containerd-wasm-shims` are onboarded for auto-update.
 - Acs-mirror hosted packages/binaries, namely `cni-plugins`, `azure-cni`, `cri-tools`, etc., are NOT onboarded for auto-update yet. There are plans to move the acs-mirror hosted packages to MCR OCI which will be downloaded by Oras. We will wait for this transition to be completed to understand the details how to manage them.
 

diff --git a/.github/renovate.json b/.github/renovate.json
@@ -8,6 +8,7 @@
   "prHourlyLimit": 50,
   "branchConcurrentLimit": 50,
   "separateMinorPatch": true,
+  "recreateWhen": "never",
   "labels": ["renovate"],
   "logLevelRemap": [
     {
@@ -16,14 +17,6 @@
     }
   ],
   "packageRules": [
-    {
-      "matchDatasources": ["docker", "custom.deb1804", "custom.deb2004", "custom.deb2204", "custom.deb2404"],
-      "matchUpdateTypes": [
-        "minor"
-      ],
-      "automerge": false,
-      "enabled": false
-    },
     {
       "matchDatasources": ["docker"],
       "matchUpdateTypes": [
@@ -70,6 +63,11 @@
     },
     {
       "matchPackageNames": ["moby-runc", "moby-containerd"],
+      "matchUpdateTypes": [
+        "patch"
+      ],
+      "automerge": true,
+      "enabled": true,
       "assignees": ["devinwong", "anujmaheshwari1", "cameronmeissner", "AlisonB319", "lilypan26", "djsly", "jason1028kr", "UtheMan", "zachary-bailey", "ganeshkumarashok"],
       "reviewers": ["devinwong", "anujmaheshwari1", "cameronmeissner", "AlisonB319", "lilypan26", "djsly", "jason1028kr", "UtheMan", "zachary-bailey", "ganeshkumarashok"]
     },
@@ -94,18 +92,28 @@
       "reviewers": ["mainred"]
     },
     {
-      "matchPackageNames": ["moby-runc", "moby-containerd"],
-      "extractVersion": "^v?(?<version>.+)$"
+      "matchPackageNames": ["oss/kubernetes-csi/*"],
+      "assignees": ["andyzhangx"],
+      "reviewers": ["andyzhangx"]
+    },
+    {
+      "matchPackageNames": ["oss/binaries/kubernetes/kubernetes-node", "oss/kubernetes/kube-proxy"],
+      "assignees": ["haitch", "wenhug", "gaopenghigh"],
+      "reviewers": ["haitch", "wenhug", "gaopenghigh"]
     },
     {
       "matchDatasources": ["docker"],
       "matchPackageNames": ["oss/binaries/kubernetes/kubernetes-node", "oss/binaries/kubernetes/azure-acr-credential-provider", "oss/binaries/deislabs/containerd-wasm-shims"],
       "extractVersion": "^(?P<version>.*?)-[^-]*-[^-]*$"
     },
+    {
+      "matchPackageNames": ["moby-runc", "moby-containerd"],
+      "extractVersion": "^v?(?<version>.+)$"
+    },
     {      
       "matchPackageNames": ["aks/aks-gpu-cuda", "aks/aks-gpu-grid"],      
       "versioning": "regex:^(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-(?<prerelease>\\d{14})$",
-      "automerge": false,      
+      "automerge": true,      
       "enabled": true,
       "ignoreUnstable": false
     }
@@ -235,4 +243,4 @@
       ]
     }
   }
-}
+}
diff --git a/.pipelines/.vsts-vhd-builder-release-windows.yaml b/.pipelines/.vsts-vhd-builder-release-windows.yaml
@@ -29,6 +29,14 @@ pr:
 pool:
   name: $(AZURE_POOL_NAME)
 
+# Some templates use POOL_NAME instead of AZURE_POOL_NAME, so set POOL_NAME here just in case.
+variables:
+  VHD_BUILD_ID: $(Build.BuildId)
+  LOCATION: $(PACKER_BUILD_LOCATION)
+  POOL_NAME: $(AZURE_POOL_NAME)
+  isPR: $[eq(variables['Build.Reason'], 'PullRequest')]
+  isNotPR: $[ne(variables['Build.Reason'], 'PullRequest')]
+
 parameters:
 - name: build2019containerd
   displayName: Build 2019 containerd
@@ -59,12 +67,6 @@ parameters:
   type: boolean
   default: False
 
-# Some templates use POOL_NAME instead of AZURE_POOL_NAME, so set POOL_NAME here just in case.
-variables:
-  VHD_BUILD_ID: $(Build.BuildId)
-  LOCATION: $(PACKER_BUILD_LOCATION)
-  POOL_NAME: $(AZURE_POOL_NAME)
-
 # Use variable group "ab-windows-ame-tenant" and link it to the pipeline "AKS Windows VHD Build"
 # Use variable group "ab-windows-ame-tenant" and link it to the pipeline "AKS Windows VHD Build - PR check-in gate"
 # Use variable group "ab-windows-ms-tenant" and link it to the pipeline "[TEST All VHDs] AKS Windows VHD Build - Msft Tenant"
@@ -88,7 +90,7 @@ stages:
       imageName: windows-2022-containerd
       windowsSku: 2022-containerd
       hyperVGeneration: V1
-      build: ${{ parameters.build2022containerd }}
+      build: ${{ and(eq(parameters.build2022containerd, true), eq(variables.isNotPR, true)) }}
       vhddebug: ${{ parameters.vhddebug }}
       dryrun: ${{ parameters.dryrun }}
 
@@ -110,7 +112,7 @@ stages:
       imageName: windows-23H2
       windowsSku: 23H2
       hyperVGeneration: V1
-      build: ${{ parameters.build23H2 }}
+      build: ${{ and(eq(parameters.build23H2, true), eq(variables.isNotPR, true)) }}
       vhddebug: ${{ parameters.vhddebug }}
       dryrun: ${{ parameters.dryrun }}
 

diff --git a/.pipelines/.vsts-vhd-builder-release.yaml b/.pipelines/.vsts-vhd-builder-release.yaml
@@ -1174,7 +1174,7 @@ stages:
               echo '##vso[task.setvariable variable=IMG_SKU]20_04-lts-cvm'
               echo '##vso[task.setvariable variable=IMG_VERSION]latest'
               echo '##vso[task.setvariable variable=HYPERV_GENERATION]V2'
-              echo '##vso[task.setvariable variable=AZURE_VM_SIZE]Standard_D16ds_v5'
+              echo '##vso[task.setvariable variable=AZURE_VM_SIZE]Standard_DC16ads_v5'
               echo '##vso[task.setvariable variable=FEATURE_FLAGS]None'
               echo '##vso[task.setvariable variable=CONTAINER_RUNTIME]containerd'
               echo '##vso[task.setvariable variable=ARCHITECTURE]X86_64'
@@ -1272,3 +1272,6 @@ stages:
       TAGS_TO_SKIP: "os=windows"
     jobs:
       - template: ./templates/e2e-template.yaml
+        parameters:
+          name: All Linux
+
diff --git a/.pipelines/e2e-windows.yaml b/.pipelines/e2e-windows.yaml
@@ -30,3 +30,5 @@ pr:
       - /**/*.md
 jobs:
   - template: ./templates/e2e-template.yaml
+    parameters:
+      name: Windows Tests
diff --git a/.pipelines/e2e.yaml b/.pipelines/e2e.yaml
@@ -29,3 +29,5 @@ pr:
       - /**/*.md
 jobs:
   - template: ./templates/e2e-template.yaml
+    parameters:
+      name: Linux Tests
diff --git a/.pipelines/templates/.build-and-test-windows-vhd-template.yaml b/.pipelines/templates/.build-and-test-windows-vhd-template.yaml
@@ -33,6 +33,8 @@ parameters:
 
 stages:
   - stage: build_${{ parameters.stageName }}
+    # Put the artifact name first so it doesn't get truncated as much in ADO
+    displayName: Build (${{ parameters.artifactName }})
     condition: and(succeeded(), eq('${{ parameters.build }}', True))
     dependsOn: [ ]
     jobs:
@@ -56,9 +58,13 @@ stages:
               artifactName: ${{ parameters.artifactName }}
 
   - stage: e2e_${{ parameters.stageName }}
+    displayName: E2E (${{ parameters.artifactName }})
     dependsOn: build_${{ parameters.stageName }}
     condition: and(succeeded(), eq('${{ parameters.build }}', True))
     variables:
       TAGS_TO_RUN: imageName=${{ parameters.imageName }}
     jobs:
       - template: ./e2e-template.yaml
+        parameters:
+          name: For image ${{ parameters.imageName }}
+
diff --git a/.pipelines/templates/.builder-release-template.yaml b/.pipelines/templates/.builder-release-template.yaml
@@ -111,6 +111,7 @@ steps:
       echo "##vso[task.setvariable variable=SKU_NAME]$SKU_NAME"
       echo "Set SKU_NAME to $SKU_NAME"
     displayName: Set SKU Name
+  
   - bash: make -f packer.mk run-packer
     displayName: Build VHD
     retryCountOnTaskFailure: 3
@@ -128,6 +129,8 @@ steps:
       PACKER_VNET_NAME="$(cat vhdbuilder/packer/settings.json | grep "vnet_name" | awk -F':' '{print $2}' | awk -F'"' '{print $2}')" && \
       CAPTURED_SIG_VERSION="$(cat vhdbuilder/packer/settings.json | grep "captured_sig_version" | awk -F':' '{print $2}' | awk -F'"' '{print $2}')" && \
       SIG_IMAGE_NAME="$(cat vhdbuilder/packer/settings.json | grep "sig_image_name" | awk -F':' '{print $2}' | awk -F'"' '{print $2}')" && \
+      BUILD_LOCATION="$(cat vhdbuilder/packer/settings.json | grep "location" | awk -F':' '{print $2}' | awk -F'"' '{print $2}')" && \
+      echo "##vso[task.setvariable variable=PACKER_BUILD_LOCATION]${BUILD_LOCATION}" && \
       echo "##vso[task.setvariable variable=PACKER_VNET_RESOURCE_GROUP_NAME]${PACKER_VNET_RESOURCE_GROUP_NAME}" && \
       echo "##vso[task.setvariable variable=PACKER_VNET_NAME]${PACKER_VNET_NAME}" && \
       echo "##vso[task.setvariable variable=VHD_NAME]${CAPTURED_SIG_VERSION}.vhd" && \

diff --git a/.pipelines/templates/e2e-template.yaml b/.pipelines/templates/e2e-template.yaml
@@ -1,3 +1,9 @@
+parameters:
+  - name: name
+    type: string
+    displayName: Additional name for the pipeline step
+    default: ""
+
 jobs:
   - job: e2e
     condition: and(succeeded(), ne(variables.SKIP_E2E_TESTS, 'true'))
@@ -6,7 +12,7 @@ jobs:
     pool:
       name: $(E2E_POOL_NAME)
     timeoutInMinutes: 90
-    displayName: Run AgentBaker E2E
+    displayName: Run AgentBaker E2E ${{parameters.name}}
     steps:
       - bash: |
           set -ex

diff --git a/CODEOWNERS b/CODEOWNERS
@@ -5,5 +5,5 @@ cse_cmd.sh @Devinwong @lilypan26 @r2k1 @timmy-wright
 nodecustomdata.yml @Devinwong @lilypan26 @r2k1 @timmy-wright
 
 # Code owners for the security patch release notes
-/release-notes/security-patch/ @yagmurbaydogan @yewmsft @juan-lee @cameronmeissner @UtheMan @ganeshkumarashok @anujmaheshwari1 @AlisonB319 @Devinwong @lilypan26 @AbelHu @junjiezhang1997 @jason1028kr @djsly @phealy @r2k1 @timmy-wright @zachary-bailey
+vhdbuilder/release-notes/security-patch/ @yagmurbaydogan @yewmsft @juan-lee @cameronmeissner @UtheMan @ganeshkumarashok @anujmaheshwari1 @AlisonB319 @Devinwong @lilypan26 @AbelHu @junjiezhang1997 @jason1028kr @djsly @phealy @r2k1 @timmy-wright @zachary-bailey
 
diff --git a/aks-node-controller/Makefile b/aks-node-controller/Makefile
@@ -13,4 +13,4 @@ proto-generate:
 .PHONY: proto-lint
 proto-lint:
 	@($(BUF) lint)
-	@($(BUF) breaking --against '../.git#branch=dev,subdir=aks-node-controller') # TODO: change to master
+	@($(BUF) breaking --against '../.git#branch=master,subdir=aks-node-controller')