Sync with upstream v1.28.0

addon-resizer/enhancements/5546-scaling-based-on-container-count/README.md

@@ -0,0 +1,123 @@
+# KEP-5546: Scaling based on container count
+
+<!-- toc -->
+- [Summary](#summary)
+    - [Goals](#goals)
+    - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+    - [Notes](#notes)
+    - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+    - [Test Plan](#test-plan)
+<!-- /toc -->
+
+## Summary
+
+Currently Addon Resizer supports scaling based on the number of nodes. Some workloads use resources proportionally to
+the number of containers in the cluster. Since number of containers per node is very different in different clusters
+it's more resource-efficient to scale such workloads based directly on the container count. 
+
+### Goals
+
+- Allow scaling workloads based on count of containers in a cluster.
+- Allow this for Addon Resizer 1.8 ([used by metrics  server]).
+
+### Non-Goals
+
+- Using both node and container count to scale workloads.
+- Bringing this change to the `master` branch of Addon Resizer.
+
+## Proposal
+
+Add flag `--scaling-mode` to Addon Resizer on the [`addon-resizer-release-1.8`] branch. Flag will
+have two valid values:
+
+- `node-proportional` - default, current behavior.
+- `container-proportional` - addon resizer will set resources, using the same algorithm it's using now but using number
+  of containers where it's currently using number of nodes.
+
+### Notes
+
+Addon Resizer 1.8 assumes in multiple places that it's scaling based on the number of nodes:
+
+- [Flag descriptions] that directly reference node counts (`--extra-cpu`, `--extra-memory`, `--extra-storage`, and
+  `--minClusterSize`) will need to be updated to instead refer to cluster size.
+- [README] will need to be updated to reference cluster size instead of node count and explain that cluster size refers
+  to either node count or container count, depending on the value of the `--scaling-mode` flag.
+- Many variable names in code which now refer to node count will refer to cluster size and should be renamed accordingly.
+
+In addition to implementing the feature we should also clean up the code and documentation.  
+
+### Risks and Mitigations
+
+One potential risk is that Addon resizer can obtain cluster size (node count or container count):
+- from metrics or
+- by querying Cluster Api Server to list all objects of the appropriate type
+
+depending on the configuration. There can be many times more containers in a cluster that there are nodes. So listing
+all containers could result in higher load on the Cluster API server. Since Addon Resizer is requesting very few fields
+I don't expect this effect to be noticeable.
+
+Also I expect metrics-server to test for this before using the feature and any other users of Addon Resizer are likely
+better off using metrics (which don't have this problem). 
+
+## Design Details
+
+- Implement function `kubernetesClient.CountContainers()`. It will be analogous to the existing
+  [`kubernetesClient.CountNodes()`] function.
+  - If using metrics to determine number of containers in the cluster:
+    - Fetch pod metrics (similar to [fetching node metrics] but use `/pods` URI instead of `/nodes`).
+    - For each pod obtain number of containers (length of the `containers` field).
+    - Sum container counts for all pods.
+  - If using API server:
+    - Fetch list pods (similar to [listing nodes])
+      - Fetch only [`Spec.InitContainers`], [`Spec.Containers`], and [`Spec.EphemeralContainers`] fields.
+      - Exclude pods in terminal states ([selector excluding pods in terminal states in VPA])
+    - Sum container count over pods.
+- Add the `--scaling-mode` flag, with two valid values:
+  - `node-proportional` - default, current behavior, scaling based on clusters node count and
+  - `container-proportional` - new behavior, scaling based on clusters container count
+- Pass value indicating if we should use node count or container count to the [`updateResources()`] function.
+- In `updateResources()` use node count or container count, depending on the value.
+
+Check that listing containers directly works
+
+Coinsider listing pods, getting containers only for working pods
+
+### Test Plan
+
+In addition to unit tests we will run manual e2e test:
+
+- Create config based on [`example.yaml`] but scaling the deployment based on the number of containers in the cluster.
+- Create config starting deployment with 100 `pause` containers.
+
+Test the feature by:
+
+- Starting the deployment scaled by Addon Resizer, based on node count.
+- Observe size of the deployment and that it's stable.
+- Start deployment with 100 `pause` containers.
+- Observe the scaled deployment change resources appropriately.
+
+Test the node-based scaling:
+
+- Apply [`example.yaml`].
+- Observe amount and stability assigned resources.
+- Resize cluster.
+- Observe change in assigned resources.
+
+Both tests should be performed with metrics- and API- based scaling.
+
+[used by metrics  server]: https://github.com/kubernetes-sigs/metrics-server/blob/0c47555e9b49cfe0719db1a0b7fb6c8dcdff3d38/charts/metrics-server/values.yaml#L121
+[`addon-resizer-release-1.8`]: https://github.com/kubernetes/autoscaler/tree/addon-resizer-release-1.8
+[Flag descriptions]: https://github.com/kubernetes/autoscaler/blob/da500188188d275a382be578ad3d0a758c3a170f/addon-resizer/nanny/main/pod_nanny.go#L47
+[README]: https://github.com/kubernetes/autoscaler/blob/da500188188d275a382be578ad3d0a758c3a170f/addon-resizer/README.md?plain=1#L1
+[`kubernetesClient.CountNodes()`]: https://github.com/kubernetes/autoscaler/blob/da500188188d275a382be578ad3d0a758c3a170f/addon-resizer/nanny/kubernetes_client.go#L58
+[fetching node metrics]: https://github.com/kubernetes/autoscaler/blob/da500188188d275a382be578ad3d0a758c3a170f/addon-resizer/nanny/kubernetes_client.go#L150
+[listing nodes]: https://github.com/kubernetes/autoscaler/blob/da500188188d275a382be578ad3d0a758c3a170f/addon-resizer/nanny/kubernetes_client.go#L71
+[`Spec.InitContainers`]: https://github.com/kubernetes/api/blob/1528256abbdf8ff2510112b28a6aacd239789a36/core/v1/types.go#L3143
+[`Spec.Containers`]: https://github.com/kubernetes/api/blob/1528256abbdf8ff2510112b28a6aacd239789a36/core/v1/types.go#L3150
+[`Spec.EphemeralContainers`]: https://github.com/kubernetes/api/blob/1528256abbdf8ff2510112b28a6aacd239789a36/core/v1/types.go#L3158
+[`Status.Phase`]: https://github.com/kubernetes/api/blob/1528256abbdf8ff2510112b28a6aacd239789a36/core/v1/types.go#L4011
+[selector excluding pods in terminal states in VPA]: https://github.com/kubernetes/autoscaler/blob/04e5bfc88363b4af9fdeb9dfd06c362ec5831f51/vertical-pod-autoscaler/e2e/v1beta2/common.go#L195
+[`updateResources()`]: https://github.com/kubernetes/autoscaler/blob/da500188188d275a382be578ad3d0a758c3a170f/addon-resizer/nanny/nanny_lib.go#L126
+[`example.yaml`]: https://github.com/kubernetes/autoscaler/blob/c8d612725c4f186d5de205ed0114f21540a8ed39/addon-resizer/deploy/example.yaml

addon-resizer/enhancements/5700-nanny-configuration-reload/README.md

@@ -0,0 +1,61 @@
+# KEP-5546: Automatic reload of nanny configuration when updated
+
+<!-- toc -->
+- [Summary](#summary)
+    - [Goals](#goals)
+    - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+    - [Notes](#notes)
+    - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+    - [Test Plan](#test-plan)
+<!-- /toc -->
+
+Sure, here's the enhancement proposal in the requested format:
+
+## Summary
+- **Goals:** The goal of this enhancement is to improve the user experience for applying nanny configuration changes in the addon-resizer 1.8 when used with the metrics server. The proposed solution involves automatically reloading the nanny configuration whenever changes occur, eliminating the need for manual intervention and sidecar containers. 
+- **Non-Goals:** This proposal does not aim to update the functional behavior of the addon-resizer.
+
+## Proposal
+The proposed solution involves updating the addon-resizer with the following steps:
+- Create a file system watcher using `fsnotify` under `utils/fswatcher` to watch nanny configurations' changes. It should run as a goroutine in the background.
+- Detect changes of the nanny configurations' file using the created `fswatcher` trigger the reloading process when configuration changes are detected. Events should be sent in a channel.
+- Re-execute the method responsible for building the NannyConfiguration `loadNannyConfiguration` to apply the updated configuration to the addon-resizer.
+- Proper error handling should be implemented to manage scenarios where the configuration file is temporarily inaccessible or if there are parsing errors in the configuration file.
+
+### Risks and Mitigations
+- There is a potential risk of filesystem-related issues causing the file watcher to malfunction. Proper testing and error handling should be implemented to handle such scenarios gracefully.
+- Errors in the configuration file could lead to unexpected behavior or crashes. The addon-resizer should handle parsing errors and fall back to the previous working configuration if necessary.
+
+## Design Details
+- Create a new package for the `fswatcher` under `utils/fswatcher`. It would contain the `fswatcher` struct and methods and unit-tests.
+    - `FsWatcher` struct would look similar to this:
+    ```go
+    type FsWatcher struct {
+        *fsnotify.Watcher
+
+        Events    chan struct{}
+        ratelimit time.Duration
+        names     []string
+        paths     map[string]struct{}
+    }
+    ```
+    - Implement the following functions:
+        - `CreateFsWatcher`: Instantiates a new `FsWatcher` and start watching on file system.
+        - `initWatcher`: Initializes the `fsnotify` watcher and initialize the `paths` that would be watched.
+        - `add`: Adds a new file to watch.
+        - `reset`: Re-initializes the `FsWatcher`.
+        - `watch`: watches for the configured files.
+- In the main function, we create a new `FsWatcher` and then we wait in an infinite loop to receive events indicating
+filesystem changes. Based on these changes, we re-execute `loadNannyConfiguration` function.
+
+> **Note:** The expected configuration file format is YAML. It has the same structure as the NannyConfiguration CRD.
+
+### Test Plan
+To ensure the proper functioning of the enhanced addon-resizer, the following test plan should be executed:
+1. **Unit Tests:** Write unit tests to validate the file watcher's functionality and ensure it triggers events when the configuration file changes.
+2. **Manual e2e Tests:** Deploy the addon-resizer with `BaseMemory` of `300Mi` and then we change the `BaseMemory` to `100Mi`. We should observer changes in the behavior of watched pod.
+
+
+[fsnotify]: https://github.com/fsnotify/fsnotify

balancer/deploy/controller.yaml

@@ -20,6 +20,12 @@ rules:
       - watch
       - patch
       - update
+  - apiGroups:
+      - balancer.x-k8s.io
+    resources:
+      - balancers/status
+    verbs:
+      - update
   - apiGroups:
       - ""
     resources:

balancer/proposals/balancer.md

@@ -10,7 +10,7 @@ These domains may include:
 * Cloud provider zones inside a single region, to ensure that the application is still up and running, even if one of the zones has issues.
 * Different types of Kubernetes nodes. These may involve nodes that are spot/preemptible, or of different machine families. 
 
-A single Kuberentes deployment may either leave the placement entirely up to the scheduler 
+A single Kubernetes deployment may either leave the placement entirely up to the scheduler 
 (most likely leading to something not entirely desired, like all pods going to a single domain) or 
 focus on a single domain (thus not achieving the goal of being in two or more domains). 
 
@@ -179,4 +179,4 @@ type BalancerStatus struct {
    // +patchStrategy=merge
    Conditions []metav1.Condition
 }
-```
+```

builder/Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.20
+FROM golang:1.20.4
 LABEL maintainer="Marcin Wielgus <[email protected]>"
 
 ENV GOPATH /gopath/

charts/cluster-autoscaler/Chart.yaml

@@ -1,5 +1,5 @@
 apiVersion: v2
-appVersion: 1.26.2
+appVersion: 1.27.2
 description: Scales Kubernetes worker nodes within autoscaling groups.
 engine: gotpl
 home: https://github.com/kubernetes/autoscaler
@@ -11,4 +11,4 @@ name: cluster-autoscaler
 sources:
   - https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
 type: application
-version: 9.28.0
+version: 9.29.2

charts/cluster-autoscaler/README.md

@@ -320,7 +320,10 @@ Though enough for the majority of installations, the default PodSecurityPolicy _
 
 ### VerticalPodAutoscaler
 
-The chart can install a [`VerticalPodAutoscaler`](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/README.md) for the Deployment if needed. A VPA can help minimize wasted resources when usage spikes periodically or remediate containers that are being OOMKilled.
+The CA Helm Chart can install a [`VerticalPodAutoscaler`](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/README.md) object from Chart version `9.27.0`
+onwards for the Cluster Autoscaler Deployment to scale the CA as appropriate, but for that, we
+need to install the VPA to the cluster separately. A VPA can help minimize wasted resources
+when usage spikes periodically or remediate containers that are being OOMKilled.
 
 The following example snippet can be used to install VPA that allows scaling down from the default recommendations of the deployment template:
 
@@ -383,7 +386,7 @@ vpa:
 | image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
 | image.pullSecrets | list | `[]` | Image pull secrets |
 | image.repository | string | `"registry.k8s.io/autoscaling/cluster-autoscaler"` | Image repository |
-| image.tag | string | `"v1.26.2"` | Image tag |
+| image.tag | string | `"v1.27.2"` | Image tag |
 | kubeTargetVersionOverride | string | `""` | Allow overriding the `.Capabilities.KubeVersion.GitVersion` check. Useful for `helm template` commands. |
 | magnumCABundlePath | string | `"/etc/kubernetes/ca-bundle.crt"` | Path to the host's CA bundle, from `ca-file` in the cloud-config file. |
 | magnumClusterName | string | `""` | Cluster name or ID in Magnum. Required if `cloudProvider=magnum` and not setting `autoDiscovery.clusterName`. |
@@ -408,6 +411,7 @@ vpa:
 | rbac.serviceAccount.name | string | `""` | The name of the ServiceAccount to use. If not set and create is `true`, a name is generated using the fullname template. |
 | replicaCount | int | `1` | Desired number of pods |
 | resources | object | `{}` | Pod resource requests and limits. |
+| secretKeyRefNameOverride | string | `""` | Overrides the name of the Secret to use when loading the secretKeyRef for AWS and Azure env variables |
 | securityContext | object | `{}` | [Security context for pod](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) |
 | service.annotations | object | `{}` | Annotations to add to service |
 | service.create | bool | `true` | If `true`, a Service will be created. |

charts/cluster-autoscaler/README.md.gotmpl

@@ -320,7 +320,10 @@ Though enough for the majority of installations, the default PodSecurityPolicy _
 
 ### VerticalPodAutoscaler
 
-The chart can install a [`VerticalPodAutoscaler`](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/README.md) for the Deployment if needed. A VPA can help minimize wasted resources when usage spikes periodically or remediate containers that are being OOMKilled.
+The CA Helm Chart can install a [`VerticalPodAutoscaler`](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/README.md) object from Chart version `9.27.0`
+onwards for the Cluster Autoscaler Deployment to scale the CA as appropriate, but for that, we
+need to install the VPA to the cluster separately. A VPA can help minimize wasted resources
+when usage spikes periodically or remediate containers that are being OOMKilled.
 
 The following example snippet can be used to install VPA that allows scaling down from the default recommendations of the deployment template:
 

charts/cluster-autoscaler/templates/clusterrole.yaml

@@ -151,13 +151,22 @@ rules:
     - cluster.x-k8s.io
     resources:
     - machinedeployments
-    - machinedeployments/scale
+    - machinepools
     - machines
     - machinesets
     verbs:
     - get
     - list
     - update
     - watch
+  - apiGroups:
+    - cluster.x-k8s.io
+    resources:
+    - machinedeployments/scale
+    - machinepools/scale
+    verbs:
+    - get
+    - patch
+    - update
 {{- end }}
 {{- end -}}

charts/cluster-autoscaler/templates/deployment.yaml

@@ -132,36 +132,36 @@ spec:
               valueFrom:
                 secretKeyRef:
                   key: AwsAccessKeyId
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- end }}
             {{- if .Values.awsSecretAccessKey }}
             - name: AWS_SECRET_ACCESS_KEY
               valueFrom:
                 secretKeyRef:
                   key: AwsSecretAccessKey
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- end }}
           {{- else if eq .Values.cloudProvider "azure" }}
             - name: ARM_SUBSCRIPTION_ID
               valueFrom:
                 secretKeyRef:
                   key: SubscriptionID
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_RESOURCE_GROUP
               valueFrom:
                 secretKeyRef:
                   key: ResourceGroup
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_VM_TYPE
               valueFrom:
                 secretKeyRef:
                   key: VMType
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: AZURE_CLUSTER_NAME
               valueFrom:
                 secretKeyRef:
                   key: ClusterName
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- if .Values.azureUseWorkloadIdentityExtension }}
             - name: ARM_USE_WORKLOAD_IDENTITY_EXTENSION
               value: "true"
@@ -173,22 +173,22 @@ spec:
               valueFrom:
                 secretKeyRef:
                   key: TenantID
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_CLIENT_ID
               valueFrom:
                 secretKeyRef:
                   key: ClientID
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_CLIENT_SECRET
               valueFrom:
                 secretKeyRef:
                   key: ClientSecret
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: AZURE_NODE_RESOURCE_GROUP
               valueFrom:
                 secretKeyRef:
                   key: NodeResourceGroup
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- end }}
           {{- end }}
           {{- range $key, $value := .Values.extraEnv }}