diff --git a/content/en/docs/concepts/cluster-administration/compatibility-version.md b/content/en/docs/concepts/cluster-administration/compatibility-version.md new file mode 100644 index 0000000000000..951eaa61f2f6d --- /dev/null +++ b/content/en/docs/concepts/cluster-administration/compatibility-version.md @@ -0,0 +1,26 @@ +--- +title: Compatibility Version For Kubernetes Control Plane Components +reviewers: +- jpbetz +- siyuanfoundation +content_type: concept +weight: 70 +--- + + + +Since release v1.32, we introduced configurable version compatibility and emulation options to Kubernetes control plane components to make upgrades safer by providing more control and increasing the granularity of steps available to cluster administrators. + + + +## Emulated Version + +The emulation option is set by the `--emulated-version` flag of control plane components. It allows the component to emulate the behavior (APIs, features, ...) of an earlier version of Kubernetes. + +When used, the capabilities available will match the emulated version: +* Any capabilities present in the binary version that were introduced after the emulation version will be unavailable. +* Any capabilities removed after the emulation version will be available. + +This enables a binary from a particular Kubernetes release to emulate the behavior of a previous version with sufficient fidelity that interoperability with other system components can be defined in terms of the emulated version. + +The `--emulated-version` must be <= `binaryVersion`. See the help message of the `--emulated-version` flag for supported range of emulated versions. \ No newline at end of file diff --git a/content/en/docs/concepts/cluster-administration/logging.md b/content/en/docs/concepts/cluster-administration/logging.md index 5cc9429578bf0..e7a3724bb1cdc 100644 --- a/content/en/docs/concepts/cluster-administration/logging.md +++ b/content/en/docs/concepts/cluster-administration/logging.md @@ -75,6 +75,37 @@ appending a container name to the command, with a `-c` flag, like so: kubectl logs counter -c count ``` + +### Container log streams + +{{< feature-state feature_gate_name="PodLogsQuerySplitStreams" >}} + +As an alpha feature, the kubelet can split out the logs from the two standard streams produced +by a container: [standard output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)) +and [standard error](https://en.wikipedia.org/wiki/Standard_streams#Standard_error_(stderr)). +To use this behavior, you must enable the `PodLogsQuerySplitStreams` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +With that feature gate enabled, Kubernetes {{< skew currentVersion >}} allows access to these +log streams directly via the Pod API. You can fetch a specific stream by specifying the stream name (either `Stdout` or `Stderr`), +using the `stream` query string. You must have access to read the `log` subresource of that Pod. + +To demonstrate this feature, you can create a Pod that periodically writes text to both the standard output and error stream. + +{{% code_sample file="debug/counter-pod-err.yaml" %}} + +To run this pod, use the following command: + +```shell +kubectl apply -f https://k8s.io/examples/debug/counter-pod-err.yaml +``` + +To fetch only the stderr log stream, you can run: + +```shell +kubectl get --raw "/api/v1/namespaces/default/pods/counter-err/log?stream=Stderr" +``` + + See the [`kubectl logs` documentation](/docs/reference/generated/kubectl/kubectl-commands#logs) for more details. diff --git a/content/en/docs/concepts/cluster-administration/node-shutdown.md b/content/en/docs/concepts/cluster-administration/node-shutdown.md index 275311b9ee3e7..f3a3711238855 100644 --- a/content/en/docs/concepts/cluster-administration/node-shutdown.md +++ b/content/en/docs/concepts/cluster-administration/node-shutdown.md @@ -217,9 +217,7 @@ these pods will be stuck in terminating status on the shutdown node forever. To mitigate the above situation, a user can manually add the taint `node.kubernetes.io/out-of-service` with either `NoExecute` or `NoSchedule` effect to a Node marking it out-of-service. -If the `NodeOutOfServiceVolumeDetach`[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -is enabled on {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}, -and a Node is marked out-of-service with this taint, the pods on the node will be forcefully deleted +If a Node is marked out-of-service with this taint, the pods on the node will be forcefully deleted if there are no matching tolerations on it and volume detach operations for the pods terminating on the node will happen immediately. This allows the Pods on the out-of-service node to recover quickly on a different node. @@ -267,6 +265,28 @@ via the [Non-Graceful Node Shutdown](#non-graceful-node-shutdown) procedure ment {{< /note >}} +## Windows Graceful node shutdown {#windows-graceful-node-shutdown} + +{{< feature-state feature_gate_name="WindowsGracefulNodeShutdown" >}} + +The Windows graceful node shutdown feature depends on kubelet running as a Windows service, +it will then have a registered [service control handler](https://learn.microsoft.com/en-us/windows/win32/services/service-control-handler-function) +to delay the presshutdown event with a given duration. + +Windows graceful node shutdown is controlled with the `WindowsGracefulNodeShutdown` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +which is introduced in 1.32 as an alpha feature. + +Windows graceful node shutdown can not be cancelled. + +If Kubelet is not running as a Windows service, it will not be able to set and monitor +the [Preshutdown](https://learn.microsoft.com/en-us/windows/win32/api/winsvc/ns-winsvc-service_preshutdown_info) event, +the node will have to go through the [Non-Graceful Node Shutdown](#non-graceful-node-shutdown) procedure mentioned above. + +In the case where the Windows graceful node shutdown feature is enabled, but the kubelet is not +running as a Windows service, the kubelet will continue running instead of failing. However, +it will log an error indicating that it needs to be run as a Windows service. + ## {{% heading "whatsnext" %}} Learn more about the following: diff --git a/content/en/docs/concepts/configuration/manage-resources-containers.md b/content/en/docs/concepts/configuration/manage-resources-containers.md index 0432fc3968a92..9b358b2e241bc 100644 --- a/content/en/docs/concepts/configuration/manage-resources-containers.md +++ b/content/en/docs/concepts/configuration/manage-resources-containers.md @@ -109,6 +109,26 @@ a Pod. For a particular resource, a *Pod resource request/limit* is the sum of the resource requests/limits of that type for each container in the Pod. +## Pod-level resource specification + +{{< feature-state feature_gate_name="PodLevelResources" >}} + +Starting in Kubernetes 1.32, you can also specify resource requests and limits at +the Pod level. the Pod level. At Pod level, Kubernetes {{< skew currentVersion >}} +only supports resource requests or limits for specific resource types: `cpu` and / +or `memory`. This feature is currently in alpha and with the feature enabled, +Kubernetes allows you to declare an overall resource budget for the Pod, which is +especially helpful when dealing with a large number of containers where it can be +difficult to accurately gauge individual resource needs. Additionally, it enables +containers within a Pod to share idle resources with each other, improving resource +utilization. + +For a Pod, you can specify resource limits and requests for CPU and memory by including the following: +* `spec.resources.limits.cpu` +* `spec.resources.limits.memory` +* `spec.resources.requests.cpu` +* `spec.resources.requests.memory` + ## Resource units in Kubernetes ### CPU resource units {#meaning-of-cpu} @@ -192,6 +212,19 @@ spec: cpu: "500m" ``` +## Pod resources example {#example-2} + +{{< feature-state feature_gate_name="PodLevelResources" >}} + +The following Pod has an explicit request of 1 CPU and 100 MiB of memory, and an +explicit limit of 1 CPU and 200 MiB of memory. The `pod-resources-demo-ctr-1` +container has explicit requests and limits set. However, the +`pod-resources-demo-ctr-2` container will simply share the resources available +within the Pod resource boundaries, as it does not have explicit requests and limits +set. + +{{% code_sample file="pods/resource/pod-level-resources.yaml" %}} + ## How Pods with resource requests are scheduled When you create a Pod, the Kubernetes scheduler selects a node for the Pod to diff --git a/content/en/docs/concepts/configuration/secret.md b/content/en/docs/concepts/configuration/secret.md index 79598b84d82ca..1ecb0890aee7c 100644 --- a/content/en/docs/concepts/configuration/secret.md +++ b/content/en/docs/concepts/configuration/secret.md @@ -666,10 +666,7 @@ Therefore, one Pod does not have access to the Secrets of another Pod. ### Configure least-privilege access to Secrets -To enhance the security measures around Secrets, Kubernetes provides a mechanism: you can -annotate a ServiceAccount as `kubernetes.io/enforce-mountable-secrets: "true"`. - -For more information, you can refer to the [documentation about this annotation](/docs/concepts/security/service-accounts/#enforce-mountable-secrets). +To enhance the security measures around Secrets, use separate namespaces to isolate access to mounted secrets. {{< warning >}} Any containers that run with `privileged: true` on a node can access all diff --git a/content/en/docs/concepts/containers/container-lifecycle-hooks.md b/content/en/docs/concepts/containers/container-lifecycle-hooks.md index 9b6b37263cd6a..37b3171af643e 100644 --- a/content/en/docs/concepts/containers/container-lifecycle-hooks.md +++ b/content/en/docs/concepts/containers/container-lifecycle-hooks.md @@ -58,6 +58,10 @@ Resources consumed by the command are counted against the Container. * Sleep - Pauses the container for a specified duration. This is a beta-level feature default enabled by the `PodLifecycleSleepAction` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +{{< note >}} +Enable the `PodLifecycleSleepActionAllowZero` feature gate if you want to set a sleep duration of zero seconds (effectively a no-op) for your Sleep lifecycle hooks. +{{< /note >}} + ### Hook handler execution When a Container lifecycle management hook is called, diff --git a/content/en/docs/concepts/containers/images.md b/content/en/docs/concepts/containers/images.md index fb7113a3ce076..6ea5d0d62202f 100644 --- a/content/en/docs/concepts/containers/images.md +++ b/content/en/docs/concepts/containers/images.md @@ -214,7 +214,7 @@ behalf of the two different Pods, when parallel image pulls is enabled. ### Maximum parallel image pulls -{{< feature-state for_k8s_version="v1.27" state="alpha" >}} +{{< feature-state for_k8s_version="v1.32" state="beta" >}} When `serializeImagePulls` is set to false, the kubelet defaults to no limit on the maximum number of images being pulled at the same time. If you would like to diff --git a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md index 9411e3c63e304..f8ecb35b4319c 100644 --- a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md +++ b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md @@ -316,9 +316,8 @@ may also be used with field selectors when included in the `spec.versions[*].sel {{< feature-state feature_gate_name="CustomResourceFieldSelectors" >}} The `spec.versions[*].selectableFields` field of a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} may be used to -declare which other fields in a custom resource may be used in field selectors -with the feature of `CustomResourceFieldSelectors` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) (This feature gate is enabled by default since Kubernetes v1.31). +declare which other fields in a custom resource may be used in field selectors. + The following example adds the `.spec.color` and `.spec.size` fields as selectable fields. diff --git a/content/en/docs/concepts/overview/working-with-objects/field-selectors.md b/content/en/docs/concepts/overview/working-with-objects/field-selectors.md index 25ecf6987afb9..b59ad835d303d 100644 --- a/content/en/docs/concepts/overview/working-with-objects/field-selectors.md +++ b/content/en/docs/concepts/overview/working-with-objects/field-selectors.md @@ -46,6 +46,14 @@ Error from server (BadRequest): Unable to find "ingresses" that match label sele | Node | `spec.unschedulable` | | CertificateSigningRequest | `spec.signerName` | +### Custom resources fields + +All custom resource types support the `metadata.name` and `metadata.namespace` fields. + +Additionally, the `spec.versions[*].selectableFields` field of a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} +declares which other fields in a custom resource may be used in field selectors. See [selectable fields for custom resources](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#crd-selectable-fields) +for more information about how to use field selectors with CustomResourceDefinitions. + ## Supported operators You can use the `=`, `==`, and `!=` operators with field selectors (`=` and `==` mean the same thing). This `kubectl` command, for example, selects all Kubernetes Services that aren't in the `default` namespace: @@ -72,4 +80,4 @@ You can use field selectors across multiple resource types. This `kubectl` comma ```shell kubectl get statefulsets,services --all-namespaces --field-selector metadata.namespace!=default -``` \ No newline at end of file +``` diff --git a/content/en/docs/concepts/overview/working-with-objects/names.md b/content/en/docs/concepts/overview/working-with-objects/names.md index f8bcf56335667..2634f18d42741 100644 --- a/content/en/docs/concepts/overview/working-with-objects/names.md +++ b/content/en/docs/concepts/overview/working-with-objects/names.md @@ -32,6 +32,12 @@ of the same resource. API resources are distinguished by their API group, resour In cases when objects represent a physical entity, like a Node representing a physical host, when the host is re-created under the same name without deleting and re-creating the Node, Kubernetes treats the new host as the old one, which may lead to inconsistencies. {{< /note >}} +The server may generate a name when `generateName` is provided instead of `name` in a resource create request. +When `generateName` is used, the provided value is used as a name prefix, which server appends a generated suffix +to. Even though the name is generated, it may conflict with existing names resulting in a HTTP 409 resopnse. This +became far less likely to happen in Kubernetes v1.31 and later, since the server will make up to 8 attempt to generate a +unique name before returning a HTTP 409 response. + Below are four types of commonly used name constraints for resources. ### DNS Subdomain Names diff --git a/content/en/docs/concepts/policy/node-resource-managers.md b/content/en/docs/concepts/policy/node-resource-managers.md index 719e8b1151f0e..3c26ef3cecc55 100644 --- a/content/en/docs/concepts/policy/node-resource-managers.md +++ b/content/en/docs/concepts/policy/node-resource-managers.md @@ -13,10 +13,304 @@ In order to support latency-critical and high-throughput workloads, Kubernetes o -The main manager, the Topology Manager, is a Kubelet component that co-ordinates the overall resource management process through its [policy](/docs/tasks/administer-cluster/topology-manager/). +## Hardware topology alignment policies + +_Topology Manager_ is a kubelet component that aims to coordinate the set of components that are +responsible for these optimizations. The the overall resource management process is governed using +the policy you specify. +To learn more, read [Control Topology Management Policies on a Node](/docs/tasks/administer-cluster/topology-manager/). + +## Policies for assigning CPUs to Pods + +{{< feature-state feature_gate_name="CPUManager" >}} + +Once a Pod is bound to a Node, the kubelet on that node may need to either multiplex the existing +hardware (for example, sharing CPUs across multiple Pods) or allocate hardware by dedicating some +resource (for example, assigning one of more CPUs for a Pod's exclusive use). + +By default, the kubelet uses [CFS quota](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler) +to enforce pod CPU limits.  When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on +whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus +work fine without any intervention. + +However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU +management policies to determine some placement preferences on the node. +This is implemented using the _CPU Manager_ and its policy. +There are two available policies: + +- `none`: the `none` policy explicitly enables the existing default CPU +affinity scheme, providing no affinity beyond what the OS scheduler does +automatically.  Limits on CPU usage for +[Guaranteed pods](/docs/concepts/workloads/pods/pod-qos/) and +[Burstable pods](/docs/concepts/workloads/pods/pod-qos/) +are enforced using CFS quota. +- `static`: the `static` policy allows containers in `Guaranteed` pods with integer CPU +`requests` access to exclusive CPUs on the node. This exclusivity is enforced +using the [cpuset cgroup controller](https://www.kernel.org/doc/Documentation/cgroup-v2.txt). + +{{< note >}} +System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods. +{{< /note >}} + +CPU Manager doesn't support offlining and onlining of CPUs at runtime. + +### Static policy + +The static policy enables finer-grained CPU management and exclusive CPU assignment. +This policy manages a shared pool of CPUs that initially contains all CPUs in the +node. The amount of exclusively allocatable CPUs is equal to the total +number of CPUs in the node minus any CPU reservations set by the kubelet configuration. +CPUs reserved by these options are taken, in integer quantity, from the initial shared pool in ascending order by physical +core ID.  This shared pool is the set of CPUs on which any containers in +`BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional +CPU `requests` also run on CPUs in the shared pool. Only containers that are +both part of a `Guaranteed` pod and have integer CPU `requests` are assigned +exclusive CPUs. + +{{< note >}} +The kubelet requires a CPU reservation greater than zero when the static policy is enabled. +This is because zero CPU reservation would allow the shared pool to become empty. +{{< /note >}} + +As `Guaranteed` pods whose containers fit the requirements for being statically +assigned are scheduled to the node, CPUs are removed from the shared pool and +placed in the cpuset for the container. CFS quota is not used to bound +the CPU usage of these containers as their usage is bound by the scheduling domain +itself. In others words, the number of CPUs in the container cpuset is equal to the integer +CPU `limit` specified in the pod spec. This static assignment increases CPU +affinity and decreases context switches due to throttling for the CPU-bound +workload. + +Consider the containers in the following pod specs: + +```yaml +spec: + containers: + - name: nginx + image: nginx +``` + +The pod above runs in the `BestEffort` QoS class because no resource `requests` or +`limits` are specified. It runs in the shared pool. + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + requests: + memory: "100Mi" +``` + +The pod above runs in the `Burstable` QoS class because resource `requests` do not +equal `limits` and the `cpu` quantity is not specified. It runs in the shared +pool. + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" + requests: + memory: "100Mi" + cpu: "1" +``` + +The pod above runs in the `Burstable` QoS class because resource `requests` do not +equal `limits`. It runs in the shared pool. + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" + requests: + memory: "200Mi" + cpu: "2" +``` + +The pod above runs in the `Guaranteed` QoS class because `requests` are equal to `limits`. +And the container's resource limit for the CPU resource is an integer greater than +or equal to one. The `nginx` container is granted 2 exclusive CPUs. + + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "1.5" + requests: + memory: "200Mi" + cpu: "1.5" +``` + +The pod above runs in the `Guaranteed` QoS class because `requests` are equal to `limits`. +But the container's resource limit for the CPU resource is a fraction. It runs in +the shared pool. + + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" +``` + +The pod above runs in the `Guaranteed` QoS class because only `limits` are specified +and `requests` are set equal to `limits` when not explicitly specified. And the +container's resource limit for the CPU resource is an integer greater than or +equal to one. The `nginx` container is granted 2 exclusive CPUs. + +#### Static policy options {#cpu-policy-static--options} + +The behavior of the static policy can be fine-tuned using the CPU Manager policy options. +The following policy options exist for the static CPU management policy: +{{/* options in alphabetical order */}} + +`align-by-socket` (alpha, hidden by default) +: Align CPUs by physical package / socket boundary, rather than logical NUMA boundaries (available since Kubernetes v1.25) +`distribute-cpus-across-cores` (alpha, hidden by default) +: Allocate virtual cores, sometimes called hardware threads, across different physical cores (available since Kubernetes v1.31) +`distribute-cpus-across-numa` (alpha, hidden by default) +: Spread CPUs across different NUMA domains, aiming for an even balance between the selected domains (available since Kubernetes v1.23) +`full-pcpus-only` (beta, visible by default) +: Always allocate full physical cores (available since Kubernetes v1.22) +`strict-cpu-reservation` (alpha, hidden by default) +: Prevent all the pods regardless of their Quality of Service class to run on reserved CPUs (available since Kubernetes v1.32) +`prefer-align-cpus-by-uncorecache` (alpha, hidden by default) +: Align CPUs by uncore (Last-Level) cache boundary on a best-effort way (available since Kubernetes v1.32) + +You can toggle groups of options on and off based upon their maturity level +using the following feature gates: +* `CPUManagerPolicyBetaOptions` (default enabled). Disable to hide beta-level options. +* `CPUManagerPolicyAlphaOptions` (default disabled). Enable to show alpha-level options. +You will still have to enable each option using the `cpuManagerPolicyOptions` field in the +kubelet configuration file. + +For more detail about the individual options you can configure, read on. + +##### `full-pcpus-only` + +If the `full-pcpus-only` policy option is specified, the static policy will always allocate full physical cores. +By default, without this option, the static policy allocates CPUs using a topology-aware best-fit allocation. +On SMT enabled systems, the policy can allocate individual virtual cores, which correspond to hardware threads. +This can lead to different containers sharing the same physical cores; this behaviour in turn contributes +to the [noisy neighbours problem](https://en.wikipedia.org/wiki/Cloud_computing_issues#Performance_interference_and_noisy_neighbors). +With the option enabled, the pod will be admitted by the kubelet only if the CPU request of all its containers +can be fulfilled by allocating full physical cores. +If the pod does not pass the admission, it will be put in Failed state with the message `SMTAlignmentError`. + +##### `distribute-cpus-across-numa` + +If the `distribute-cpus-across-numa`policy option is specified, the static +policy will evenly distribute CPUs across NUMA nodes in cases where more than +one NUMA node is required to satisfy the allocation. +By default, the `CPUManager` will pack CPUs onto one NUMA node until it is +filled, with any remaining CPUs simply spilling over to the next NUMA node. +This can cause undesired bottlenecks in parallel code relying on barriers (and +similar synchronization primitives), as this type of code tends to run only as +fast as its slowest worker (which is slowed down by the fact that fewer CPUs +are available on at least one NUMA node). +By distributing CPUs evenly across NUMA nodes, application developers can more +easily ensure that no single worker suffers from NUMA effects more than any +other, improving the overall performance of these types of applications. + +##### `align-by-socket` + +If the `align-by-socket` policy option is specified, CPUs will be considered +aligned at the socket boundary when deciding how to allocate CPUs to a +container. By default, the `CPUManager` aligns CPU allocations at the NUMA +boundary, which could result in performance degradation if CPUs need to be +pulled from more than one NUMA node to satisfy the allocation. Although it +tries to ensure that all CPUs are allocated from the _minimum_ number of NUMA +nodes, there is no guarantee that those NUMA nodes will be on the same socket. +By directing the `CPUManager` to explicitly align CPUs at the socket boundary +rather than the NUMA boundary, we are able to avoid such issues. Note, this +policy option is not compatible with `TopologyManager` `single-numa-node` +policy and does not apply to hardware where the number of sockets is greater +than number of NUMA nodes. + +##### `distribute-cpus-across-cores` + +If the `distribute-cpus-across-cores` policy option is specified, the static policy +will attempt to allocate virtual cores (hardware threads) across different physical cores. +By default, the `CPUManager` tends to pack cpus onto as few physical cores as possible, +which can lead to contention among cpus on the same physical core and result +in performance bottlenecks. By enabling the `distribute-cpus-across-cores` policy, +the static policy ensures that cpus are distributed across as many physical cores +as possible, reducing the contention on the same physical core and thereby +improving overall performance. However, it is important to note that this strategy +might be less effective when the system is heavily loaded. Under such conditions, +the benefit of reducing contention diminishes. Conversely, default behavior +can help in reducing inter-core communication overhead, potentially providing +better performance under high load conditions. + +##### `strict-cpu-reservation` + +The `reservedSystemCPUs` parameter in [KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/), +or the deprecated kubelet command line option `--reserved-cpus`, defines an explicit CPU set for OS system daemons +and kubernetes system daemons. More details of this parameter can be found on the +[Explicitly Reserved CPU List](/docs/tasks/administer-cluster/reserve-compute-resources/#explicitly-reserved-cpu-list) page. +By default this isolation is implemented only for guaranteed pods with integer CPU requests not for burstable and best-effort pods +(and guaranteed pods with fractional CPU requests). Admission is only comparing the cpu requests against the allocatable cpus. +Since the cpu limit is higher than the request, the default behaviour allows burstable and best-effort pods to use up the capacity +of `reservedSystemCPUs` and cause host OS services to starve in real life deployments. +If the `strict-cpu-reservation` policy option is enabled, the static policy will not allow +any workload to use the CPU cores specified in `reservedSystemCPUs`. + +##### `prefer-align-cpus-by-uncorecache` + +If the `prefer-align-cpus-by-uncorecache` policy is specified, the static policy +will allocate CPU resources for individual containers such that all CPUs assigned +to a container share the same uncore cache block (also known as the Last-Level Cache +or LLC). By default, the `CPUManager` will tightly pack CPU assignments which can +result in containers being assigned CPUs from multiple uncore caches. This option +enables the `CPUManager` to allocate CPUs in a way that maximizes the efficient use +of the uncore cache. Allocation is performed on a best-effort basis, aiming to +affine as many CPUs as possible within the same uncore cache. If the container's +CPU requirement exceeds the CPU capacity of a single uncore cache, the `CPUManager` +minimizes the number of uncore caches used in order to maintain optimal uncore +cache alignment. Specific workloads can benefit in performance from the reduction +of inter-cache latency and noisy neighbors at the cache level. If the `CPUManager` +cannot align optimally while the node has sufficient resources, the container will +still be admitted using the default packed behavior. + + +## Memory Management Policies + +{{< feature-state feature_gate_name="MemoryManager" >}} + +The Kubernetes *Memory Manager* enables the feature of guaranteed memory (and hugepages) +allocation for pods in the `Guaranteed` {{< glossary_tooltip text="QoS class" term_id="qos-class" >}}. + +The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod. +The Memory Manager feeds the central manager (*Topology Manager*) with these affinity hints. +Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node. + +Moreover, the Memory Manager ensures that the memory which a pod requests +is allocated from a minimum number of NUMA nodes. + +## Other resource managers The configuration of individual managers is elaborated in dedicated documents: -- [CPU Manager Policies](/docs/tasks/administer-cluster/cpu-management-policies/) - [Device Manager](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-integration-with-the-topology-manager) -- [Memory Manager Policies](/docs/tasks/administer-cluster/memory-manager/) diff --git a/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md b/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md index a598eee3475c8..94d308475738f 100644 --- a/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md +++ b/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md @@ -5,18 +5,21 @@ reviewers: title: Dynamic Resource Allocation content_type: concept weight: 65 +api_metadata: +- apiVersion: "resource.k8s.io/v1beta1" + kind: "ResourceClaim" +- apiVersion: "resource.k8s.io/v1beta1" + kind: "ResourceClaimTemplate" +- apiVersion: "resource.k8s.io/v1beta1" + kind: "DeviceClass" +- apiVersion: "resource.k8s.io/v1beta1" + kind: "ResourceSlice" --- -Core Dynamic Resource Allocation with structured parameters: - {{< feature-state feature_gate_name="DynamicResourceAllocation" >}} -Dynamic Resource Allocation with control plane controller: - -{{< feature-state feature_gate_name="DRAControlPlaneController" >}} - Dynamic resource allocation is an API for requesting and sharing resources between pods and containers inside a pod. It is a generalization of the persistent volumes API for generic resources. Typically those resources @@ -28,8 +31,10 @@ resources handled by Kubernetes via _structured parameters_ (introduced in Kuber Different kinds of resources support arbitrary parameters for defining requirements and initialization. -When a driver provides a _control plane controller_, the driver itself -handles allocation in cooperation with the Kubernetes scheduler. +Kubernetes v1.26 through to 1.31 included an (alpha) implementation of _classic DRA_, +which is no longer supported. This documentation, which is for Kubernetes +v{{< skew currentVersion >}}, explains the current approach to dynamic resource +allocation within Kubernetes. ## {{% heading "prerequisites" %}} @@ -43,7 +48,7 @@ v{{< skew currentVersion>}}, check the documentation for that version of Kuberne ## API -The `resource.k8s.io/v1alpha3` +The `resource.k8s.io/v1beta1` {{< glossary_tooltip text="API group" term_id="api-group" >}} provides these types: ResourceClaim @@ -65,25 +70,14 @@ DeviceClass when installing a resource driver. Each request to allocate a device in a ResourceClaim must reference exactly one DeviceClass. -PodSchedulingContext -: Used internally by the control plane and resource drivers - to coordinate pod scheduling when ResourceClaims need to be allocated - for a Pod and those ResourceClaims use a control plane controller. - ResourceSlice -: Used with structured parameters to publish information about resources +: Used by DRA drivers to publish information about resources that are available in the cluster. -The developer of a resource driver decides whether they want to handle -allocation themselves with a control plane controller or instead rely on allocation -through Kubernetes with structured parameters. A -custom controller provides more flexibility, but cluster autoscaling is not -going to work reliably for node-local resources. Structured parameters enable -cluster autoscaling, but might not satisfy all use-cases. - -When a driver uses structured parameters, all parameters that select devices -are defined in the ResourceClaim and DeviceClass with in-tree types. Configuration -parameters can be embedded there as arbitrary JSON objects. +All parameters that select devices are defined in the ResourceClaim and +DeviceClass with in-tree types. Configuration parameters can be embedded there. +Which configuration parameters are valid depends on the DRA driver -- Kubernetes +only passes them through without interpreting them. The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a `resourceClaims` field. Entries in that list reference either a ResourceClaim @@ -100,7 +94,7 @@ Here is an example for a fictional resource driver. Two ResourceClaim objects will get created for this Pod and each container gets access to one of them. ```yaml -apiVersion: resource.k8s.io/v1alpha3 +apiVersion: resource.k8s.io/v1beta1 kind: DeviceClass name: resource.example.com spec: @@ -108,7 +102,7 @@ spec: - cel: expression: device.driver == "resource-driver.example.com" --- -apiVersion: resource.k8s.io/v1alpha2 +apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaimTemplate metadata: name: large-black-cat-claim-template @@ -151,51 +145,7 @@ spec: ## Scheduling -### With control plane controller - -In contrast to native resources (CPU, RAM) and extended resources (managed by a -device plugin, advertised by kubelet), without structured parameters -the scheduler has no knowledge of what -dynamic resources are available in a cluster or how they could be split up to -satisfy the requirements of a specific ResourceClaim. Resource drivers are -responsible for that. They mark ResourceClaims as "allocated" once resources -for it are reserved. This also then tells the scheduler where in the cluster a -ResourceClaim is available. - -When a pod gets scheduled, the scheduler checks all ResourceClaims needed by a Pod and -creates a PodScheduling object where it informs the resource drivers -responsible for those ResourceClaims about nodes that the scheduler considers -suitable for the Pod. The resource drivers respond by excluding nodes that -don't have enough of the driver's resources left. Once the scheduler has that -information, it selects one node and stores that choice in the PodScheduling -object. The resource drivers then allocate their ResourceClaims so that the -resources will be available on that node. Once that is complete, the Pod -gets scheduled. - -As part of this process, ResourceClaims also get reserved for the -Pod. Currently ResourceClaims can either be used exclusively by a single Pod or -an unlimited number of Pods. - -One key feature is that Pods do not get scheduled to a node unless all of -their resources are allocated and reserved. This avoids the scenario where a Pod -gets scheduled onto one node and then cannot run there, which is bad because -such a pending Pod also blocks all other resources like RAM or CPU that were -set aside for it. - -{{< note >}} - -Scheduling of pods which use ResourceClaims is going to be slower because of -the additional communication that is required. Beware that this may also impact -pods that don't use ResourceClaims because only one pod at a time gets -scheduled, blocking API calls are made while handling a pod with -ResourceClaims, and thus scheduling the next pod gets delayed. - -{{< /note >}} - -### With structured parameters - -When a driver uses structured parameters, the scheduler takes over the -responsibility of allocating resources to a ResourceClaim whenever a pod needs +The scheduler is responsible for allocating resources to a ResourceClaim whenever a pod needs them. It does so by retrieving the full list of available resources from ResourceSlice objects, tracking which of those resources have already been allocated to existing ResourceClaims, and then selecting from those resources @@ -235,14 +185,9 @@ later. Such a situation can also arise when support for dynamic resource allocation was not enabled in the scheduler at the time when the Pod got scheduled (version skew, configuration, feature gate, etc.). kube-controller-manager -detects this and tries to make the Pod runnable by triggering allocation and/or -reserving the required ResourceClaims. - -{{< note >}} - -This only works with resource drivers that don't use structured parameters. - -{{< /note >}} +detects this and tries to make the Pod runnable by reserving the required +ResourceClaims. However, this only works if those were allocated by +the scheduler for some other pod. It is better to avoid bypassing the scheduler because a Pod that is assigned to a node blocks normal resources (RAM, CPU) that then cannot be used for other Pods @@ -264,17 +209,132 @@ spec: You may also be able to mutate the incoming Pod, at admission time, to unset the `.spec.nodeName` field and to use a node selector instead. +## Admin access + +{{< feature-state feature_gate_name="DRAAdminAccess" >}} + +You can mark a request in a ResourceClaim or ResourceClaimTemplate as having privileged features. +A request with admin access grants access to devices which are in use and +may enable additional permissions when making the device available in a +container: + +```yaml +apiVersion: resource.k8s.io/v1beta1 +kind: ResourceClaimTemplate +metadata: + name: large-black-cat-claim-template +spec: + spec: + devices: + requests: + - name: req-0 + deviceClassName: resource.example.com + adminAccess: true +``` + +If this feature is disabled, the `adminAccess` field will be removed +automatically when creating such a ResourceClaim. + +Admin access is a privileged mode which should not be made available to normal +users in a multi-tenant cluster. Cluster administrators can restrict usage of +this feature by installing a validating admission policy similar to the following +example. Cluster administrators need to adapt at least the names and replace +"dra.example.com". + +```yaml +# Permission to use admin access is granted only in namespaces which have the +# "admin-access.dra.example.com" label. Other ways of making that decision are +# also possible. + +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: resourceclaim-policy.dra.example.com +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: ["resource.k8s.io"] + apiVersions: ["v1alpha3", "v1beta1"] + operations: ["CREATE", "UPDATE"] + resources: ["resourceclaims"] + validations: + - expression: '! object.spec.devices.requests.exists(e, has(e.adminAccess) && e.adminAccess)' + reason: Forbidden + messageExpression: '"admin access to devices not enabled"' +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: resourceclaim-binding.dra.example.com +spec: + policyName: resourceclaim-policy.dra.example.com + validationActions: [Deny] + matchResources: + namespaceSelector: + matchExpressions: + - key: admin-access.dra.example.com + operator: DoesNotExist +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: resourceclaimtemplate-policy.dra.example.com +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: ["resource.k8s.io"] + apiVersions: ["v1alpha3", "v1beta1"] + operations: ["CREATE", "UPDATE"] + resources: ["resourceclaimtemplates"] + validations: + - expression: '! object.spec.spec.devices.requests.exists(e, has(e.adminAccess) && e.adminAccess)' + reason: Forbidden + messageExpression: '"admin access to devices not enabled"' +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: resourceclaimtemplate-binding.dra.example.com +spec: + policyName: resourceclaimtemplate-policy.dra.example.com + validationActions: [Deny] + matchResources: + namespaceSelector: + matchExpressions: + - key: admin-access.dra.example.com + operator: DoesNotExist +``` + +## ResourceClaim Device Status + +{{< feature-state feature_gate_name="DRAResourceClaimDeviceStatus" >}} + +The drivers can report driver-specific device status data for each allocated device +in a resource claim. For example, IPs assigned to a network interface device can be +reported in the ResourceClaim status. + +The drivers setting the status, the accuracy of the information depends on the implementation +of those DRA Drivers. Therefore, the reported status of the device may not always reflect the +real time changes of the state of the device. + +When the feature is disabled, that field automatically gets cleared when storing the ResourceClaim. + +A ResourceClaim device status is supported when it is possible, from a DRA driver, to update an +existing ResourceClaim where the `status.devices` field is set. + ## Enabling dynamic resource allocation -Dynamic resource allocation is an *alpha feature* and only enabled when the +Dynamic resource allocation is a *beta feature* which is off by default and only enabled when the `DynamicResourceAllocation` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -and the `resource.k8s.io/v1alpha3` {{< glossary_tooltip text="API group" term_id="api-group" >}} +and the `resource.k8s.io/v1beta1` {{< glossary_tooltip text="API group" term_id="api-group" >}} are enabled. For details on that, see the `--feature-gates` and `--runtime-config` [kube-apiserver parameters](/docs/reference/command-line-tools-reference/kube-apiserver/). kube-scheduler, kube-controller-manager and kubelet also need the feature gate. -When a resource driver uses a control plane controller, then the -`DRAControlPlaneController` feature gate has to be enabled in addition to +When a resource driver reports the status of the devices, then the +`DRAResourceClaimDeviceStatus` feature gate has to be enabled in addition to `DynamicResourceAllocation`. A quick check whether a Kubernetes cluster supports the feature is to list @@ -297,11 +357,6 @@ If not supported, this error is printed instead: error: the server doesn't have a resource type "deviceclasses" ``` -A control plane controller is supported when it is possible to create a -ResourceClaim where the `spec.controller` field is set. When the -`DRAControlPlaneController` feature is disabled, that field automatically -gets cleared when storing the ResourceClaim. - The default configuration of kube-scheduler enables the "DynamicResources" plugin if and only if the feature gate is enabled and when using the v1 configuration API. Custom configurations may have to be modified to @@ -310,9 +365,21 @@ include it. In addition to enabling the feature in the cluster, a resource driver also has to be installed. Please refer to the driver's documentation for details. +### Enabling admin access + +[Admin access](#admin-access) is an *alpha feature* and only enabled when the +`DRAAdminAccess` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +is enabled in the kube-apiserver and kube-scheduler. + +### Enabling Device Status + +[ResourceClaim Device Status](#resourceclaim-device-status) is an *alpha feature* +and only enabled when the `DRAResourceClaimDeviceStatus` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +is enabled in the kube-apiserver. + ## {{% heading "whatsnext" %}} - For more information on the design, see the [Dynamic Resource Allocation with Structured Parameters](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4381-dra-structured-parameters) - and the - [Dynamic Resource Allocation with Control Plane Controller](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3063-dynamic-resource-allocation/README.md) KEPs. + KEP. diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index 63a8c7d3e6fec..d76ad3e1d0809 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -74,7 +74,7 @@ Plugins that implement PreEnqueue, PreFilter, Filter, Reserve or Permit should i ### QueueingHint -{{< feature-state for_k8s_version="v1.28" state="beta" >}} +{{< feature-state for_k8s_version="v1.32" state="beta" >}} QueueingHint is a callback function for deciding whether a Pod can be requeued to the active queue or backoff queue. It's executed every time a certain kind of event or change happens in the cluster. @@ -84,11 +84,8 @@ so that the scheduler will retry the scheduling of the Pod. {{< note >}} QueueingHint evaluation during scheduling is a beta-level feature. -The v1.28 release series initially enabled the associated feature gate; however, after the -discovery of an excessive memory footprint, the Kubernetes project set that feature gate -to be disabled by default. In Kubernetes {{< skew currentVersion >}}, this feature gate is -disabled and you need to enable it manually. -You can enable it via the +In Kubernetes {{< skew currentVersion >}}, this feature gate is enabled by default, +and you can disable it via the `SchedulerQueueingHints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). {{< /note >}} diff --git a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md index a20f15b9ce246..935ab1c7a9176 100644 --- a/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md +++ b/content/en/docs/concepts/scheduling-eviction/topology-spread-constraints.md @@ -99,7 +99,7 @@ your cluster. Those fields are: {{< note >}} Before Kubernetes v1.30, the `minDomains` field was only available if the - `MinDomainsInPodTopologySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) + `MinDomainsInPodTopologySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates-removed/) was enabled (default since v1.28). In older Kubernetes clusters it might be explicitly disabled or the field might not be available. {{< /note >}} diff --git a/content/en/docs/concepts/security/secrets-good-practices.md b/content/en/docs/concepts/security/secrets-good-practices.md index 3e18929f90f3b..0075fa9ebec77 100644 --- a/content/en/docs/concepts/security/secrets-good-practices.md +++ b/content/en/docs/concepts/security/secrets-good-practices.md @@ -62,11 +62,8 @@ recommendations include: * Implement audit rules that alert on specific events, such as concurrent reading of multiple Secrets by a single user -#### Additional ServiceAccount annotations for Secret management - -You can also use the `kubernetes.io/enforce-mountable-secrets` annotation on -a ServiceAccount to enforce specific rules on how Secrets are used in a Pod. -For more details, see the [documentation on this annotation](/docs/reference/labels-annotations-taints/#enforce-mountable-secrets). +#### Restrict Access for Secrets +Use separate namespaces to isolate access to mounted secrets. ### Improve etcd management policies diff --git a/content/en/docs/concepts/security/service-accounts.md b/content/en/docs/concepts/security/service-accounts.md index 95fde2f28a81f..e1ed8ac958475 100644 --- a/content/en/docs/concepts/security/service-accounts.md +++ b/content/en/docs/concepts/security/service-accounts.md @@ -197,7 +197,13 @@ or using a custom mechanism such as an [authentication webhook](/docs/reference/ You can also use TokenRequest to obtain short-lived tokens for your external application. {{< /note >}} -### Restricting access to Secrets {#enforce-mountable-secrets} +### Restricting access to Secrets (deprecated) {#enforce-mountable-secrets} + +{{< feature-state for_k8s_version="v1.32" state="deprecated" >}} + +{{< note >}} +`kubernetes.io/enforce-mountable-secrets` is deprecated since Kubernetes v1.32. Use separate namespaces to isolate access to mounted secrets. +{{< /note >}} Kubernetes provides an annotation called `kubernetes.io/enforce-mountable-secrets` that you can add to your ServiceAccounts. When this annotation is applied, diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index 8681d74a53d9b..c14e4765c7429 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -681,14 +681,11 @@ The value of `spec.loadBalancerClass` must be a label-style identifier, with an optional prefix such as "`internal-vip`" or "`example.com/internal-vip`". Unprefixed names are reserved for end-users. -#### Specifying IPMode of load balancer status {#load-balancer-ip-mode} +#### Load balancer IP address mode {#load-balancer-ip-mode} {{< feature-state feature_gate_name="LoadBalancerIPMode" >}} -As a Beta feature in Kubernetes 1.30, -a [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -named `LoadBalancerIPMode` allows you to set the `.status.loadBalancer.ingress.ipMode` -for a Service with `type` set to `LoadBalancer`. +For a Service of `type: LoadBalancer`, a controller can set `.status.loadBalancer.ingress.ipMode`. The `.status.loadBalancer.ingress.ipMode` specifies how the load-balancer IP behaves. It may be specified only when the `.status.loadBalancer.ingress.ip` field is also specified. diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index 39891d5e49959..90b79be3381f5 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -478,18 +478,17 @@ administrator intervention. {{% /tab %}} {{% tab name="By requesting expansion to smaller size" %}} -{{% feature-state for_k8s_version="v1.23" state="alpha" %}} +{{< feature-state feature_gate_name="RecoverVolumeExpansionFailure" >}} {{< note >}} -Recovery from failing PVC expansion by users is available as an alpha feature -since Kubernetes 1.23. The `RecoverVolumeExpansionFailure` feature must be -enabled for this feature to work. Refer to the +Recover from failing PVC expansion by users (`RecoverVolumeExpansionFailure`) is available as an beta feature +since Kubernetes 1.32 and should be enabled by default. Refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. {{< /note >}} -If the feature gates `RecoverVolumeExpansionFailure` is -enabled in your cluster, and expansion has failed for a PVC, you can retry expansion with a + +When using `RecoverVolumeExpansionFailure` feature, if expansion has failed for a PVC, you can retry expansion with a smaller size than the previously requested value. To request a new expansion attempt with a smaller proposed size, edit `.spec.resources` for that PVC and choose a value that is less than the value you previously tried. diff --git a/content/en/docs/concepts/storage/volume-snapshots.md b/content/en/docs/concepts/storage/volume-snapshots.md index 3f1744e108751..f2cbc90db99f7 100644 --- a/content/en/docs/concepts/storage/volume-snapshots.md +++ b/content/en/docs/concepts/storage/volume-snapshots.md @@ -65,6 +65,9 @@ Users need to be aware of the following when using this feature: the csi-snapshotter. See [CSI Driver documentation](https://kubernetes-csi.github.io/docs/) for details. - The CRDs and snapshot controller installations are the responsibility of the Kubernetes distribution. +For advanced use cases, such as creating group snapshots of multiple volumes, see the external +[CSI Volume Group Snapshot documentation](https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html). + ## Lifecycle of a volume snapshot and volume snapshot content `VolumeSnapshotContents` are resources in the cluster. `VolumeSnapshots` are requests diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index bd0f40b9556ba..b431f62eb236c 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -242,19 +242,12 @@ the `emptyDir.medium` field to `"Memory"`, Kubernetes mounts a tmpfs (RAM-backed filesystem) for you instead. While tmpfs is very fast be aware that, unlike disks, files you write count against the memory limit of the container that wrote them. - A size limit can be specified for the default medium, which limits the capacity of the `emptyDir` volume. The storage is allocated from [node ephemeral storage](/docs/concepts/configuration/manage-resources-containers/#setting-requests-and-limits-for-local-ephemeral-storage). If that is filled up from another source (for example, log files or image overlays), the `emptyDir` may run out of capacity before this limit. - -{{< note >}} -You can specify a size for memory backed volumes, provided that the `SizeMemoryBackedVolumes` -[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -is enabled in your cluster (this has been beta, and active by default, since the Kubernetes 1.22 release). -If you don't specify a volume size, memory backed volumes are sized to node allocatable memory. -{{< /note>}} +If no size is specified, memory backed volumes are sized to node allocatable memory. {{< caution >}} Please check [here](/docs/concepts/configuration/manage-resources-containers/#memory-backed-emptydir) @@ -281,6 +274,27 @@ spec: sizeLimit: 500Mi ``` +#### emptyDir memory configuration example + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: test-pd +spec: + containers: + - image: registry.k8s.io/test-webserver + name: test-container + volumeMounts: + - mountPath: /cache + name: cache-volume + volumes: + - name: cache-volume + emptyDir: + sizeLimit: 500Mi + medium: Memory +``` + ### fc (fibre channel) {#fc} An `fc` volume type allows an existing fibre channel block storage volume diff --git a/content/en/docs/concepts/workloads/autoscaling.md b/content/en/docs/concepts/workloads/autoscaling.md index ff154d71599be..f2a5f1fbe0718 100644 --- a/content/en/docs/concepts/workloads/autoscaling.md +++ b/content/en/docs/concepts/workloads/autoscaling.md @@ -79,7 +79,7 @@ Mode | Description #### Requirements for in-place resizing -{{< feature-state for_k8s_version="v1.27" state="alpha" >}} +{{< feature-state feature_gate_name="InPlacePodVerticalScaling" >}} Resizing a workload in-place **without** restarting the {{< glossary_tooltip text="Pods" term_id="pod" >}} or its {{< glossary_tooltip text="Containers" term_id="container" >}} requires Kubernetes version 1.27 or later. diff --git a/content/en/docs/concepts/workloads/controllers/cron-jobs.md b/content/en/docs/concepts/workloads/controllers/cron-jobs.md index 0af2404a54eb6..397d7a4050beb 100644 --- a/content/en/docs/concepts/workloads/controllers/cron-jobs.md +++ b/content/en/docs/concepts/workloads/controllers/cron-jobs.md @@ -216,6 +216,10 @@ are certain circumstances where two Jobs might be created, or no Job might be cr Kubernetes tries to avoid those situations, but does not completely prevent them. Therefore, the Jobs that you define should be _idempotent_. +Starting with Kubernetes v1.32, CronJobs apply an annotation +`batch.kubernetes.io/cronjob-scheduled-timestamp` to their created Jobs. This annotation +indicates the originally scheduled creation time for the Job and is formatted in RFC3339. + If `startingDeadlineSeconds` is set to a large value or left unset (the default) and if `concurrencyPolicy` is set to `Allow`, the Jobs will always run at least once. diff --git a/content/en/docs/concepts/workloads/controllers/job.md b/content/en/docs/concepts/workloads/controllers/job.md index 7cd1f5b928fa1..901660fa305ec 100644 --- a/content/en/docs/concepts/workloads/controllers/job.md +++ b/content/en/docs/concepts/workloads/controllers/job.md @@ -695,8 +695,8 @@ triggered and all Pod finalizers were removed. However, some Pods would still be running or terminating at the moment that the terminal condition was added. In Kubernetes v1.31 and later, the controller only adds the Job terminal conditions -_after_ all of the Pods are terminated. You can enable this behavior by using the -`JobManagedBy` or the `JobPodReplacementPolicy` (enabled by default) +_after_ all of the Pods are terminated. You can control this behavior by using the +`JobManagedBy` and the `JobPodReplacementPolicy` (both enabled by default) [feature gates](/docs/reference/command-line-tools-reference/feature-gates/). ### Termination of Job pods @@ -1137,7 +1137,7 @@ status: {{< note >}} You can only set the `managedBy` field on Jobs if you enable the `JobManagedBy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -(disabled by default). +(enabled by default). {{< /note >}} This feature allows you to disable the built-in Job controller, for a specific diff --git a/content/en/docs/concepts/workloads/controllers/statefulset.md b/content/en/docs/concepts/workloads/controllers/statefulset.md index 177dd3b371abd..3a79abb5d6841 100644 --- a/content/en/docs/concepts/workloads/controllers/statefulset.md +++ b/content/en/docs/concepts/workloads/controllers/statefulset.md @@ -252,13 +252,13 @@ the StatefulSet. ### Pod index label -{{< feature-state for_k8s_version="v1.28" state="beta" >}} +{{< feature-state feature_gate_name="PodIndexLabel" >}} When the StatefulSet {{}} creates a Pod, the new Pod is labelled with `apps.kubernetes.io/pod-index`. The value of this label is the ordinal index of the Pod. This label allows you to route traffic to a particular pod index, filter logs/metrics -using the pod index label, and more. Note the feature gate `PodIndexLabel` must be enabled for this -feature, and it is enabled by default. +using the pod index label, and more. Note the feature gate `PodIndexLabel` is enabled and locked by default for this +feature, in order to disable it, users will have to use server emulated version v1.31. ## Deployment and Scaling Guarantees @@ -386,7 +386,7 @@ StatefulSet will then begin to recreate the Pods using the reverted template. ## PersistentVolumeClaim retention -{{< feature-state for_k8s_version="v1.27" state="beta" >}} +{{< feature-state feature_gate_name="StatefulSetAutoDeletePVC" >}} The optional `.spec.persistentVolumeClaimRetentionPolicy` field controls if and how PVCs are deleted during the lifecycle of a StatefulSet. You must enable the @@ -493,4 +493,4 @@ the `.spec.replicas` field automatically. Read the {{< api-reference page="workload-resources/stateful-set-v1" >}} object definition to understand the API for stateful sets. * Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how - you can use it to manage application availability during disruptions. \ No newline at end of file + you can use it to manage application availability during disruptions. diff --git a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md index eef73e8a26bd8..7d08a384fcdb0 100644 --- a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md +++ b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md @@ -260,6 +260,38 @@ problems, the kubelet resets the restart backoff timer for that container. [Sidecar containers and Pod lifecycle](/docs/concepts/workloads/pods/sidecar-containers/#sidecar-containers-and-pod-lifecycle) explains the behaviour of `init containers` when specify `restartpolicy` field on it. +### Configurable container restart delay + +{{< feature-state feature_gate_name="KubeletCrashLoopBackOffMax" >}} + +With the alpha feature gate `KubeletCrashLoopBackOffMax` enabled, you can +reconfigure the maximum delay between container start retries from the default +of 300s (5 minutes). This configuration is set per node using kubelet +configuration. In your [kubelet configuration](/docs/tasks/administer-cluster/kubelet-config-file/), +under `crashLoopBackOff` set the `maxContainerRestartPeriod` field between +`"1s"` and `"300s"`. As described above in [Container restart +policy](#restart-policy), delays on that node will still start at 10s and +increase exponentially by 2x each restart, but will now be capped at your +configured maximum. If the `maxContainerRestartPeriod` you configure is less +than the default initial value of 10s, the initial delay will instead be set to +the configured maximum. + +See the following kubelet configuration examples: + +```yaml +# container restart delays will start at 10s, increasing +# 2x each time they are restarted, to a maximum of 100s +kind: KubeletConfiguration +crashLoopBackOff: + maxContainerRestartPeriod: "100s" +``` + +```yaml +# delays between container restarts will always be 2s +kind: KubeletConfiguration +crashLoopBackOff: + maxContainerRestartPeriod: "2s" +``` ## Pod conditions @@ -677,8 +709,7 @@ Additionally, PodGC cleans up any Pods which satisfy any of the following condit 1. are orphan Pods - bound to a node which no longer exists, 1. are unscheduled terminating Pods, 1. are terminating Pods, bound to a non-ready node tainted with - [`node.kubernetes.io/out-of-service`](/docs/reference/labels-annotations-taints/#node-kubernetes-io-out-of-service), - when the `NodeOutOfServiceVolumeDetach` feature gate is enabled. + [`node.kubernetes.io/out-of-service`](/docs/reference/labels-annotations-taints/#node-kubernetes-io-out-of-service). Along with cleaning up the Pods, PodGC will also mark them as failed if they are in a non-terminal phase. Also, PodGC adds a Pod disruption condition when cleaning up an orphan Pod. diff --git a/content/en/docs/reference/access-authn-authz/admission-controllers.md b/content/en/docs/reference/access-authn-authz/admission-controllers.md index de2c6f447a216..fe41eb8351724 100644 --- a/content/en/docs/reference/access-authn-authz/admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/admission-controllers.md @@ -794,9 +794,7 @@ The Kubernetes project strongly recommends enabling this admission controller. You should enable this admission controller if you intend to make any use of Kubernetes `ServiceAccount` objects. -Regarding the annotation `kubernetes.io/enforce-mountable-secrets`: While the annotation's name suggests it only concerns the mounting of Secrets, -its enforcement also extends to other ways Secrets are used in the context of a Pod. -Therefore, it is crucial to ensure that all the referenced secrets are correctly specified in the ServiceAccount. +To enhance the security measures around Secrets, use separate namespaces to isolate access to mounted secrets. ### StorageObjectInUseProtection diff --git a/content/en/docs/reference/access-authn-authz/authentication.md b/content/en/docs/reference/access-authn-authz/authentication.md index f6371199b4510..1f52aea2799c4 100644 --- a/content/en/docs/reference/access-authn-authz/authentication.md +++ b/content/en/docs/reference/access-authn-authz/authentication.md @@ -466,6 +466,12 @@ jwt: expression: 'claims.sub' # extra attributes to be added to the UserInfo object. Keys must be domain-prefix path and must be unique. extra: + # key is a string to use as the extra attribute key. + # key must be a domain-prefix path (e.g. example.org/foo). All characters before the first "/" must be a valid + # subdomain as defined by RFC 1123. All characters trailing the first "/" must + # be valid HTTP Path characters as defined by RFC 3986. + # k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use and cannot be used. + # key must be lowercase and unique across all extra attributes. - key: 'example.com/tenant' # valueExpression is a CEL expression that evaluates to a string or a list of strings. valueExpression: 'claims.tenant' @@ -1087,9 +1093,8 @@ that grant access to the `*` user or `*` group do not include anonymous users. {{< feature-state feature_gate_name="AnonymousAuthConfigurableEndpoints" >}} The `AuthenticationConfiguration` can be used to configure the anonymous -authenticator. To enable configuring anonymous auth via the config file you need -enable the `AnonymousAuthConfigurableEndpoints` feature gate. When this feature -gate is enabled you cannot set the `--anonymous-auth` flag. +authenticator. If you set the anonymous field in the `AuthenticationConfiguration` +file then you cannot set the `--anonymous-auth` flag. The main advantage of configuring anonymous authenticator using the authentication configuration file is that in addition to enabling and disabling anonymous authentication diff --git a/content/en/docs/reference/access-authn-authz/authorization.md b/content/en/docs/reference/access-authn-authz/authorization.md index d68c39765cf17..292bf6f8ae4e8 100644 --- a/content/en/docs/reference/access-authn-authz/authorization.md +++ b/content/en/docs/reference/access-authn-authz/authorization.md @@ -165,41 +165,14 @@ to the built-in `cluster-admin` ClusterRole. ### Authorization mode configuration {#choice-of-authz-config} You can configure the Kubernetes API server's authorizer chain using either -[command line arguments](#using-flags-for-your-authorization-module) only or, as a beta feature, -using a [configuration file](#using-configuration-file-for-authorization). +a [configuration file](#using-configuration-file-for-authorization) only or +[command line arguments](#using-flags-for-your-authorization-module). You have to pick one of the two configuration approaches; setting both `--authorization-config` path and configuring an authorization webhook using the `--authorization-mode` and `--authorization-webhook-*` command line arguments is not allowed. If you try this, the API server reports an error message during startup, then exits immediately. -### Command line authorization mode configuration {#using-flags-for-your-authorization-module} - -{{< feature-state state="stable" for_k8s_version="v1.8" >}} - -You can use the following modes: - -* `--authorization-mode=ABAC` (Attribute-based access control mode) -* `--authorization-mode=RBAC` (Role-based access control mode) -* `--authorization-mode=Node` (Node authorizer) -* `--authorization-mode=Webhook` (Webhook authorization mode) -* `--authorization-mode=AlwaysAllow` (always allows requests; carries [security risks](#warning-always-allow)) -* `--authorization-mode=AlwaysDeny` (always denies requests) - -You can choose more than one authorization mode; for example: -`--authorization-mode=Node,Webhook` - -Kubernetes checks authorization modules based on the order that you specify them -on the API server's command line, so an earlier module has higher priority to allow -or deny a request. - -You cannot combine the `--authorization-mode` command line argument with the -`--authorization-config` command line argument used for -[configuring authorization using a local file](#using-configuration-file-for-authorization-mode). - -For more information on command line arguments to the API server, read the -[`kube-apiserver` reference](/docs/reference/command-line-tools-reference/kube-apiserver/). - @@ -207,7 +180,7 @@ For more information on command line arguments to the API server, read the {{< feature-state feature_gate_name="StructuredAuthorizationConfiguration" >}} -As a beta feature, Kubernetes lets you configure authorization chains that can include multiple +Kubernetes lets you configure authorization chains that can include multiple webhooks. The authorization items in that chain can have well-defined parameters that validate requests in a particular order, offering you fine-grained control, such as explicit Deny on failures. @@ -230,7 +203,7 @@ are only available if you use an authorization configuration file. # # DO NOT USE THE CONFIG AS IS. THIS IS AN EXAMPLE. # -apiVersion: apiserver.config.k8s.io/v1beta1 +apiVersion: apiserver.config.k8s.io/v1 kind: AuthorizationConfiguration authorizers: - type: Webhook @@ -347,6 +320,31 @@ A reload **must not** add or remove Node or RBAC authorizers (they can be reorde but cannot be added or removed). {{< /note >}} +### Command line authorization mode configuration {#using-flags-for-your-authorization-module} + +You can use the following modes: + +* `--authorization-mode=ABAC` (Attribute-based access control mode) +* `--authorization-mode=RBAC` (Role-based access control mode) +* `--authorization-mode=Node` (Node authorizer) +* `--authorization-mode=Webhook` (Webhook authorization mode) +* `--authorization-mode=AlwaysAllow` (always allows requests; carries [security risks](#warning-always-allow)) +* `--authorization-mode=AlwaysDeny` (always denies requests) + +You can choose more than one authorization mode; for example: +`--authorization-mode=Node,RBAC,Webhook` + +Kubernetes checks authorization modules based on the order that you specify them +on the API server's command line, so an earlier module has higher priority to allow +or deny a request. + +You cannot combine the `--authorization-mode` command line argument with the +`--authorization-config` command line argument used for +[configuring authorization using a local file](#using-configuration-file-for-authorization-mode). + +For more information on command line arguments to the API server, read the +[`kube-apiserver` reference](/docs/reference/command-line-tools-reference/kube-apiserver/). + ## Privilege escalation via workload creation or edits {#privilege-escalation-via-pod-creation} Users who can create/edit pods in a namespace, either directly or through an object that diff --git a/content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md b/content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md index 87c6dde16c7e8..7de116784f675 100644 --- a/content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md +++ b/content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md @@ -85,3 +85,38 @@ flags passed to the apiserver is authorized for the following attributes: * verb=\*, resource=nodes, subresource=log * verb=\*, resource=nodes, subresource=spec * verb=\*, resource=nodes, subresource=metrics + +### Fine-grained authorization + +{{< feature-state feature_gate_name="KubeletFineGrainedAuthz" >}} + +When the feature gate `KubeletFineGrainedAuthz` is enabled kubelet performs a +fine-grained check before falling back to the `proxy` subresource for the `/pods`, +`/runningPods`, `/configz` and `/healthz` endpoints. The resource and subresource +are determined from the incoming request's path: + +Kubelet API | resource | subresource +--------------|----------|------------ +/stats/\* | nodes | stats +/metrics/\* | nodes | metrics +/logs/\* | nodes | log +/spec/\* | nodes | spec +/pods | nodes | pods, proxy +/runningPods/ | nodes | pods, proxy +/healthz | nodes | healthz, proxy +/configz | nodes | configz, proxy +*all others* | nodes | proxy + + +When the feature-gate `KubeletFineGrainedAuthz` is enabled, ensure the user +identified by the `--kubelet-client-certificate` and `--kubelet-client-key` +flags passed to the API server is authorized for the following attributes: + +* verb=\*, resource=nodes, subresource=proxy +* verb=\*, resource=nodes, subresource=stats +* verb=\*, resource=nodes, subresource=log +* verb=\*, resource=nodes, subresource=spec +* verb=\*, resource=nodes, subresource=metrics +* verb=\*, resource=nodes, subresource=configz +* verb=\*, resource=nodes, subresource=healthz +* verb=\*, resource=nodes, subresource=pods diff --git a/content/en/docs/reference/access-authn-authz/mutating-admission-policy.md b/content/en/docs/reference/access-authn-authz/mutating-admission-policy.md new file mode 100644 index 0000000000000..14174d34296c2 --- /dev/null +++ b/content/en/docs/reference/access-authn-authz/mutating-admission-policy.md @@ -0,0 +1,216 @@ +--- +reviewers: +- deads2k +- sttts +- cici37 +title: Mutating Admission Policy +content_type: concept +--- + + + +{{< feature-state for_k8s_version="v1.32" state="alpha" >}} + + +This page provides an overview of _MutatingAdmissionPolicies_. + + + +## What are MutatingAdmissionPolicies? + +Mutating admission policies offer a declarative, in-process alternative to mutating admission webhooks. + +Mutating admission policies use the Common Expression Language (CEL) to declare mutations to resources. +Mutations can be defined either with an *apply configuration* that is merged using the +[server side apply merge strategy](/docs/reference/using-api/server-side-apply/#merge-strategy), +or a [JSON patch](https://jsonpatch.com/). + +Mutating admission policies are highly configurable, enabling policy authors to define policies +that can be parameterized and scoped to resources as needed by cluster administrators. + +## What resources make a policy + +A policy is generally made up of three resources: + +- The MutatingAdmissionPolicy describes the abstract logic of a policy + (think: "this policy sets a particular label to a particular value"). + +- A _parameter resource_ provides information to a MutatingAdmissionPolicy to make it a concrete + statement (think "set the `owner` label to something like `company.example.com`"). + Parameter resources refer to Kubernetes resources, available in the Kubernetes API. They can be built-in types or extensions, + such as a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} (CRD). For example, you can use a ConfigMap as a parameter. +- A MutatingAdmissionPolicyBinding links the above (MutatingAdmissionPolicy and parameter) resources together and provides scoping. + If you only want to set an `owner` label for `Pods`, and not other API kinds, the binding is where you + specify this mutation. + + + +At least a MutatingAdmissionPolicy and a corresponding MutatingAdmissionPolicyBinding +must be defined for a policy to have an effect. + +If a MutatingAdmissionPolicy does not need to be configured via parameters, simply leave +`spec.paramKind` in MutatingAdmissionPolicy not specified. + +## Getting Started with MutatingAdmissionPolicies + +Mutating admission policy is part of the cluster control-plane. You should write +and deploy them with great caution. The following describes how to quickly +experiment with Mutating admission policy. + +### Create a MutatingAdmissionPolicy + +The following is an example of a MutatingAdmissionPolicy. This policy mutates newly created Pods to have a sidecar container if it does not exist. + +{{% code_sample language="yaml" file="mutatingadmissionpolicy/applyconfiguration-example.yaml" %}} + +The `.spec.mutations` field consists of a list of expressions that evaluate to resource patches. +The emitted patches may be either [apply configurations](#patch-type-apply-configuration) or [JSON Patch](#patch-type-json-patch) +patches. You cannot specify an empty list of mutations. After evaluating all the +expressions, the API server applies those changes to the resource that is +passing through admission. + +To configure a mutating admission policy for use in a cluster, a binding is +required. The MutatingAdmissionPolicy will only be active if a corresponding +binding exists with the referenced `spec.policyName` matching the `spec.name` of +a policy. + +Once the binding and policy are created, any resource request that matches the +`spec.matchConditions` of a policy will trigger the set of mutations defined. + +In the example above, creating a Pod will add the `mesh-proxy` initContainer mutation: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: myapp + namespace: default +spec: + ... + initContainers: + - name: mesh-proxy + image: mesh/proxy:v1.0.0 + args: ["proxy", "sidecar"] + restartPolicy: Always + - name: myapp-initializer + image: example/initializer:v1.0.0 + ... +``` + +#### Parameter resources + +Parameter resources allow a policy configuration to be separate from its +definition. A policy can define `paramKind`, which outlines GVK of the parameter +resource, and then a policy binding ties a policy by name (via `policyName`) to a +particular parameter resource via `paramRef`. + +Please refer to [parameter resources](/docs/reference/access-authn-authz/validating-admission-policy/#parameter-resources) for more information. + +#### `ApplyConfiguration` {#patch-type-apply-configuration} + +MutatingAdmissionPolicy expressions are always CEL. Each apply configuration +`expression` must evaluate to a CEL object (declared using `Object()` +initialization). + +Apply configurations may not modify atomic structs, maps or arrays due to the risk of accidental deletion of +values not included in the apply configuration. + +CEL expressions have access to the object types needed to create apply configurations: + +- `Object` - CEL type of the resource object. +- `Object.` - CEL type of object field (such as `Object.spec`) +- `Object.....` - CEL type of nested field (such as `Object.spec.containers`) + +CEL expressions have access to the contents of the API request, organized into CEL variables as well as some other useful variables: + +- `object` - The object from the incoming request. The value is null for DELETE requests. +- `oldObject` - The existing object. The value is null for CREATE requests. +- `request` - Attributes of the API request. +- `params` - Parameter resource referred to by the policy binding being evaluated. Only populated if the policy has a ParamKind. +- `namespaceObject` - The namespace object that the incoming object belongs to. The value is null for cluster-scoped resources. +- `variables` - Map of composited variables, from its name to its lazily evaluated value. + For example, a variable named `foo` can be accessed as `variables.foo`. +- `authorizer` - A CEL Authorizer. May be used to perform authorization checks for the principal (user or service account) of the request. + See https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#Authz +- `authorizer.requestResource` - A CEL ResourceCheck constructed from the `authorizer` and configured with the + request resource. + +The `apiVersion`, `kind`, `metadata.name`, `metadata.generateName` and `metadata.labels` are always accessible from the root of the +object. No other metadata properties are accessible. + +#### `JSONPatch` {#patch-type-json-patch} + +The same mutation can be written as a [JSON Patch](https://jsonpatch.com/) as follows: + +{{% code_sample language="yaml" file="mutatingadmissionpolicy/json-patch-example.yaml" %}} + +The expression will be evaluated by CEL to create a [JSON patch](https://jsonpatch.com/). +ref: https://github.com/google/cel-spec + +Each evaluated `expression` must return an array of `JSONPatch` values. The +`JSONPatch` type represents one operation from a JSON patch. + +For example, this CEL expression returns a JSON patch to conditionally modify a value: + +``` + [ + JSONPatch{op: "test", path: "/spec/example", value: "Red"}, + JSONPatch{op: "replace", path: "/spec/example", value: "Green"} + ] +``` + +To define a JSON object for the patch operation `value`, use CEL `Object` types. For example: + +``` + [ + JSONPatch{ + op: "add", + path: "/spec/selector", + value: Object.spec.selector{matchLabels: {"environment": "test"}} + } + ] +``` + +To use strings containing '/' and '~' as JSONPatch path keys, use `jsonpatch.escapeKey()`. For example: + +``` + [ + JSONPatch{ + op: "add", + path: "/metadata/labels/" + jsonpatch.escapeKey("example.com/environment"), + value: "test" + }, + ] +``` + +CEL expressions have access to the types needed to create JSON patches and objects: + +- `JSONPatch` - CEL type of JSON Patch operations. JSONPatch has the fields `op`, `from`, `path` and `value`. + See [JSON patch](https://jsonpatch.com/) for more details. The `value` field may be set to any of: string, + integer, array, map or object. If set, the `path` and `from` fields must be set to a + [JSON pointer](https://datatracker.ietf.org/doc/html/rfc6901/) string, where the `jsonpatch.escapeKey()` CEL + function may be used to escape path keys containing `/` and `~`. +- `Object` - CEL type of the resource object. +- `Object.` - CEL type of object field (such as `Object.spec`) +- `Object.....` - CEL type of nested field (such as `Object.spec.containers`) + +CEL expressions have access to the contents of the API request, organized into CEL variables as well as some other useful variables: + +- `object` - The object from the incoming request. The value is null for DELETE requests. +- `oldObject` - The existing object. The value is null for CREATE requests. +- `request` - Attributes of the API request. +- `params` - Parameter resource referred to by the policy binding being evaluated. Only populated if the policy has a ParamKind. +- `namespaceObject` - The namespace object that the incoming object belongs to. The value is null for cluster-scoped resources. +- `variables` - Map of composited variables, from its name to its lazily evaluated value. + For example, a variable named `foo` can be accessed as `variables.foo`. +- `authorizer` - A CEL Authorizer. May be used to perform authorization checks for the principal (user or service account) of the request. + See https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#Authz +- `authorizer.requestResource` - A CEL ResourceCheck constructed from the `authorizer` and configured with the + request resource. + +CEL expressions have access to [Kubernetes CEL function libraries](/docs/reference/using-api/cel/#cel-options-language-features-and-libraries) +as well as: + +- `jsonpatch.escapeKey` - Performs JSONPatch key escaping. `~` and `/` are escaped as `~0` and `~1` respectively. + +Only property names of the form `[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible. \ No newline at end of file diff --git a/content/en/docs/reference/access-authn-authz/node.md b/content/en/docs/reference/access-authn-authz/node.md index d39b404c0c80b..dd144d7fa1403 100644 --- a/content/en/docs/reference/access-authn-authz/node.md +++ b/content/en/docs/reference/access-authn-authz/node.md @@ -69,7 +69,24 @@ the local `hostname` and the `--hostname-override` option. For specifics about how the kubelet determines the hostname, see the [kubelet options reference](/docs/reference/command-line-tools-reference/kubelet/). -To enable the Node authorizer, start the apiserver with `--authorization-mode=Node`. +To enable the Node authorizer, start the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} +with the `--authorization-config` flag set to a file that includes the `Node` authorizer; for example: + +```yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: AuthorizationConfiguration +authorizers: + ... + - type: Node + ... +``` + +Or, start the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} with +the `--authorization-mode` flag set to a comma-separated list that includes `Node`; +for example: +```shell +kube-apiserver --authorization-mode=...,Node --other-options --more-options +``` To limit the API objects kubelets are able to write, enable the [NodeRestriction](/docs/reference/access-authn-authz/admission-controllers#noderestriction) diff --git a/content/en/docs/reference/access-authn-authz/rbac.md b/content/en/docs/reference/access-authn-authz/rbac.md index a18903db4cb2e..2608ba32284f2 100644 --- a/content/en/docs/reference/access-authn-authz/rbac.md +++ b/content/en/docs/reference/access-authn-authz/rbac.md @@ -20,10 +20,22 @@ RBAC authorization uses the `rbac.authorization.k8s.io` decisions, allowing you to dynamically configure policies through the Kubernetes API. To enable RBAC, start the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} -with the `--authorization-mode` flag set to a comma-separated list that includes `RBAC`; +with the `--authorization-config` flag set to a file that includes the `RBAC` authorizer; for example: + +```yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: AuthorizationConfiguration +authorizers: + ... + - type: RBAC + ... +``` + +Or, start the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} with +the `--authorization-mode` flag set to a comma-separated list that includes `RBAC`; for example: ```shell -kube-apiserver --authorization-mode=Example,RBAC --other-options --more-options +kube-apiserver --authorization-mode=...,RBAC --other-options --more-options ``` ## API objects {#api-overview} diff --git a/content/en/docs/reference/access-authn-authz/service-accounts-admin.md b/content/en/docs/reference/access-authn-authz/service-accounts-admin.md index 791654393f51e..3dc41cd3f9227 100644 --- a/content/en/docs/reference/access-authn-authz/service-accounts-admin.md +++ b/content/en/docs/reference/access-authn-authz/service-accounts-admin.md @@ -18,7 +18,8 @@ For an introduction to service accounts, read [configure service accounts](/docs This task guide explains some of the concepts behind ServiceAccounts. The guide also explains how to obtain or revoke tokens that represent -ServiceAccounts. +ServiceAccounts, and how to (optionally) bind a ServiceAccount's validity to +the lifetime of an API object. @@ -68,7 +69,7 @@ Supported object types are as follows: * Pod (used for projected volume mounts, see below) * Secret (can be used to allow revoking a token by deleting the Secret) -* Node (in v1.30, creating new node-bound tokens is alpha, using existing node-bound tokens is beta) +* Node (in v1.32, creating new node-bound tokens is beta, using existing node-bound tokens is GA) When a token is bound to an object, the object's `metadata.name` and `metadata.uid` are stored as extra 'private claims' in the issued JWT. @@ -159,6 +160,70 @@ does not actually persist the TokenReview object into etcd. Hence `kubectl get tokenreview` is not a valid command. {{< /note >}} +#### Schema for service account private claims + +The schema for the Kubernetes-specific claims within JWT tokens is not currently documented, +however the relevant code area can be found in +[the serviceaccount package](https://github.com/kubernetes/kubernetes/blob/d8919343526597e0788a1efe133c70d9a0c07f69/pkg/serviceaccount/claims.go#L56-L68) +in the Kubernetes codebase. + +You can inspect a JWT using standard JWT decoding tool. Below is an example of a JWT for the +`my-serviceaccount` ServiceAccount, bound to a Pod object named `my-pod` which is scheduled +to the Node `my-node`, in the `my-namespace` namespace: + +```json +{ + "aud": [ + "https://my-audience.example.com" + ], + "exp": 1729605240, + "iat": 1729601640, + "iss": "https://my-cluster.example.com", + "jti": "aed34954-b33a-4142-b1ec-389d6bbb4936", + "kubernetes.io": { + "namespace": "my-namespace", + "node": { + "name": "my-node", + "uid": "646e7c5e-32d6-4d42-9dbd-e504e6cbe6b1" + }, + "pod": { + "name": "my-pod", + "uid": "5e0bd49b-f040-43b0-99b7-22765a53f7f3" + }, + "serviceaccount": { + "name": "my-serviceaccount", + "uid": "14ee3fa4-a7e2-420f-9f9a-dbc4507c3798" + } + }, + "nbf": 1729601640, + "sub": "system:serviceaccount:my-namespace:my-serviceaccount" +} +``` + +{{< note >}} +The `aud` and `iss` fields in this JWT may differ between different Kubernetes clusters depending +on your configuration. + +The presence of both the `pod` and `node` claim implies that this token is bound +to a *Pod* object. When verifying Pod bound ServiceAccount tokens, the API server **does not** +verify the existence of the referenced Node object. +{{< /note >}} + +Services that run outside of Kubernetes and want to perform offline validation of JWTs may +use this schema, along with a compliant JWT validator configured with OpenID Discovery information +from the API server, to verify presented JWTs without requiring use of the TokenReview API. + +Services that verify JWTs in this way **do not verify** the claims embedded in the JWT token to be +current and still valid. +This means if the token is bound to an object, and that object no longer exists, the token will still +be considered valid (until the configured token expires). + +Clients that require assurance that a token's bound claims are still valid **MUST** use the TokenReview +API to present the token to the `kube-apiserver` for it to verify and expand the embedded claims, using +similar steps to the [Verifying and inspecting private claims](#verifying-and-inspecting-private-claims) +section above, but with a [supported client library](/docs/reference/using-api/client-libraries/). +For more information on JWTs and their structure, see the [JSON Web Token RFC](https://datatracker.ietf.org/doc/html/rfc7519). + ## Bound service account token volume mechanism {#bound-service-account-token-volume} {{< feature-state feature_gate_name="BoundServiceAccountTokenVolume" >}} @@ -319,6 +384,12 @@ Similarly, you must pass the corresponding public key to the `kube-apiserver` using the `--service-account-key-file` flag. The public key will be used to verify the tokens during authentication. +{{< feature-state feature_gate_name="ExternalServiceAccountTokenSigner" >}} + +An alternate setup to setting `--service-account-private-key-file` and `--service-account-key-file` flags is +to configure an external JWT signer for [external ServiceAccount token signing and key management](#external-serviceaccount-token-signing-and-key-management). +Note that these setups are mutually exclusive and cannot be configured together. + ### ServiceAccount admission controller The modification of pods is implemented via a plugin @@ -544,6 +615,22 @@ Then, delete the Secret you now know the name of: kubectl -n examplens delete secret/example-automated-thing-token-zyxwv ``` +## External ServiceAccount token signing and key management + +{{< feature-state feature_gate_name="ExternalServiceAccountTokenSigner" >}} + +The kube-apiserver can be configured to use external signer for token signing and token verifying key management. +This feature enables kubernetes distributions to integrate with key management solutions of their choice (eg: HSMs, cloud KMSes) for service account credential signing and verification. +To configure kube-apiserver to use external-jwt-signer set the `--service-account-signing-endpoint` flag to the location of a Unix domain socket (UDS) on a filesystem, or be prefixed with an @ symbol and name a UDS in the abstract socket namespace. +At the configured UDS, shall be an RPC server which implements [ExternalJWTSigner](https://github.com/kubernetes/kubernetes/blob/release-1.32/staging/src/k8s.io/externaljwt/apis/v1alpha1/api.proto). +The external-jwt-signer must be healthy and be ready to serve supported service account keys for the kube-apiserver to start. + +Check out [KEP-740](https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/740-service-account-external-signing) for more details on ExternalJWTSigner. + +{{< note >}} +The kube-apiserver flags `--service-account-key-file` and `--service-account-signing-key-file` will continue to be used for reading from files unless `--service-account-signing-endpoint` is set; they are mutually exclusive ways of supporting JWT signing and authentication. +{{< /note >}} + ## Clean up If you created a namespace `examplens` to experiment with, you can remove it: diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/allow-unsafe-malformed-object-deletion.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/allow-unsafe-malformed-object-deletion.md new file mode 100644 index 0000000000000..83ee02bb33abc --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/allow-unsafe-malformed-object-deletion.md @@ -0,0 +1,16 @@ +--- +title: AllowUnsafeMalformedObjectDeletion +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables the cluster operator to identify corrupt resource(s) using the **list** +operation, and introduces an option `ignoreStoreReadErrorWithClusterBreakingPotential` +that the operator can set to perform unsafe and force **delete** operation of +such corrupt resource(s) using the Kubernetes API. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/anonymous-auth-configurable-endpoints.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/anonymous-auth-configurable-endpoints.md index 38c4f6666debf..f731f04625a33 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/anonymous-auth-configurable-endpoints.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/anonymous-auth-configurable-endpoints.md @@ -9,6 +9,10 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.31" - + toVersion: "1.31" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- -Enable configuring anonymous authentication / authorization for only certain API server endpoints. +Enable [configurable endpoints for anonymous auth](/docs/reference/access-authn-authz/authentication/#anonymous-authenticator-configuration) +for the API server. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-node-with-selectors.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-node-with-selectors.md index 9c09c59f97290..db45f7ec27dbd 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-node-with-selectors.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-node-with-selectors.md @@ -9,6 +9,10 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.31" + toVersion: "1.31" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- Make the [Node authorizer](/docs/reference/access-authn-authz/node/) use fine-grained selector authorization. Requires `AuthorizeWithSelectors` to be enabled. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-with-selectors.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-with-selectors.md index 4626d486b174c..d53ff8d6305ea 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-with-selectors.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/authorize-with-selectors.md @@ -9,6 +9,10 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.31" + toVersion: "1.31" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- Allows authorization to use field and label selectors. Enables `fieldSelector` and `labelSelector` fields in the [SubjectAccessReview API](/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/), diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/cbor-serving-and-storage.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/cbor-serving-and-storage.md new file mode 100644 index 0000000000000..ac3e76422a41e --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/cbor-serving-and-storage.md @@ -0,0 +1,15 @@ +--- +title: CBORServingAndStorage +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables CBOR as a [supported encoding for requests and +responses](/docs/reference/using-api/api-concepts/#cbor-encoding), and as the preferred storage +encoding for custom resources. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/cloud-dual-stack-node-ips.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/cloud-dual-stack-node-ips.md index 970fc6ae8e3d6..f36aab7d5179d 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/cloud-dual-stack-node-ips.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/cloud-dual-stack-node-ips.md @@ -17,7 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" +removed: true --- Enables dual-stack `kubelet --node-ip` with external cloud providers. See [Configure IPv4/IPv6 dual-stack](/docs/concepts/services-networking/dual-stack/#configure-ipv4-ipv6-dual-stack) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/component-flagz.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/component-flagz.md new file mode 100644 index 0000000000000..07854760d78c6 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/component-flagz.md @@ -0,0 +1,14 @@ +--- +title: ComponentFlagz +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables the component's flagz endpoint. +See [zpages](/docs/reference/instrumentation/zpages/) for more information. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/component-statusz.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/component-statusz.md new file mode 100644 index 0000000000000..60c15d36a6671 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/component-statusz.md @@ -0,0 +1,14 @@ +--- +title: ComponentStatusz +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables the component's statusz endpoint. +See [zpages](/docs/reference/instrumentation/zpages/) for more information. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/cron-jobs-scheduled-annotation.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/cron-jobs-scheduled-annotation.md index db6c450923338..32ba03a320fec 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/cron-jobs-scheduled-annotation.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/cron-jobs-scheduled-annotation.md @@ -9,6 +9,11 @@ stages: - stage: beta defaultValue: true fromVersion: "1.28" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" + --- Set the scheduled job time as an {{< glossary_tooltip text="annotation" term_id="annotation" >}} on Jobs that were created diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/custom-resource-field-selectors.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/custom-resource-field-selectors.md index 77c579c748951..d6e261e0aef57 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/custom-resource-field-selectors.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/custom-resource-field-selectors.md @@ -12,7 +12,11 @@ stages: toVersion: "1.30" - stage: beta defaultValue: true - fromVersion: "1.31" + fromVersion: "1.31" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Enable `selectableFields` in the diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-admin-access.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-admin-access.md new file mode 100644 index 0000000000000..7720c3786d955 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-admin-access.md @@ -0,0 +1,18 @@ +--- +title: DRAAdminAccess +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables support for requesting [admin access](/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#admin-access) +in a ResourceClaim. A ResourceClaim +with admin access grants access to devices which are in use and may enable +additional access permissions when making the device available in a container. + +This feature gate has no effect unless you also enable the `DynamicResourceAllocation` feature gate. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-control-plane-controller.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-control-plane-controller.md index b5af438f14578..9b31d8f1c7915 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-control-plane-controller.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-control-plane-controller.md @@ -9,6 +9,9 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.26" + toVersion: "1.31" + +removed: true --- Enables support for resources with custom parameters and a lifecycle that is independent of a Pod. Allocation of resources is handled diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-resource-claim-device-status.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-resource-claim-device-status.md new file mode 100644 index 0000000000000..ac33e12a3a59d --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/dra-resource-claim-device-status.md @@ -0,0 +1,14 @@ +--- +title: DRAResourceClaimDeviceStatus +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables support the ResourceClaim.status.devices field and for setting this +status from DRA drivers. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/dynamic-resource-allocation.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/dynamic-resource-allocation.md index 142a0e8c45bb6..adaad7cf6ae85 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/dynamic-resource-allocation.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/dynamic-resource-allocation.md @@ -8,7 +8,16 @@ _build: stages: - stage: alpha defaultValue: false - fromVersion: "1.26" + fromVersion: "1.30" + toVersion: "1.31" + - stage: beta + defaultValue: false + fromVersion: "1.32" + +# TODO: as soon as this is locked to "true" (= GA), comments about other DRA +# feature gate(s) like "unless you also enable the `DynamicResourceAllocation` feature gate" +# can be removed (for example, in dra-admin-access.md). + --- Enables support for resources with custom parameters and a lifecycle that is independent of a Pod. Allocation of resources is handled diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/external-service-account-token-signer.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/external-service-account-token-signer.md new file mode 100644 index 0000000000000..23e5e2a673eee --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/external-service-account-token-signer.md @@ -0,0 +1,13 @@ +--- +title: ExternalServiceAccountTokenSigner +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enable setting `--service-account-signing-endpoint` to make the kube-apiserver use [external signer](/docs/reference/access-authn-authz/service-account-admin#external-serviceaccount-token-signing-and-key-management) for token signing and token verifying key management. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/hpa-container-metrics.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/hpa-container-metrics.md index 84d076cb4b6a6..ad5b945b9927d 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/hpa-container-metrics.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/hpa-container-metrics.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + +removed: true --- Allow {{< glossary_tooltip text="HorizontalPodAutoscalers" term_id="horizontal-pod-autoscaler" >}} to scale based on metrics from individual containers within target pods. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/job-managed-by.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/job-managed-by.md index 38733b6de66ff..cbf74738143a3 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/job-managed-by.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/job-managed-by.md @@ -10,5 +10,9 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.30" + toVersion: "1.31" + - stage: beta + defaultValue: false + fromVersion: "1.32" --- Allows to delegate reconciliation of a Job object to an external controller. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2-kdf.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2-kdf.md index 213c9a664cb4d..96add97b1fac2 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2-kdf.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2-kdf.md @@ -12,7 +12,10 @@ stages: toVersion: "1.28" - stage: stable defaultValue: true - fromVersion: "1.29" + fromVersion: "1.29" + toVersion: "1.31" + +removed: true --- Enables KMS v2 to generate single use data encryption keys. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2.md index 52ce6b7b6ab5e..d7890e0c2c3f3 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/kmsv2.md @@ -16,6 +16,9 @@ stages: toVersion: "1.28" - stage: stable defaultValue: true - fromVersion: "1.29" + fromVersion: "1.29" + toVersion: "1.31" + +removed: true --- Enables KMS v2 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/kubelet-crash-loop-back-off-max.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/kubelet-crash-loop-back-off-max.md new file mode 100644 index 0000000000000..36d3493f1f396 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/kubelet-crash-loop-back-off-max.md @@ -0,0 +1,14 @@ +--- +title: KubeletCrashLoopBackOffMax +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables support for configurable per-node backoff maximums for restarting +containers in the CrashLoopBackOff state. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/kubelet-finegrained-authz.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/kubelet-finegrained-authz.md new file mode 100644 index 0000000000000..0fb0afb8b4bdb --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/kubelet-finegrained-authz.md @@ -0,0 +1,14 @@ +--- +title: KubeletFineGrainedAuthz +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enable [fine-grained authorization](/docs/reference/access-authn-authz/kubelet-authn-authz/#fine-grained-authorization) +for the kubelet's HTTP(s) API. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/legacy-service-account-token-clean-up.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/legacy-service-account-token-clean-up.md index f22aaae479dff..9adcaa0e74d57 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/legacy-service-account-token-clean-up.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/legacy-service-account-token-clean-up.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + +removed: true --- Enable cleaning up Secret-based [service account tokens](/docs/concepts/security/service-accounts/#get-a-token) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/load-balancer-ip-mode.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/load-balancer-ip-mode.md index 6b87fd3abff38..9b692c6668ec0 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/load-balancer-ip-mode.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/load-balancer-ip-mode.md @@ -13,6 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Allows setting `ipMode` for Services where `type` is set to `LoadBalancer`. See [Specifying IPMode of load balancer status](/docs/concepts/services-networking/service/#load-balancer-ip-mode) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/memory-manager.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/memory-manager.md index e9db5c4021155..87b2206e8c82d 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/memory-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/memory-manager.md @@ -13,6 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.22" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Allows setting memory affinity for a container based on NUMA topology. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/min-domains-in-pod-topology-spread.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/min-domains-in-pod-topology-spread.md index ae8a3f7f383ad..4a09546d540a9 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/min-domains-in-pod-topology-spread.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/min-domains-in-pod-topology-spread.md @@ -21,6 +21,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + +removed: true --- Enable `minDomains` in [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/). diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/mutating-admission-policy.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/mutating-admission-policy.md index e25313e80a7e0..4da5395b3296c 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/mutating-admission-policy.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/mutating-admission-policy.md @@ -10,7 +10,7 @@ stages: defaultValue: false fromVersion: "1.30" --- -In Kubernetes {{< skew currentVersion >}}, this feature gate has no effect. -A future release of Kubernetes may use this feature gate to enable -the MutatingAdmissionPolicy in admission chain. +Enable [MutatingAdmissionPolicy](/docs/reference/access-authn-authz/mutating-admission-policy/) support for [CEL](https://kubernetes.io/docs/reference/using-api/cel/) mutations be used in admission control. + +For Kubernetes v1.30 and v1.31, this feature gate existed but had no effect. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/new-volume-manager-reconstruction.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/new-volume-manager-reconstruction.md index 6756213d92eb6..ff2a4a1d5f90d 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/new-volume-manager-reconstruction.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/new-volume-manager-reconstruction.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + +removed: true --- Enables improved discovery of mounted volumes during kubelet startup. Since the associated code had been significantly refactored, Kubernetes versions 1.25 to 1.29 diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/node-out-of-service-volume-detach.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/node-out-of-service-volume-detach.md index e3f0998d80799..ad5d5de72a3b3 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/node-out-of-service-volume-detach.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/node-out-of-service-volume-detach.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.28" + toVersion: "1.31" + +removed: true --- When a Node is marked out-of-service using the `node.kubernetes.io/out-of-service` taint, Pods on the node will be forcefully deleted diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-host-ips.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-host-ips.md index 0f39a10790f3c..c6ddd0084b1ee 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-host-ips.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-host-ips.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" +removed: true + --- Enable the `status.hostIPs` field for pods and the {{< glossary_tooltip term_id="downward-api" text="downward API" >}}. The field lets you expose host IP addresses to workloads. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-index-label.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-index-label.md index be509292c662d..40118ee7499e9 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-index-label.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-index-label.md @@ -9,5 +9,9 @@ stages: - stage: beta defaultValue: true fromVersion: "1.28" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Enables the Job controller and StatefulSet controller to add the pod index as a label when creating new pods. See [Job completion mode docs](/docs/concepts/workloads/controllers/job#completion-mode) and [StatefulSet pod index label docs](/docs/concepts/workloads/controllers/statefulset/#pod-index-label) for more details. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-level-resources.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-level-resources.md new file mode 100644 index 0000000000000..80cbc1027de4c --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-level-resources.md @@ -0,0 +1,14 @@ +--- +title: PodLevelResources +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enable _Pod level resources_: the ability to specify resource requests and limits +at the Pod level, rather than only for specific containers. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-lifecycle-sleep-action-allow-zero.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-lifecycle-sleep-action-allow-zero.md new file mode 100644 index 0000000000000..831f9555c1fc3 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-lifecycle-sleep-action-allow-zero.md @@ -0,0 +1,13 @@ +--- +title: PodLifecycleSleepActionAllowZero +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables setting zero value for the `sleep` action in [container lifecycle hooks](/docs/concepts/containers/container-lifecycle-hooks/). diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-logs-query-split-streams.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-logs-query-split-streams.md new file mode 100644 index 0000000000000..ba7a066d0a7b5 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/pod-logs-query-split-streams.md @@ -0,0 +1,13 @@ +--- +title: PodLogsQuerySplitStreams +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enable fetching specific log streams (either stdout or stderr) from a container's log streams, using the Pod API. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/prefer-align-cpus-by-uncorecache.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/prefer-align-cpus-by-uncorecache.md new file mode 100644 index 0000000000000..9914843dd8b05 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/prefer-align-cpus-by-uncorecache.md @@ -0,0 +1,13 @@ +--- +title: PreferAlignCpusByUncoreCache +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +When `PreferAlignCpusByUncoreCache` is enabled while the CPU Manager Policy is set to `static`, containers within a `Guaranteed` pod will individually be aligned to an uncore cache group at a best-effort policy. This feature can optimize performance for certain cache-sensitive workloads by minimizing the cpu allocation across uncore caches. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/relaxed-environment-variable-validation.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/relaxed-environment-variable-validation.md index 862ae57214bae..870553d0f9cfa 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/relaxed-environment-variable-validation.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/relaxed-environment-variable-validation.md @@ -9,5 +9,9 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.30" + toVersion: "1.31" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- Allow almost all printable ASCII characters in environment variables. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/retry-generate-name.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/retry-generate-name.md index 838951950bbb1..dd5a45854b08a 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/retry-generate-name.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/retry-generate-name.md @@ -13,7 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.31" - + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Enables retrying of object creation when the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}} diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-async-preemption.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-async-preemption.md new file mode 100644 index 0000000000000..7c39bd39c0e80 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-async-preemption.md @@ -0,0 +1,16 @@ +--- +title: SchedulerAsyncPreemption +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- + +Enable running some expensive operations within the scheduler, associated with +[preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/), asynchronously. +Asynchronous processing of preemption improves overall Pod scheduling latency. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-queueing-hints.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-queueing-hints.md index b60c4ddc506c9..dba2cb95a9a40 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-queueing-hints.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/scheduler-queueing-hints.md @@ -13,8 +13,12 @@ stages: - stage: beta defaultValue: false fromVersion: "1.29" + toVersion: "1.31" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- -Enables [the scheduler's _queueing hints_ enhancement](https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4247-queueinghint/README.md), +Enables [the scheduler's _queueing hints_ feature](/docs/concepts/scheduling-eviction/scheduling-framework/#queueinghint), which benefits to reduce the useless requeueing. The scheduler retries scheduling pods if something changes in the cluster that could make the pod scheduled. Queueing hints are internal signals that allow the scheduler to filter the changes in the cluster diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-change-policy.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-change-policy.md new file mode 100644 index 0000000000000..ed2e3d4e11cdc --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-change-policy.md @@ -0,0 +1,20 @@ +--- +title: SELinuxChangePolicy +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Enables `spec.securityContext.seLinuxChangePolicy` field. +This field can be used to opt-out from applying the SELinux label to the pod +volumes using mount options. This is required when a single volume that supports +mounting with SELinux mount option is shared between Pods that have different +SELinux labels, such as a privileged and unprivileged Pods. + +Enabling the `SELinuxChangePolicy` feature gate requires the feature gate `SELinuxMountReadWriteOncePod` to +be enabled. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-mount.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-mount.md index 124862976773c..6e786d50d8a9c 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-mount.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/selinux-mount.md @@ -16,5 +16,5 @@ recursively. It widens the performance improvements behind the `SELinuxMountReadWriteOncePod` feature gate by extending the implementation to all volumes. -Enabling the `SELinuxMount` feature gate requires the feature gate `SELinuxMountReadWriteOncePod` to -be enabled. +Enabling the `SELinuxMount` feature gate requires the feature gates `SELinuxMountReadWriteOncePod` +and `SELinuxChangePolicy` to be enabled. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-apply.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-apply.md index 8decb8b3c69a2..edeb86d7a4792 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-apply.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-apply.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.22" + toVersion: "1.31" + +removed: true --- Enables the [Sever Side Apply (SSA)](/docs/reference/using-api/server-side-apply/) feature on the API Server. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-field-validation.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-field-validation.md index 91849ecb76b49..3399e3e1af917 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-field-validation.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/server-side-field-validation.md @@ -17,6 +17,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.27" + toVersion: "1.31" + +removed: true --- Enables server-side field validation. This means the validation of resource schema is performed at the API server side rather than the client side diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-jti.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-jti.md index ab82953ada6da..3e63377aa1520 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-jti.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-jti.md @@ -13,6 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Controls whether JTIs (UUIDs) are embedded into generated service account tokens, and whether these JTIs are recorded into the Kubernetes audit log for future requests made by these tokens. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-node-binding-validation.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-node-binding-validation.md index 94021587aef52..023556884e344 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-node-binding-validation.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-node-binding-validation.md @@ -13,6 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Controls whether the apiserver will validate a Node reference in service account tokens. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-pod-node-info.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-pod-node-info.md index 86d8940b55ec2..624fd96242ac3 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-pod-node-info.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/service-account-token-pod-node-info.md @@ -13,6 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Controls whether the apiserver embeds the node name and uid for the associated node when issuing service account tokens bound to Pod objects. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/size-memory-backed-volumes.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/size-memory-backed-volumes.md index ea6ea9a4c5cce..69162233c6014 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/size-memory-backed-volumes.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/size-memory-backed-volumes.md @@ -13,6 +13,10 @@ stages: - stage: beta defaultValue: true fromVersion: "1.22" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Enable kubelets to determine the size limit for memory-backed volumes (mainly `emptyDir` volumes). diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/stable-load-balancer-node-set.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/stable-load-balancer-node-set.md index e2968c340482d..943d90606cf86 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/stable-load-balancer-node-set.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/stable-load-balancer-node-set.md @@ -13,6 +13,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + +removed: true --- Enables less load balancer re-configurations by the service controller (KCCM) as an effect of changing node state. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/stateful-set-auto-delete-pvc.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/stateful-set-auto-delete-pvc.md index 99967d2059258..b04022452ea5d 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/stateful-set-auto-delete-pvc.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/stateful-set-auto-delete-pvc.md @@ -14,9 +14,13 @@ stages: - stage: beta defaultValue: true fromVersion: "1.27" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Allows the use of the optional `.spec.persistentVolumeClaimRetentionPolicy` field, providing control over the deletion of PVCs in a StatefulSet's lifecycle. See [PersistentVolumeClaim retention](/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention) -for more details. \ No newline at end of file +for more details. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/structured-authorization-configuration.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/structured-authorization-configuration.md index d2f1a47283c6a..b40c8193298d0 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/structured-authorization-configuration.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/structured-authorization-configuration.md @@ -6,13 +6,17 @@ _build: render: false stages: - - stage: alpha + - stage: alpha defaultValue: false fromVersion: "1.29" toVersion: "1.29" - - stage: beta + - stage: beta defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Enable structured authorization configuration, so that cluster administrators can specify more than one [authorization webhook](/docs/reference/access-authn-authz/webhook/) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/systemd-watchdog.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/systemd-watchdog.md new file mode 100644 index 0000000000000..830720dcca33b --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/systemd-watchdog.md @@ -0,0 +1,15 @@ +--- +title: SystemdWatchdog +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: beta + defaultValue: true + fromVersion: "1.32" +--- +Allow using systemd watchdog to monitor the health status of kubelet. +See [Kubelet Systemd Watchdog](/docs/reference/node/systemd-watchdog/) +for more details. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/topology-manager-policy-options.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/topology-manager-policy-options.md index 6a59d8f40bd4e..a2cbab3f385c4 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/topology-manager-policy-options.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/topology-manager-policy-options.md @@ -6,13 +6,17 @@ _build: render: false stages: - - stage: alpha + - stage: alpha defaultValue: false fromVersion: "1.26" toVersion: "1.27" - stage: beta defaultValue: true fromVersion: "1.28" + toVersion: "1.31" + - stage: stable + defaultValue: true + fromVersion: "1.32" --- Enable [fine-tuning](/docs/tasks/administer-cluster/topology-manager/#topology-manager-policy-options) -of topology manager policies. +of topology manager policies. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list-client.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list-client.md new file mode 100644 index 0000000000000..826e19712c81f --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list-client.md @@ -0,0 +1,17 @@ +--- +title: WatchListClient +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: beta + defaultValue: false + fromVersion: "1.30" +--- +Allows an API client to request a stream of data rather than fetching a full list. +This functionality is available in `client-go` and requires the +[WatchList](/docs/reference/command-line-tools-reference/feature-gates/) +feature to be enabled on the server. +If the `WatchList` is not supported on the server, the client will seamlessly fall back to a standard list request. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list.md index cd258d1afd54a..1595cd251dce2 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/watch-list.md @@ -9,5 +9,9 @@ stages: - stage: alpha defaultValue: false fromVersion: "1.27" + toVersion: "1.31" + - stage: beta + defaultValue: true + fromVersion: "1.32" --- Enable support for [streaming initial state of objects in watch requests](/docs/reference/using-api/api-concepts/#streaming-lists). diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md new file mode 100644 index 0000000000000..00dfa1b4c4a3a --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md @@ -0,0 +1,17 @@ +--- +title: WindowsCPUAndMemoryAffinity +content_type: feature_gate + +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- + +Add CPU and Memory Affinity support to Windows nodes with [CPUManager](/docs/tasks/administer-cluster/cpu-management-policies/#windows-support), +[MemoryManager](/docs/tasks/administer-cluster/memory-manager/#windows-support) +and topology manager. \ No newline at end of file diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-graceful-node-shutdown.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-graceful-node-shutdown.md new file mode 100644 index 0000000000000..278b5926ddb1e --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-graceful-node-shutdown.md @@ -0,0 +1,18 @@ +--- +title: WindowsGracefulNodeShutdown +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" + +--- +Enables support for windows node graceful shutdown in kubelet. +During a system shutdown, kubelet will attempt to detect the shutdown event +and gracefully terminate pods running on the node. See +[Graceful Node Shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown) +for more details. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/zero-limited-nominal-concurrency-shares.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/zero-limited-nominal-concurrency-shares.md index c6fb5e8635090..3dcffd4521cdf 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/zero-limited-nominal-concurrency-shares.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/zero-limited-nominal-concurrency-shares.md @@ -13,6 +13,9 @@ stages: - stage: stable defaultValue: true fromVersion: "1.30" + toVersion: "1.31" + +removed: true --- Allow [priority & fairness](/docs/concepts/cluster-administration/flow-control/) in the API server to use a zero value for the `nominalConcurrencyShares` field of diff --git a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md index a283e8c6a40c3..645ed79125920 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md @@ -638,7 +638,7 @@ kube-apiserver [flags] --feature-gates colonSeparatedMultimapStringString -

Comma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.
If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:
kube:APIResponseCompression=true|false (BETA - default=true)
kube:APIServerIdentity=true|false (BETA - default=true)
kube:APIServerTracing=true|false (BETA - default=true)
kube:APIServingWithRoutine=true|false (ALPHA - default=false)
kube:AllAlpha=true|false (ALPHA - default=false)
kube:AllBeta=true|false (BETA - default=false)
kube:AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
kube:AnyVolumeDataSource=true|false (BETA - default=true)
kube:AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
kube:AuthorizeWithSelectors=true|false (ALPHA - default=false)
kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:CPUManagerPolicyOptions=true|false (BETA - default=true)
kube:CRDValidationRatcheting=true|false (BETA - default=true)
kube:CSIMigrationPortworx=true|false (BETA - default=true)
kube:CSIVolumeHealth=true|false (ALPHA - default=false)
kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)
kube:ClusterTrustBundle=true|false (ALPHA - default=false)
kube:ClusterTrustBundleProjection=true|false (ALPHA - default=false)
kube:ComponentSLIs=true|false (BETA - default=true)
kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)
kube:ConsistentListFromCache=true|false (BETA - default=true)
kube:ContainerCheckpoint=true|false (BETA - default=true)
kube:ContextualLogging=true|false (BETA - default=true)
kube:CoordinatedLeaderElection=true|false (ALPHA - default=false)
kube:CronJobsScheduledAnnotation=true|false (BETA - default=true)
kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
kube:CustomResourceFieldSelectors=true|false (BETA - default=true)
kube:DRAControlPlaneController=true|false (ALPHA - default=false)
kube:DisableAllocatorDualWrite=true|false (ALPHA - default=false)
kube:DisableNodeKubeProxyVersion=true|false (BETA - default=true)
kube:DynamicResourceAllocation=true|false (ALPHA - default=false)
kube:EventedPLEG=true|false (ALPHA - default=false)
kube:GracefulNodeShutdown=true|false (BETA - default=true)
kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
kube:HPAScaleToZero=true|false (ALPHA - default=false)
kube:HonorPVReclaimPolicy=true|false (BETA - default=true)
kube:ImageMaximumGCAge=true|false (BETA - default=true)
kube:ImageVolume=true|false (ALPHA - default=false)
kube:InPlacePodVerticalScaling=true|false (ALPHA - default=false)
kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
kube:InformerResourceVersion=true|false (ALPHA - default=false)
kube:JobBackoffLimitPerIndex=true|false (BETA - default=true)
kube:JobManagedBy=true|false (ALPHA - default=false)
kube:JobPodReplacementPolicy=true|false (BETA - default=true)
kube:JobSuccessPolicy=true|false (BETA - default=true)
kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
kube:KubeletInUserNamespace=true|false (ALPHA - default=false)
kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)
kube:KubeletSeparateDiskGC=true|false (BETA - default=true)
kube:KubeletTracing=true|false (BETA - default=true)
kube:LoadBalancerIPMode=true|false (BETA - default=true)
kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
kube:LoggingAlphaOptions=true|false (ALPHA - default=false)
kube:LoggingBetaOptions=true|false (BETA - default=true)
kube:MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
kube:MemoryManager=true|false (BETA - default=true)
kube:MemoryQoS=true|false (ALPHA - default=false)
kube:MultiCIDRServiceAllocator=true|false (BETA - default=false)
kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)
kube:NFTablesProxyMode=true|false (BETA - default=true)
kube:NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
kube:NodeLogQuery=true|false (BETA - default=false)
kube:NodeSwap=true|false (BETA - default=true)
kube:OpenAPIEnums=true|false (BETA - default=true)
kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
kube:PodDeletionCost=true|false (BETA - default=true)
kube:PodIndexLabel=true|false (BETA - default=true)
kube:PodLifecycleSleepAction=true|false (BETA - default=true)
kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)
kube:PortForwardWebsockets=true|false (BETA - default=true)
kube:ProcMountType=true|false (BETA - default=false)
kube:QOSReserved=true|false (ALPHA - default=false)
kube:RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
kube:RecursiveReadOnlyMounts=true|false (BETA - default=true)
kube:RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)
kube:ResourceHealthStatus=true|false (ALPHA - default=false)
kube:RetryGenerateName=true|false (BETA - default=true)
kube:RotateKubeletServerCertificate=true|false (BETA - default=true)
kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
kube:SELinuxMount=true|false (ALPHA - default=false)
kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
kube:SchedulerQueueingHints=true|false (BETA - default=false)
kube:SeparateCacheWatchRPC=true|false (BETA - default=true)
kube:SeparateTaintEvictionController=true|false (BETA - default=true)
kube:ServiceAccountTokenJTI=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
kube:ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
kube:ServiceTrafficDistribution=true|false (BETA - default=true)
kube:SidecarContainers=true|false (BETA - default=true)
kube:SizeMemoryBackedVolumes=true|false (BETA - default=true)
kube:StatefulSetAutoDeletePVC=true|false (BETA - default=true)
kube:StorageNamespaceIndex=true|false (BETA - default=true)
kube:StorageVersionAPI=true|false (ALPHA - default=false)
kube:StorageVersionHash=true|false (BETA - default=true)
kube:StorageVersionMigrator=true|false (ALPHA - default=false)
kube:StrictCostEnforcementForVAP=true|false (BETA - default=false)
kube:StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)
kube:StructuredAuthorizationConfiguration=true|false (BETA - default=true)
kube:SupplementalGroupsPolicy=true|false (ALPHA - default=false)
kube:TopologyAwareHints=true|false (BETA - default=true)
kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:TopologyManagerPolicyOptions=true|false (BETA - default=true)
kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
kube:UserNamespacesSupport=true|false (BETA - default=false)
kube:VolumeAttributesClass=true|false (BETA - default=false)
kube:VolumeCapacityPriority=true|false (ALPHA - default=false)
kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
kube:WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
kube:WatchList=true|false (ALPHA - default=false)
kube:WatchListClient=true|false (BETA - default=false)
kube:WinDSR=true|false (ALPHA - default=false)
kube:WinOverlay=true|false (BETA - default=true)
kube:WindowsHostNetwork=true|false (ALPHA - default=true)

+

Comma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.
If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:
kube:APIResponseCompression=true|false (BETA - default=true)
kube:APIServerIdentity=true|false (BETA - default=true)
kube:APIServerTracing=true|false (BETA - default=true)
kube:APIServingWithRoutine=true|false (ALPHA - default=false)
kube:AllAlpha=true|false (ALPHA - default=false)
kube:AllBeta=true|false (BETA - default=false)
kube:AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
kube:AnyVolumeDataSource=true|false (BETA - default=true)
kube:AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
kube:AuthorizeWithSelectors=true|false (ALPHA - default=false)
kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:CPUManagerPolicyOptions=true|false (BETA - default=true)
kube:CRDValidationRatcheting=true|false (BETA - default=true)
kube:CSIMigrationPortworx=true|false (BETA - default=true)
kube:CSIVolumeHealth=true|false (ALPHA - default=false)
kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)
kube:ClusterTrustBundle=true|false (ALPHA - default=false)
kube:ClusterTrustBundleProjection=true|false (ALPHA - default=false)
kube:ComponentSLIs=true|false (BETA - default=true)
kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)
kube:ConsistentListFromCache=true|false (BETA - default=true)
kube:ContainerCheckpoint=true|false (BETA - default=true)
kube:ContextualLogging=true|false (BETA - default=true)
kube:CoordinatedLeaderElection=true|false (ALPHA - default=false)
kube:CronJobsScheduledAnnotation=true|false (BETA - default=true)
kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
kube:CustomResourceFieldSelectors=true|false (BETA - default=true)
kube:DRAControlPlaneController=true|false (ALPHA - default=false)
kube:DisableAllocatorDualWrite=true|false (ALPHA - default=false)
kube:DisableNodeKubeProxyVersion=true|false (BETA - default=true)
kube:DynamicResourceAllocation=true|false (ALPHA - default=false)
kube:EventedPLEG=true|false (ALPHA - default=false)
kube:GracefulNodeShutdown=true|false (BETA - default=true)
kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
kube:HPAScaleToZero=true|false (ALPHA - default=false)
kube:HonorPVReclaimPolicy=true|false (BETA - default=true)
kube:ImageMaximumGCAge=true|false (BETA - default=true)
kube:ImageVolume=true|false (ALPHA - default=false)
kube:InPlacePodVerticalScaling=true|false (ALPHA - default=false)
kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
kube:InformerResourceVersion=true|false (ALPHA - default=false)
kube:JobBackoffLimitPerIndex=true|false (BETA - default=true)
kube:JobManagedBy=true|false (ALPHA - default=false)
kube:JobPodReplacementPolicy=true|false (BETA - default=true)
kube:JobSuccessPolicy=true|false (BETA - default=true)
kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
kube:KubeletInUserNamespace=true|false (ALPHA - default=false)
kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)
kube:KubeletSeparateDiskGC=true|false (BETA - default=true)
kube:KubeletTracing=true|false (BETA - default=true)
kube:LoadBalancerIPMode=true|false (BETA - default=true)
kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
kube:LoggingAlphaOptions=true|false (ALPHA - default=false)
kube:LoggingBetaOptions=true|false (BETA - default=true)
kube:MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
kube:MemoryManager=true|false (BETA - default=true)
kube:MemoryQoS=true|false (ALPHA - default=false)
kube:MultiCIDRServiceAllocator=true|false (BETA - default=false)
kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)
kube:NFTablesProxyMode=true|false (BETA - default=true)
kube:NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
kube:NodeLogQuery=true|false (BETA - default=false)
kube:NodeSwap=true|false (BETA - default=true)
kube:OpenAPIEnums=true|false (BETA - default=true)
kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
kube:PodDeletionCost=true|false (BETA - default=true)
kube:PodIndexLabel=true|false (BETA - default=true)
kube:PodLifecycleSleepAction=true|false (BETA - default=true)
kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)
kube:PortForwardWebsockets=true|false (BETA - default=true)
kube:ProcMountType=true|false (BETA - default=false)
kube:QOSReserved=true|false (ALPHA - default=false)
kube:RecoverVolumeExpansionFailure=true|false (BETA - default=true)
kube:RecursiveReadOnlyMounts=true|false (BETA - default=true)
kube:RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)
kube:ResourceHealthStatus=true|false (ALPHA - default=false)
kube:RetryGenerateName=true|false (BETA - default=true)
kube:RotateKubeletServerCertificate=true|false (BETA - default=true)
kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
kube:SELinuxMount=true|false (ALPHA - default=false)
kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
kube:SchedulerQueueingHints=true|false (BETA - default=false)
kube:SeparateCacheWatchRPC=true|false (BETA - default=true)
kube:SeparateTaintEvictionController=true|false (BETA - default=true)
kube:ServiceAccountTokenJTI=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
kube:ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
kube:ServiceTrafficDistribution=true|false (BETA - default=true)
kube:SidecarContainers=true|false (BETA - default=true)
kube:SizeMemoryBackedVolumes=true|false (BETA - default=true)
kube:StatefulSetAutoDeletePVC=true|false (BETA - default=true)
kube:StorageNamespaceIndex=true|false (BETA - default=true)
kube:StorageVersionAPI=true|false (ALPHA - default=false)
kube:StorageVersionHash=true|false (BETA - default=true)
kube:StorageVersionMigrator=true|false (ALPHA - default=false)
kube:StrictCostEnforcementForVAP=true|false (BETA - default=false)
kube:StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)
kube:StructuredAuthorizationConfiguration=true|false (BETA - default=true)
kube:SupplementalGroupsPolicy=true|false (ALPHA - default=false)
kube:TopologyAwareHints=true|false (BETA - default=true)
kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:TopologyManagerPolicyOptions=true|false (BETA - default=true)
kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
kube:UserNamespacesSupport=true|false (BETA - default=false)
kube:VolumeAttributesClass=true|false (BETA - default=false)
kube:VolumeCapacityPriority=true|false (ALPHA - default=false)
kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
kube:WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
kube:WatchList=true|false (ALPHA - default=false)
kube:WatchListClient=true|false (BETA - default=false)
kube:WinDSR=true|false (ALPHA - default=false)
kube:WinOverlay=true|false (BETA - default=true)
kube:WindowsHostNetwork=true|false (ALPHA - default=true)

diff --git a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md index 5b915fdbd64f3..ce11df836c14a 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md @@ -495,7 +495,7 @@ kube-controller-manager [flags] --feature-gates colonSeparatedMultimapStringString -

Comma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.
If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:
kube:APIResponseCompression=true|false (BETA - default=true)
kube:APIServerIdentity=true|false (BETA - default=true)
kube:APIServerTracing=true|false (BETA - default=true)
kube:APIServingWithRoutine=true|false (ALPHA - default=false)
kube:AllAlpha=true|false (ALPHA - default=false)
kube:AllBeta=true|false (BETA - default=false)
kube:AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
kube:AnyVolumeDataSource=true|false (BETA - default=true)
kube:AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
kube:AuthorizeWithSelectors=true|false (ALPHA - default=false)
kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:CPUManagerPolicyOptions=true|false (BETA - default=true)
kube:CRDValidationRatcheting=true|false (BETA - default=true)
kube:CSIMigrationPortworx=true|false (BETA - default=true)
kube:CSIVolumeHealth=true|false (ALPHA - default=false)
kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)
kube:ClusterTrustBundle=true|false (ALPHA - default=false)
kube:ClusterTrustBundleProjection=true|false (ALPHA - default=false)
kube:ComponentSLIs=true|false (BETA - default=true)
kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)
kube:ConsistentListFromCache=true|false (BETA - default=true)
kube:ContainerCheckpoint=true|false (BETA - default=true)
kube:ContextualLogging=true|false (BETA - default=true)
kube:CoordinatedLeaderElection=true|false (ALPHA - default=false)
kube:CronJobsScheduledAnnotation=true|false (BETA - default=true)
kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
kube:CustomResourceFieldSelectors=true|false (BETA - default=true)
kube:DRAControlPlaneController=true|false (ALPHA - default=false)
kube:DisableAllocatorDualWrite=true|false (ALPHA - default=false)
kube:DisableNodeKubeProxyVersion=true|false (BETA - default=true)
kube:DynamicResourceAllocation=true|false (ALPHA - default=false)
kube:EventedPLEG=true|false (ALPHA - default=false)
kube:GracefulNodeShutdown=true|false (BETA - default=true)
kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
kube:HPAScaleToZero=true|false (ALPHA - default=false)
kube:HonorPVReclaimPolicy=true|false (BETA - default=true)
kube:ImageMaximumGCAge=true|false (BETA - default=true)
kube:ImageVolume=true|false (ALPHA - default=false)
kube:InPlacePodVerticalScaling=true|false (ALPHA - default=false)
kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
kube:InformerResourceVersion=true|false (ALPHA - default=false)
kube:JobBackoffLimitPerIndex=true|false (BETA - default=true)
kube:JobManagedBy=true|false (ALPHA - default=false)
kube:JobPodReplacementPolicy=true|false (BETA - default=true)
kube:JobSuccessPolicy=true|false (BETA - default=true)
kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
kube:KubeletInUserNamespace=true|false (ALPHA - default=false)
kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)
kube:KubeletSeparateDiskGC=true|false (BETA - default=true)
kube:KubeletTracing=true|false (BETA - default=true)
kube:LoadBalancerIPMode=true|false (BETA - default=true)
kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
kube:LoggingAlphaOptions=true|false (ALPHA - default=false)
kube:LoggingBetaOptions=true|false (BETA - default=true)
kube:MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
kube:MemoryManager=true|false (BETA - default=true)
kube:MemoryQoS=true|false (ALPHA - default=false)
kube:MultiCIDRServiceAllocator=true|false (BETA - default=false)
kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)
kube:NFTablesProxyMode=true|false (BETA - default=true)
kube:NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
kube:NodeLogQuery=true|false (BETA - default=false)
kube:NodeSwap=true|false (BETA - default=true)
kube:OpenAPIEnums=true|false (BETA - default=true)
kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
kube:PodDeletionCost=true|false (BETA - default=true)
kube:PodIndexLabel=true|false (BETA - default=true)
kube:PodLifecycleSleepAction=true|false (BETA - default=true)
kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)
kube:PortForwardWebsockets=true|false (BETA - default=true)
kube:ProcMountType=true|false (BETA - default=false)
kube:QOSReserved=true|false (ALPHA - default=false)
kube:RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
kube:RecursiveReadOnlyMounts=true|false (BETA - default=true)
kube:RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)
kube:ResourceHealthStatus=true|false (ALPHA - default=false)
kube:RetryGenerateName=true|false (BETA - default=true)
kube:RotateKubeletServerCertificate=true|false (BETA - default=true)
kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
kube:SELinuxMount=true|false (ALPHA - default=false)
kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
kube:SchedulerQueueingHints=true|false (BETA - default=false)
kube:SeparateCacheWatchRPC=true|false (BETA - default=true)
kube:SeparateTaintEvictionController=true|false (BETA - default=true)
kube:ServiceAccountTokenJTI=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
kube:ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
kube:ServiceTrafficDistribution=true|false (BETA - default=true)
kube:SidecarContainers=true|false (BETA - default=true)
kube:SizeMemoryBackedVolumes=true|false (BETA - default=true)
kube:StatefulSetAutoDeletePVC=true|false (BETA - default=true)
kube:StorageNamespaceIndex=true|false (BETA - default=true)
kube:StorageVersionAPI=true|false (ALPHA - default=false)
kube:StorageVersionHash=true|false (BETA - default=true)
kube:StorageVersionMigrator=true|false (ALPHA - default=false)
kube:StrictCostEnforcementForVAP=true|false (BETA - default=false)
kube:StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)
kube:StructuredAuthorizationConfiguration=true|false (BETA - default=true)
kube:SupplementalGroupsPolicy=true|false (ALPHA - default=false)
kube:TopologyAwareHints=true|false (BETA - default=true)
kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:TopologyManagerPolicyOptions=true|false (BETA - default=true)
kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
kube:UserNamespacesSupport=true|false (BETA - default=false)
kube:VolumeAttributesClass=true|false (BETA - default=false)
kube:VolumeCapacityPriority=true|false (ALPHA - default=false)
kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
kube:WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
kube:WatchList=true|false (ALPHA - default=false)
kube:WatchListClient=true|false (BETA - default=false)
kube:WinDSR=true|false (ALPHA - default=false)
kube:WinOverlay=true|false (BETA - default=true)
kube:WindowsHostNetwork=true|false (ALPHA - default=true)

+

Comma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.
If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:
kube:APIResponseCompression=true|false (BETA - default=true)
kube:APIServerIdentity=true|false (BETA - default=true)
kube:APIServerTracing=true|false (BETA - default=true)
kube:APIServingWithRoutine=true|false (ALPHA - default=false)
kube:AllAlpha=true|false (ALPHA - default=false)
kube:AllBeta=true|false (BETA - default=false)
kube:AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
kube:AnyVolumeDataSource=true|false (BETA - default=true)
kube:AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
kube:AuthorizeWithSelectors=true|false (ALPHA - default=false)
kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:CPUManagerPolicyOptions=true|false (BETA - default=true)
kube:CRDValidationRatcheting=true|false (BETA - default=true)
kube:CSIMigrationPortworx=true|false (BETA - default=true)
kube:CSIVolumeHealth=true|false (ALPHA - default=false)
kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)
kube:ClusterTrustBundle=true|false (ALPHA - default=false)
kube:ClusterTrustBundleProjection=true|false (ALPHA - default=false)
kube:ComponentSLIs=true|false (BETA - default=true)
kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)
kube:ConsistentListFromCache=true|false (BETA - default=true)
kube:ContainerCheckpoint=true|false (BETA - default=true)
kube:ContextualLogging=true|false (BETA - default=true)
kube:CoordinatedLeaderElection=true|false (ALPHA - default=false)
kube:CronJobsScheduledAnnotation=true|false (BETA - default=true)
kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
kube:CustomResourceFieldSelectors=true|false (BETA - default=true)
kube:DRAControlPlaneController=true|false (ALPHA - default=false)
kube:DisableAllocatorDualWrite=true|false (ALPHA - default=false)
kube:DisableNodeKubeProxyVersion=true|false (BETA - default=true)
kube:DynamicResourceAllocation=true|false (ALPHA - default=false)
kube:EventedPLEG=true|false (ALPHA - default=false)
kube:GracefulNodeShutdown=true|false (BETA - default=true)
kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
kube:HPAScaleToZero=true|false (ALPHA - default=false)
kube:HonorPVReclaimPolicy=true|false (BETA - default=true)
kube:ImageMaximumGCAge=true|false (BETA - default=true)
kube:ImageVolume=true|false (ALPHA - default=false)
kube:InPlacePodVerticalScaling=true|false (ALPHA - default=false)
kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
kube:InformerResourceVersion=true|false (ALPHA - default=false)
kube:JobBackoffLimitPerIndex=true|false (BETA - default=true)
kube:JobManagedBy=true|false (ALPHA - default=false)
kube:JobPodReplacementPolicy=true|false (BETA - default=true)
kube:JobSuccessPolicy=true|false (BETA - default=true)
kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
kube:KubeletInUserNamespace=true|false (ALPHA - default=false)
kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)
kube:KubeletSeparateDiskGC=true|false (BETA - default=true)
kube:KubeletTracing=true|false (BETA - default=true)
kube:LoadBalancerIPMode=true|false (BETA - default=true)
kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
kube:LoggingAlphaOptions=true|false (ALPHA - default=false)
kube:LoggingBetaOptions=true|false (BETA - default=true)
kube:MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
kube:MemoryManager=true|false (BETA - default=true)
kube:MemoryQoS=true|false (ALPHA - default=false)
kube:MultiCIDRServiceAllocator=true|false (BETA - default=false)
kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)
kube:NFTablesProxyMode=true|false (BETA - default=true)
kube:NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
kube:NodeLogQuery=true|false (BETA - default=false)
kube:NodeSwap=true|false (BETA - default=true)
kube:OpenAPIEnums=true|false (BETA - default=true)
kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
kube:PodDeletionCost=true|false (BETA - default=true)
kube:PodIndexLabel=true|false (BETA - default=true)
kube:PodLifecycleSleepAction=true|false (BETA - default=true)
kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)
kube:PortForwardWebsockets=true|false (BETA - default=true)
kube:ProcMountType=true|false (BETA - default=false)
kube:QOSReserved=true|false (ALPHA - default=false)
kube:RecoverVolumeExpansionFailure=true|false (BETA - default=true)
kube:RecursiveReadOnlyMounts=true|false (BETA - default=true)
kube:RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)
kube:ResourceHealthStatus=true|false (ALPHA - default=false)
kube:RetryGenerateName=true|false (BETA - default=true)
kube:RotateKubeletServerCertificate=true|false (BETA - default=true)
kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
kube:SELinuxMount=true|false (ALPHA - default=false)
kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
kube:SchedulerQueueingHints=true|false (BETA - default=false)
kube:SeparateCacheWatchRPC=true|false (BETA - default=true)
kube:SeparateTaintEvictionController=true|false (BETA - default=true)
kube:ServiceAccountTokenJTI=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
kube:ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
kube:ServiceTrafficDistribution=true|false (BETA - default=true)
kube:SidecarContainers=true|false (BETA - default=true)
kube:SizeMemoryBackedVolumes=true|false (BETA - default=true)
kube:StatefulSetAutoDeletePVC=true|false (BETA - default=true)
kube:StorageNamespaceIndex=true|false (BETA - default=true)
kube:StorageVersionAPI=true|false (ALPHA - default=false)
kube:StorageVersionHash=true|false (BETA - default=true)
kube:StorageVersionMigrator=true|false (ALPHA - default=false)
kube:StrictCostEnforcementForVAP=true|false (BETA - default=false)
kube:StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)
kube:StructuredAuthorizationConfiguration=true|false (BETA - default=true)
kube:SupplementalGroupsPolicy=true|false (ALPHA - default=false)
kube:TopologyAwareHints=true|false (BETA - default=true)
kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:TopologyManagerPolicyOptions=true|false (BETA - default=true)
kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
kube:UserNamespacesSupport=true|false (BETA - default=false)
kube:VolumeAttributesClass=true|false (BETA - default=false)
kube:VolumeCapacityPriority=true|false (ALPHA - default=false)
kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
kube:WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
kube:WatchList=true|false (ALPHA - default=false)
kube:WatchListClient=true|false (BETA - default=false)
kube:WinDSR=true|false (ALPHA - default=false)
kube:WinOverlay=true|false (BETA - default=true)
kube:WindowsHostNetwork=true|false (ALPHA - default=true)

diff --git a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md index 92dd1488ba12c..f470a40daccb9 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md @@ -158,7 +158,7 @@ kube-proxy [flags] --feature-gates <comma-separated 'key=True|False' pairs> -

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (BETA - default=true)
APIServerTracing=true|false (BETA - default=true)
APIServingWithRoutine=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
AuthorizeWithSelectors=true|false (ALPHA - default=false)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CRDValidationRatcheting=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
CloudControllerManagerWebhook=true|false (ALPHA - default=false)
ClusterTrustBundle=true|false (ALPHA - default=false)
ClusterTrustBundleProjection=true|false (ALPHA - default=false)
ComponentSLIs=true|false (BETA - default=true)
ConcurrentWatchObjectDecode=true|false (BETA - default=false)
ConsistentListFromCache=true|false (BETA - default=true)
ContainerCheckpoint=true|false (BETA - default=true)
ContextualLogging=true|false (BETA - default=true)
CoordinatedLeaderElection=true|false (ALPHA - default=false)
CronJobsScheduledAnnotation=true|false (BETA - default=true)
CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceFieldSelectors=true|false (BETA - default=true)
DRAControlPlaneController=true|false (ALPHA - default=false)
DisableAllocatorDualWrite=true|false (ALPHA - default=false)
DisableNodeKubeProxyVersion=true|false (BETA - default=true)
DynamicResourceAllocation=true|false (ALPHA - default=false)
EventedPLEG=true|false (ALPHA - default=false)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (BETA - default=true)
ImageMaximumGCAge=true|false (BETA - default=true)
ImageVolume=true|false (ALPHA - default=false)
InPlacePodVerticalScaling=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InformerResourceVersion=true|false (ALPHA - default=false)
JobBackoffLimitPerIndex=true|false (BETA - default=true)
JobManagedBy=true|false (ALPHA - default=false)
JobPodReplacementPolicy=true|false (BETA - default=true)
JobSuccessPolicy=true|false (BETA - default=true)
KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
KubeletPodResourcesGet=true|false (ALPHA - default=false)
KubeletSeparateDiskGC=true|false (BETA - default=true)
KubeletTracing=true|false (BETA - default=true)
LoadBalancerIPMode=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
LoggingAlphaOptions=true|false (ALPHA - default=false)
LoggingBetaOptions=true|false (BETA - default=true)
MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MultiCIDRServiceAllocator=true|false (BETA - default=false)
MutatingAdmissionPolicy=true|false (ALPHA - default=false)
NFTablesProxyMode=true|false (BETA - default=true)
NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
NodeLogQuery=true|false (BETA - default=false)
NodeSwap=true|false (BETA - default=true)
OpenAPIEnums=true|false (BETA - default=true)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodIndexLabel=true|false (BETA - default=true)
PodLifecycleSleepAction=true|false (BETA - default=true)
PodReadyToStartContainersCondition=true|false (BETA - default=true)
PortForwardWebsockets=true|false (BETA - default=true)
ProcMountType=true|false (BETA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RecursiveReadOnlyMounts=true|false (BETA - default=true)
RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
ResilientWatchCacheInitialization=true|false (BETA - default=true)
ResourceHealthStatus=true|false (ALPHA - default=false)
RetryGenerateName=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
SELinuxMount=true|false (ALPHA - default=false)
SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
SchedulerQueueingHints=true|false (BETA - default=false)
SeparateCacheWatchRPC=true|false (BETA - default=true)
SeparateTaintEvictionController=true|false (BETA - default=true)
ServiceAccountTokenJTI=true|false (BETA - default=true)
ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
ServiceTrafficDistribution=true|false (BETA - default=true)
SidecarContainers=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (BETA - default=true)
StorageNamespaceIndex=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
StorageVersionMigrator=true|false (ALPHA - default=false)
StrictCostEnforcementForVAP=true|false (BETA - default=false)
StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
StructuredAuthenticationConfiguration=true|false (BETA - default=true)
StructuredAuthorizationConfiguration=true|false (BETA - default=true)
SupplementalGroupsPolicy=true|false (ALPHA - default=false)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
TopologyManagerPolicyOptions=true|false (BETA - default=true)
TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
UserNamespacesSupport=true|false (BETA - default=false)
VolumeAttributesClass=true|false (BETA - default=false)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
WatchList=true|false (ALPHA - default=false)
WatchListClient=true|false (BETA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostNetwork=true|false (ALPHA - default=true)
This parameter is ignored if a config file is specified by --config.

+

A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIResponseCompression=true|false (BETA - default=true)
APIServerIdentity=true|false (BETA - default=true)
APIServerTracing=true|false (BETA - default=true)
APIServingWithRoutine=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
AuthorizeWithSelectors=true|false (ALPHA - default=false)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CRDValidationRatcheting=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (BETA - default=true)
CSIVolumeHealth=true|false (ALPHA - default=false)
CloudControllerManagerWebhook=true|false (ALPHA - default=false)
ClusterTrustBundle=true|false (ALPHA - default=false)
ClusterTrustBundleProjection=true|false (ALPHA - default=false)
ComponentSLIs=true|false (BETA - default=true)
ConcurrentWatchObjectDecode=true|false (BETA - default=false)
ConsistentListFromCache=true|false (BETA - default=true)
ContainerCheckpoint=true|false (BETA - default=true)
ContextualLogging=true|false (BETA - default=true)
CoordinatedLeaderElection=true|false (ALPHA - default=false)
CronJobsScheduledAnnotation=true|false (BETA - default=true)
CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceFieldSelectors=true|false (BETA - default=true)
DRAControlPlaneController=true|false (ALPHA - default=false)
DisableAllocatorDualWrite=true|false (ALPHA - default=false)
DisableNodeKubeProxyVersion=true|false (BETA - default=true)
DynamicResourceAllocation=true|false (ALPHA - default=false)
EventedPLEG=true|false (ALPHA - default=false)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (BETA - default=true)
ImageMaximumGCAge=true|false (BETA - default=true)
ImageVolume=true|false (ALPHA - default=false)
InPlacePodVerticalScaling=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InformerResourceVersion=true|false (ALPHA - default=false)
JobBackoffLimitPerIndex=true|false (BETA - default=true)
JobManagedBy=true|false (ALPHA - default=false)
JobPodReplacementPolicy=true|false (BETA - default=true)
JobSuccessPolicy=true|false (BETA - default=true)
KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
KubeletPodResourcesGet=true|false (ALPHA - default=false)
KubeletSeparateDiskGC=true|false (BETA - default=true)
KubeletTracing=true|false (BETA - default=true)
LoadBalancerIPMode=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
LoggingAlphaOptions=true|false (ALPHA - default=false)
LoggingBetaOptions=true|false (BETA - default=true)
MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MultiCIDRServiceAllocator=true|false (BETA - default=false)
MutatingAdmissionPolicy=true|false (ALPHA - default=false)
NFTablesProxyMode=true|false (BETA - default=true)
NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
NodeLogQuery=true|false (BETA - default=false)
NodeSwap=true|false (BETA - default=true)
OpenAPIEnums=true|false (BETA - default=true)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodIndexLabel=true|false (BETA - default=true)
PodLifecycleSleepAction=true|false (BETA - default=true)
PodReadyToStartContainersCondition=true|false (BETA - default=true)
PortForwardWebsockets=true|false (BETA - default=true)
ProcMountType=true|false (BETA - default=false)
QOSReserved=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (BETA - default=true)
RecursiveReadOnlyMounts=true|false (BETA - default=true)
RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
ResilientWatchCacheInitialization=true|false (BETA - default=true)
ResourceHealthStatus=true|false (ALPHA - default=false)
RetryGenerateName=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
SELinuxMount=true|false (ALPHA - default=false)
SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
SchedulerQueueingHints=true|false (BETA - default=false)
SeparateCacheWatchRPC=true|false (BETA - default=true)
SeparateTaintEvictionController=true|false (BETA - default=true)
ServiceAccountTokenJTI=true|false (BETA - default=true)
ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
ServiceTrafficDistribution=true|false (BETA - default=true)
SidecarContainers=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (BETA - default=true)
StorageNamespaceIndex=true|false (BETA - default=true)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
StorageVersionMigrator=true|false (ALPHA - default=false)
StrictCostEnforcementForVAP=true|false (BETA - default=false)
StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
StructuredAuthenticationConfiguration=true|false (BETA - default=true)
StructuredAuthorizationConfiguration=true|false (BETA - default=true)
SupplementalGroupsPolicy=true|false (ALPHA - default=false)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
TopologyManagerPolicyOptions=true|false (BETA - default=true)
TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
UserNamespacesSupport=true|false (BETA - default=false)
VolumeAttributesClass=true|false (BETA - default=false)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
WatchList=true|false (ALPHA - default=false)
WatchListClient=true|false (BETA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostNetwork=true|false (ALPHA - default=true)
This parameter is ignored if a config file is specified by --config.

diff --git a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md index ab374b1242362..23fc29d7d3489 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md @@ -173,7 +173,7 @@ kube-scheduler [flags] --feature-gates colonSeparatedMultimapStringString -

Comma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.
If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:
kube:APIResponseCompression=true|false (BETA - default=true)
kube:APIServerIdentity=true|false (BETA - default=true)
kube:APIServerTracing=true|false (BETA - default=true)
kube:APIServingWithRoutine=true|false (ALPHA - default=false)
kube:AllAlpha=true|false (ALPHA - default=false)
kube:AllBeta=true|false (BETA - default=false)
kube:AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
kube:AnyVolumeDataSource=true|false (BETA - default=true)
kube:AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
kube:AuthorizeWithSelectors=true|false (ALPHA - default=false)
kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:CPUManagerPolicyOptions=true|false (BETA - default=true)
kube:CRDValidationRatcheting=true|false (BETA - default=true)
kube:CSIMigrationPortworx=true|false (BETA - default=true)
kube:CSIVolumeHealth=true|false (ALPHA - default=false)
kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)
kube:ClusterTrustBundle=true|false (ALPHA - default=false)
kube:ClusterTrustBundleProjection=true|false (ALPHA - default=false)
kube:ComponentSLIs=true|false (BETA - default=true)
kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)
kube:ConsistentListFromCache=true|false (BETA - default=true)
kube:ContainerCheckpoint=true|false (BETA - default=true)
kube:ContextualLogging=true|false (BETA - default=true)
kube:CoordinatedLeaderElection=true|false (ALPHA - default=false)
kube:CronJobsScheduledAnnotation=true|false (BETA - default=true)
kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
kube:CustomResourceFieldSelectors=true|false (BETA - default=true)
kube:DRAControlPlaneController=true|false (ALPHA - default=false)
kube:DisableAllocatorDualWrite=true|false (ALPHA - default=false)
kube:DisableNodeKubeProxyVersion=true|false (BETA - default=true)
kube:DynamicResourceAllocation=true|false (ALPHA - default=false)
kube:EventedPLEG=true|false (ALPHA - default=false)
kube:GracefulNodeShutdown=true|false (BETA - default=true)
kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
kube:HPAScaleToZero=true|false (ALPHA - default=false)
kube:HonorPVReclaimPolicy=true|false (BETA - default=true)
kube:ImageMaximumGCAge=true|false (BETA - default=true)
kube:ImageVolume=true|false (ALPHA - default=false)
kube:InPlacePodVerticalScaling=true|false (ALPHA - default=false)
kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
kube:InformerResourceVersion=true|false (ALPHA - default=false)
kube:JobBackoffLimitPerIndex=true|false (BETA - default=true)
kube:JobManagedBy=true|false (ALPHA - default=false)
kube:JobPodReplacementPolicy=true|false (BETA - default=true)
kube:JobSuccessPolicy=true|false (BETA - default=true)
kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
kube:KubeletInUserNamespace=true|false (ALPHA - default=false)
kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)
kube:KubeletSeparateDiskGC=true|false (BETA - default=true)
kube:KubeletTracing=true|false (BETA - default=true)
kube:LoadBalancerIPMode=true|false (BETA - default=true)
kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
kube:LoggingAlphaOptions=true|false (ALPHA - default=false)
kube:LoggingBetaOptions=true|false (BETA - default=true)
kube:MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
kube:MemoryManager=true|false (BETA - default=true)
kube:MemoryQoS=true|false (ALPHA - default=false)
kube:MultiCIDRServiceAllocator=true|false (BETA - default=false)
kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)
kube:NFTablesProxyMode=true|false (BETA - default=true)
kube:NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
kube:NodeLogQuery=true|false (BETA - default=false)
kube:NodeSwap=true|false (BETA - default=true)
kube:OpenAPIEnums=true|false (BETA - default=true)
kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
kube:PodDeletionCost=true|false (BETA - default=true)
kube:PodIndexLabel=true|false (BETA - default=true)
kube:PodLifecycleSleepAction=true|false (BETA - default=true)
kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)
kube:PortForwardWebsockets=true|false (BETA - default=true)
kube:ProcMountType=true|false (BETA - default=false)
kube:QOSReserved=true|false (ALPHA - default=false)
kube:RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
kube:RecursiveReadOnlyMounts=true|false (BETA - default=true)
kube:RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)
kube:ResourceHealthStatus=true|false (ALPHA - default=false)
kube:RetryGenerateName=true|false (BETA - default=true)
kube:RotateKubeletServerCertificate=true|false (BETA - default=true)
kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
kube:SELinuxMount=true|false (ALPHA - default=false)
kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
kube:SchedulerQueueingHints=true|false (BETA - default=false)
kube:SeparateCacheWatchRPC=true|false (BETA - default=true)
kube:SeparateTaintEvictionController=true|false (BETA - default=true)
kube:ServiceAccountTokenJTI=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
kube:ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
kube:ServiceTrafficDistribution=true|false (BETA - default=true)
kube:SidecarContainers=true|false (BETA - default=true)
kube:SizeMemoryBackedVolumes=true|false (BETA - default=true)
kube:StatefulSetAutoDeletePVC=true|false (BETA - default=true)
kube:StorageNamespaceIndex=true|false (BETA - default=true)
kube:StorageVersionAPI=true|false (ALPHA - default=false)
kube:StorageVersionHash=true|false (BETA - default=true)
kube:StorageVersionMigrator=true|false (ALPHA - default=false)
kube:StrictCostEnforcementForVAP=true|false (BETA - default=false)
kube:StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)
kube:StructuredAuthorizationConfiguration=true|false (BETA - default=true)
kube:SupplementalGroupsPolicy=true|false (ALPHA - default=false)
kube:TopologyAwareHints=true|false (BETA - default=true)
kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:TopologyManagerPolicyOptions=true|false (BETA - default=true)
kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
kube:UserNamespacesSupport=true|false (BETA - default=false)
kube:VolumeAttributesClass=true|false (BETA - default=false)
kube:VolumeCapacityPriority=true|false (ALPHA - default=false)
kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
kube:WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
kube:WatchList=true|false (ALPHA - default=false)
kube:WatchListClient=true|false (BETA - default=false)
kube:WinDSR=true|false (ALPHA - default=false)
kube:WinOverlay=true|false (BETA - default=true)
kube:WindowsHostNetwork=true|false (ALPHA - default=true)

+

Comma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.
If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:
kube:APIResponseCompression=true|false (BETA - default=true)
kube:APIServerIdentity=true|false (BETA - default=true)
kube:APIServerTracing=true|false (BETA - default=true)
kube:APIServingWithRoutine=true|false (ALPHA - default=false)
kube:AllAlpha=true|false (ALPHA - default=false)
kube:AllBeta=true|false (BETA - default=false)
kube:AnonymousAuthConfigurableEndpoints=true|false (ALPHA - default=false)
kube:AnyVolumeDataSource=true|false (BETA - default=true)
kube:AuthorizeNodeWithSelectors=true|false (ALPHA - default=false)
kube:AuthorizeWithSelectors=true|false (ALPHA - default=false)
kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:CPUManagerPolicyOptions=true|false (BETA - default=true)
kube:CRDValidationRatcheting=true|false (BETA - default=true)
kube:CSIMigrationPortworx=true|false (BETA - default=true)
kube:CSIVolumeHealth=true|false (ALPHA - default=false)
kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)
kube:ClusterTrustBundle=true|false (ALPHA - default=false)
kube:ClusterTrustBundleProjection=true|false (ALPHA - default=false)
kube:ComponentSLIs=true|false (BETA - default=true)
kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)
kube:ConsistentListFromCache=true|false (BETA - default=true)
kube:ContainerCheckpoint=true|false (BETA - default=true)
kube:ContextualLogging=true|false (BETA - default=true)
kube:CoordinatedLeaderElection=true|false (ALPHA - default=false)
kube:CronJobsScheduledAnnotation=true|false (BETA - default=true)
kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
kube:CustomResourceFieldSelectors=true|false (BETA - default=true)
kube:DRAControlPlaneController=true|false (ALPHA - default=false)
kube:DisableAllocatorDualWrite=true|false (ALPHA - default=false)
kube:DisableNodeKubeProxyVersion=true|false (BETA - default=true)
kube:DynamicResourceAllocation=true|false (ALPHA - default=false)
kube:EventedPLEG=true|false (ALPHA - default=false)
kube:GracefulNodeShutdown=true|false (BETA - default=true)
kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
kube:HPAScaleToZero=true|false (ALPHA - default=false)
kube:HonorPVReclaimPolicy=true|false (BETA - default=true)
kube:ImageMaximumGCAge=true|false (BETA - default=true)
kube:ImageVolume=true|false (ALPHA - default=false)
kube:InPlacePodVerticalScaling=true|false (ALPHA - default=false)
kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
kube:InformerResourceVersion=true|false (ALPHA - default=false)
kube:JobBackoffLimitPerIndex=true|false (BETA - default=true)
kube:JobManagedBy=true|false (ALPHA - default=false)
kube:JobPodReplacementPolicy=true|false (BETA - default=true)
kube:JobSuccessPolicy=true|false (BETA - default=true)
kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)
kube:KubeletInUserNamespace=true|false (ALPHA - default=false)
kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)
kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)
kube:KubeletSeparateDiskGC=true|false (BETA - default=true)
kube:KubeletTracing=true|false (BETA - default=true)
kube:LoadBalancerIPMode=true|false (BETA - default=true)
kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)
kube:LoggingAlphaOptions=true|false (ALPHA - default=false)
kube:LoggingBetaOptions=true|false (BETA - default=true)
kube:MatchLabelKeysInPodAffinity=true|false (BETA - default=true)
kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
kube:MemoryManager=true|false (BETA - default=true)
kube:MemoryQoS=true|false (ALPHA - default=false)
kube:MultiCIDRServiceAllocator=true|false (BETA - default=false)
kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)
kube:NFTablesProxyMode=true|false (BETA - default=true)
kube:NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
kube:NodeLogQuery=true|false (BETA - default=false)
kube:NodeSwap=true|false (BETA - default=true)
kube:OpenAPIEnums=true|false (BETA - default=true)
kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
kube:PodDeletionCost=true|false (BETA - default=true)
kube:PodIndexLabel=true|false (BETA - default=true)
kube:PodLifecycleSleepAction=true|false (BETA - default=true)
kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)
kube:PortForwardWebsockets=true|false (BETA - default=true)
kube:ProcMountType=true|false (BETA - default=false)
kube:QOSReserved=true|false (ALPHA - default=false)
kube:RecoverVolumeExpansionFailure=true|false (BETA - default=true)
kube:RecursiveReadOnlyMounts=true|false (BETA - default=true)
kube:RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)
kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)
kube:ResourceHealthStatus=true|false (ALPHA - default=false)
kube:RetryGenerateName=true|false (BETA - default=true)
kube:RotateKubeletServerCertificate=true|false (BETA - default=true)
kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)
kube:SELinuxMount=true|false (ALPHA - default=false)
kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)
kube:SchedulerQueueingHints=true|false (BETA - default=false)
kube:SeparateCacheWatchRPC=true|false (BETA - default=true)
kube:SeparateTaintEvictionController=true|false (BETA - default=true)
kube:ServiceAccountTokenJTI=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBinding=true|false (BETA - default=true)
kube:ServiceAccountTokenNodeBindingValidation=true|false (BETA - default=true)
kube:ServiceAccountTokenPodNodeInfo=true|false (BETA - default=true)
kube:ServiceTrafficDistribution=true|false (BETA - default=true)
kube:SidecarContainers=true|false (BETA - default=true)
kube:SizeMemoryBackedVolumes=true|false (BETA - default=true)
kube:StatefulSetAutoDeletePVC=true|false (BETA - default=true)
kube:StorageNamespaceIndex=true|false (BETA - default=true)
kube:StorageVersionAPI=true|false (ALPHA - default=false)
kube:StorageVersionHash=true|false (BETA - default=true)
kube:StorageVersionMigrator=true|false (ALPHA - default=false)
kube:StrictCostEnforcementForVAP=true|false (BETA - default=false)
kube:StrictCostEnforcementForWebhooks=true|false (BETA - default=false)
kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)
kube:StructuredAuthorizationConfiguration=true|false (BETA - default=true)
kube:SupplementalGroupsPolicy=true|false (ALPHA - default=false)
kube:TopologyAwareHints=true|false (BETA - default=true)
kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)
kube:TopologyManagerPolicyOptions=true|false (BETA - default=true)
kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)
kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)
kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)
kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)
kube:UserNamespacesSupport=true|false (BETA - default=false)
kube:VolumeAttributesClass=true|false (BETA - default=false)
kube:VolumeCapacityPriority=true|false (ALPHA - default=false)
kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)
kube:WatchFromStorageWithoutResourceVersion=true|false (BETA - default=false)
kube:WatchList=true|false (ALPHA - default=false)
kube:WatchListClient=true|false (BETA - default=false)
kube:WinDSR=true|false (ALPHA - default=false)
kube:WinOverlay=true|false (BETA - default=true)
kube:WindowsHostNetwork=true|false (ALPHA - default=true)

diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index f410293c12e4a..129b9a55f840c 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -426,7 +426,6 @@ LoggingBetaOptions=true|false (BETA - default=true)
MatchLabelKeysInPodAffinity=true|false (ALPHA - default=false)
MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
-MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MultiCIDRServiceAllocator=true|false (ALPHA - default=false)
MutatingAdmissionPolicy=true|false (ALPHA - default=false)
@@ -446,7 +445,7 @@ PodReadyToStartContainersCondition=true|false (BETA - default=true)
PortForwardWebsockets=true|false (ALPHA - default=false)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
-RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
+RecoverVolumeExpansionFailure=true|false (BETA - default=true)
RecursiveReadOnlyMounts=true|false (ALPHA - default=false)
RelaxedEnvironmentVariableValidation=true|false (ALPHA - default=false)
RetryGenerateName=true|false (ALPHA - default=false)
diff --git a/content/en/docs/reference/instrumentation/metrics.md b/content/en/docs/reference/instrumentation/metrics.md index 6acaf5eecc6ca..bb64782fa535b 100644 --- a/content/en/docs/reference/instrumentation/metrics.md +++ b/content/en/docs/reference/instrumentation/metrics.md @@ -6,10 +6,10 @@ description: >- Details of the metric data that Kubernetes components export. --- -## Metrics (v1.31) +## Metrics (v1.32) - - + + This page details the metrics that different Kubernetes components export. You can query the metrics endpoint for these components using an HTTP scrape, and fetch the current metrics data in Prometheus format. @@ -495,7 +495,7 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
  • operationrejectedtype
    • apiserver_admission_webhook_fail_open_count
    -
    Admission webhook fail open count, identified by name and broken out for each admission type (validating or mutating).
    +
    Admission webhook fail open count, identified by name and broken out for each admission type (validating or admit).
    • ALPHA
    • Counter
      • @@ -509,7 +509,7 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • error_typenameoperationrejection_codetype
    apiserver_admission_webhook_request_total
    -
    Admission webhook request total, identified by name and broken out for each admission type (validating or mutating) and operation. Additional labels specify whether the request was rejected or not and an HTTP status code. Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
    +
    Admission webhook request total, identified by name and broken out for each admission type (validating or admit) and operation. Additional labels specify whether the request was rejected or not and an HTTP status code. Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
    • ALPHA
    • Counter
      • @@ -844,6 +844,41 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Histogram
    • grpc_status_codemethod_nameprovider_name
    +
    apiserver_externaljwt_fetch_keys_data_timestamp
    +
    Unix Timestamp in seconds of the last successful FetchKeys data_timestamp value returned by the external signer
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    +
    +
    apiserver_externaljwt_fetch_keys_request_total
    +
    Total attempts at syncing supported JWKs
    +
      +
    • ALPHA
      • +
    • Counter
      • +
    • code
    +
    +
    apiserver_externaljwt_fetch_keys_success_timestamp
    +
    Unix Timestamp in seconds of the last successful FetchKeys request
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    +
    +
    apiserver_externaljwt_request_duration_seconds
    +
    Request duration and time for calls to external-jwt-signer
    +
      +
    • ALPHA
      • +
    • Histogram
      • +
    • codemethod
    +
    +
    apiserver_externaljwt_sign_request_total
    +
    Total attempts at signing JWT
    +
      +
    • ALPHA
      • +
    • Counter
      • +
    • code
    +
    apiserver_flowcontrol_current_inqueue_seats
    Number of seats currently pending in queues of the API Priority and Fairness subsystem
      @@ -1216,11 +1251,11 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • transformation_typetransformer_prefix
    apiserver_storage_transformation_operations_total
    -
    Total number of transformations. Successful transformation will have a status 'OK' and a varied status string when the transformation fails. This status and transformation_type fields may be used for alerting on encryption/decryption failure using transformation_type from_storage for decryption and to_storage for encryption
    +
    Total number of transformations. Successful transformation will have a status 'OK' and a varied status string when the transformation fails. The status, resource, and transformation_type fields can be used for alerting purposes. For example, you can monitor for encryption/decryption failures using the transformation_type (e.g., from_storage for decryption and to_storage for encryption). Additionally, these fields can be used to ensure that the correct transformers are applied to each resource.
    • ALPHA
    • Counter
      • -
    • statustransformation_typetransformer_prefix
    +
  • resourcestatustransformation_typetransformer_prefix
    • apiserver_stream_translator_requests_total
    Total number of requests that were handled by the StreamTranslatorProxy, which processes streaming RemoteCommand/V5
    @@ -1418,6 +1453,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
  • Counter
  • codewebhook
    • +
    clustertrustbundle_publisher_sync_duration_seconds
    +
    The time it took to sync a cluster trust bundle.
    +
      +
    • ALPHA
      • +
    • Histogram
      • +
    • code
    +
    +
    clustertrustbundle_publisher_sync_total
    +
    Number of syncs that occurred in cluster trust bundle publisher.
    +
      +
    • ALPHA
      • +
    • Counter
      • +
    • code
    +
    container_swap_usage_bytes
    Current amount of the container swap usage in bytes. Reported only on non-windows systems
      @@ -1432,6 +1481,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Histogram
    • driver_namegrpc_status_codemethod_namemigrated
    +
    dra_grpc_operations_duration_seconds
    +
    Duration in seconds of the DRA gRPC operations
    +
      +
    • ALPHA
      • +
    • Histogram
      • +
    • driver_namegrpc_status_codemethod_name
    +
    +
    dra_operations_duration_seconds
    +
    Latency histogram in seconds for the duration of handling all ResourceClaims referenced by a pod when the pod starts or stops. Identified by the name of the operation (PrepareResources or UnprepareResources) and separated by the success of the operation. The number of failed operations is provided through the histogram's overall count.
    +
      +
    • ALPHA
      • +
    • Histogram
      • +
    • is_erroroperation_name
    +
    endpoint_slice_controller_changes
    Number of EndpointSlice changes
      @@ -1559,14 +1622,14 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    ephemeral_volume_controller_create_failures_total
    -
    Number of PersistenVolumeClaims creation requests
    +
    Number of PersistentVolumeClaim creation requests
    • ALPHA
    • Counter
    ephemeral_volume_controller_create_total
    -
    Number of PersistenVolumeClaims creation requests
    +
    Number of PersistentVolumeClaim creation requests
    • ALPHA
    • Counter
      • @@ -1803,6 +1866,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Gauge
    • static
    +
    kubelet_admission_rejections_total
    +
    Cumulative number pod admission rejections by the Kubelet.
    +
      +
    • ALPHA
      • +
    • Counter
      • +
    • reason
    +
    kubelet_certificate_manager_client_expiration_renew_errors
    Counter of certificate renewal errors.
      @@ -1845,6 +1915,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Gauge
    +
    kubelet_container_aligned_compute_resources_count
    +
    Cumulative number of aligned compute resources allocated to containers by alignment type.
    +
      +
    • ALPHA
      • +
    • Counter
      • +
    • boundaryscope
    +
    kubelet_container_log_filesystem_used_bytes
    Bytes used by the container's logs on the filesystem.
      @@ -1859,6 +1936,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Histogram
    +
    kubelet_cpu_manager_exclusive_cpu_allocation_count
    +
    The total number of CPUs exclusively allocated to containers running on this node
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    +
    kubelet_cpu_manager_pinning_errors_total
    The number of cpu core allocations which required pinning failed.
      @@ -1873,6 +1957,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Counter
    +
    kubelet_cpu_manager_shared_pool_size_millicores
    +
    The size of the shared CPU pool for non-guaranteed QoS pods, in millicores.
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    +
    kubelet_credential_provider_plugin_duration
    Duration of execution in seconds for credential provider plugin
      @@ -1944,7 +2035,7 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • eviction_signal
    kubelet_graceful_shutdown_end_time_seconds
    -
    Last graceful shutdown start time since unix epoch in seconds
    +
    Last graceful shutdown end time since unix epoch in seconds
    • ALPHA
    • Gauge
      • @@ -2797,6 +2888,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Histogram
    +
    resourceclaim_controller_allocated_resource_claims
    +
    Number of allocated ResourceClaims
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    +
    resourceclaim_controller_create_attempts_total
    Number of ResourceClaims creation requests
      @@ -2811,6 +2909,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Counter
    +
    resourceclaim_controller_resource_claims
    +
    Number of ResourceClaims
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    +
    rest_client_dns_resolution_duration_seconds
    DNS resolver latency in seconds. Broken down by host.
      @@ -2944,6 +3049,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Gauge
    • operation
    +
    scheduler_inflight_events
    +
    Number of events currently tracked in the scheduling queue.
    +
      +
    • ALPHA
      • +
    • Gauge
      • +
    • event
    +
    scheduler_permit_wait_duration_seconds
    Duration of waiting on permit.
      @@ -2965,6 +3077,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Histogram
    • extension_pointpluginstatus
    +
    scheduler_preemption_goroutines_duration_seconds
    +
    Duration in seconds for running goroutines for the preemption.
    +
      +
    • ALPHA
      • +
    • Histogram
      • +
    • result
    +
    +
    scheduler_preemption_goroutines_execution_total
    +
    Number of preemption goroutines executed.
    +
      +
    • ALPHA
      • +
    • Counter
      • +
    • result
    +
    scheduler_queueing_hint_execution_duration_seconds
    Duration for running a queueing hint function of a plugin.
      @@ -3014,6 +3140,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
    • Custom
    • 1.29.0
    +
    selinux_warning_controller_selinux_volume_conflict
    +
    Conflict between two Pods using the same volume
    +
      +
    • ALPHA
      • +
    • Custom
      • +
    • propertypod1_namespacepod1_namepod1_valuepod2_namespacepod2_namepod2_value
    +
    service_controller_loadbalancer_sync_total
    A metric counting the amount of times any load balancer has been configured, as an effect of service/node changes on the cluster