Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CPDEV_105612] k8s_1.31_adoption #689

Merged
merged 19 commits into from
Oct 17, 2024
Merged

[CPDEV_105612] k8s_1.31_adoption #689

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
0a3669c
new k8s v1.31.0
Sep 4, 2024
a782d5e
Convert list of dictionaries
Sep 11, 2024
c818664
revert
Sep 11, 2024
5d33ac5
Convert list of dictionaries_1
Sep 11, 2024
2ddeae8
Installation.md
Sep 11, 2024
2b4f65c
From PR 644
Sep 11, 2024
ca0a9e3
return to unspecified patch version in base images
Imadzuma Sep 10, 2024
5ca2a33
Plan to remove W/A to patch kubelet configmap
ilia1243 Apr 18, 2024
6fa3b8b
Revert "From PR 644"
Sep 11, 2024
b60df89
applied containerd restart fix from !694
koryaga Sep 18, 2024
ed48e63
cherry-pick
Sep 30, 2024
093ff25
support ControlPlaneKubeletLocalMode=True to fix ubgrade to 1.31
Imadzuma Oct 9, 2024
7e50878
linter fixes
Imadzuma Oct 9, 2024
63f210f
Merge pull request #696 from Netcracker/control-plane-kubelet-local-m…
koryaga Oct 11, 2024
d9cab2b
v1.31.1_&_dl.k8s.io_instead_of_storage.googleapis
Oct 15, 2024
92b3a5f
removed_v1.31.0
Oct 15, 2024
db384c3
removed v1.31.0
koryaga Oct 15, 2024
e572b9b
Merge remote-tracking branch 'origin' into CPDEV_105612_k8s_1.31_adop…
koryaga Oct 15, 2024
129f010
Doc_update
Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ COPY --from=go-build /opt/ipip_check.gz /opt/kubemarine/kubemarine/resources/scr
WORKDIR /opt/kubemarine/

RUN apt update && \
pip3 install --no-cache-dir setuptools wheel && \
pip3 install --no-cache-dir build && \
python3 -m build -n && \
# In any if branch delete source code, but preserve specific directories for different service aims
Expand Down
30 changes: 30 additions & 0 deletions documentation/Installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6142,3 +6142,33 @@ The tables below shows the correspondence of versions that are supported and is
| | kubernetesui/dashboard | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | Required only if Kubernetes Dashboard plugin is set to be installed. |
| | kubernetesui/metrics-scraper | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | Required only if Kubernetes Dashboard plugin is set to be installed. |
| | rancher/local-path-provisioner | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | Required only if local-path provisioner plugin is set to be installed. |

## Default Dependent Components Versions for Kubernetes Versions v1.31.0
| Type | Name | Versions | | | | | | | Note |
|----------|----------------------------------------------------------------|------------------|------------------------------|--------------|--------------|-------------------|-----------|-----------|------------------------------------------------------------------------------------------------------------|
| | | CentOS RHEL 7.5+ | CentOS RHEL Oracle Linux 8.4 | Ubuntu 20.04 | Ubuntu 22.04 | Oracle Linux 7.5+ | RHEL 8.6+ | RockyLinux 8.6+ | |
| binaries | kubeadm | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | SHA1: 3ac8ce4f0f17a21e5531213025b0090c42310d08 |
| | kubelet | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | SHA1: b5cb543e21faaa600a5aac54a2316a9fd5e5b8c6 |
| | kubectl | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | SHA1: 34214a4983223aa4e8d29063990a265b220bef5b |
| | calicoctl | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | SHA1: 80f164a8248b8fa501f35f12cbcac0b059e665e2 Required only if calico is installed. |
| | crictl | v1.30.0 | v1.30.0 | v1.30.0 | v1.30.0 | v1.30.0 | v1.30.0 | v1.30.0 | SHA1: c81e76d5d4bf64d6b513485490722d2fc0a9a83b |
| rpms | containerd.io | 1.6.* | 1.6.* | 1.6.* | 1.6.* | 1.6.* | 1.6.* | 1.6.* | |
| | haproxy/rh-haproxy | 1.8 | 1.8 | 2.* | 2.* | 1.8 | 1.8 | 1.8 | Required only if balancers are presented in the deployment scheme. |
| | keepalived | 1.3 | 2.1 | 2.* | 2.* | 1.3 | 2.1 | 2.1 | Required only if VRRP is presented in the deployment scheme. |
| images | registry.k8s.io/kube-apiserver | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | |
| | registry.k8s.io/kube-controller-manager | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | |
| | registry.k8s.io/kube-proxy | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | |
| | registry.k8s.io/kube-scheduler | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | v1.31.0 | |
| | registry.k8s.io/coredns | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | |
| | registry.k8s.io/pause | 3.10 | 3.10 | 3.10 | 3.10 | 3.10 | 3.10 | 3.10 | |
| | registry.k8s.io/etcd | 3.5.15-0 | 3.5.15-0 | 3.5.15-0 | 3.5.15-0 | 3.5.15-0 | 3.5.15-0 | 3.5.15-0 | |
| | calico/typha | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | Required only if Typha is enabled in Calico config. |
| | calico/cni | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | |
| | calico/node | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | |
| | calico/kube-controllers | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | |
| | calico/apiserver | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | v3.28.1 | Required only if API server is enabled in Calico config. |
| | registry.k8s.io/ingress-nginx/controller | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | v1.11.1 | |
| | registry.k8s.io/kube-webhook-certgen | v1.4.1 | v1.4.1 | v1.4.1 | v1.4.1 | v1.4.1 | v1.4.1 | v1.4.1 | |
| | kubernetesui/dashboard | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | v2.7.0 | Required only if Kubernetes Dashboard plugin is set to be installed. |
| | kubernetesui/metrics-scraper | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | v1.0.8 | Required only if Kubernetes Dashboard plugin is set to be installed. |
| | rancher/local-path-provisioner | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | v0.0.27 | Required only if local-path provisioner plugin is set to be installed. |
4 changes: 3 additions & 1 deletion kubemarine/cri/containerd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,10 +299,12 @@ def configure_containerd(group: NodeGroup) -> RunnersGroupResult:
backup=True, sudo=True, mkdir=True)

log.debug("Restarting Containerd on %s node..." % node.get_node_name())
# to restart and wait untill containerd is up&running
node.sudo(
f"chmod 600 {os_specific_associations['config_location']} && "
f"sudo systemctl restart {os_specific_associations['service_name']} && "
f"systemctl status {os_specific_associations['service_name']}", callback=collector)
f"systemctl status {os_specific_associations['service_name']} && "
f"timeout 10 sh -c 'until sudo ctr version 2>&1; do sleep 1; done' ", callback=collector)
return collector.result


Expand Down
4 changes: 4 additions & 0 deletions kubemarine/procedures/check_paas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1511,7 +1511,11 @@ def kubernetes_admission_status(cluster: KubernetesCluster) -> None:

kubeadm_config = components.KubeadmConfig(cluster)
cluster_config = kubeadm_config.load('kubeadm-config', first_control_plane)
# Check if extraArgs is a list of dictionaries or a dictionary
apiserver_actual_args = cluster_config["apiServer"]["extraArgs"]
if isinstance(apiserver_actual_args, list):
# Convert list of dictionaries to a single dictionary
apiserver_actual_args = {arg['name']: arg['value'] for arg in apiserver_actual_args}

actual_state = "disabled"
if "admission-control-config-file" in apiserver_actual_args and (
Expand Down
70 changes: 70 additions & 0 deletions kubemarine/procedures/upgrade.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
from collections import OrderedDict
from typing import List, Callable, Dict
import uuid
import yaml
from kubemarine import kubernetes, plugins, admission, jinja
from kubemarine.core import flow, log, resources as res
from kubemarine.core import utils
Expand Down Expand Up @@ -47,6 +48,75 @@ def prepull_images(cluster: KubernetesCluster) -> None:
upgrade_group.call(kubernetes.images_grouped_prepull)


def enable_control_plane_kubelet_local_mode(cluster: KubernetesCluster) -> None:
nikhil1697 marked this conversation as resolved.
Show resolved Hide resolved
"""
Enable ControlPlaneKubeletLocalMode feature gate to ensure the kubelet communicates
with the local API server on control-plane nodes, but only for Kubernetes v1.31.x and above.
"""
# Check if the target version is v1.31.x or above
target_version = kubernetes.get_procedure_upgrade_version(cluster)
if not target_version.startswith("v1.31"):
cluster.log.debug(f"Skipping ControlPlaneKubeletLocalMode feature gate. Target version is {target_version}.")
return

cluster.log.debug("Enabling ControlPlaneKubeletLocalMode feature gate in kubeadm config for v1.31.x or above.")

# Fetch the existing kubeadm-config ConfigMap
control_plane_group = cluster.nodes['control-plane']
config_map_result = control_plane_group.sudo("kubectl get configmap kubeadm-config -n kube-system -o yaml")

# Extract the stdout from the RunnersGroupResult object
config_map_yaml = config_map_result.get_simple_out()

# Log the output to ensure the ConfigMap is fetched correctly
cluster.log.debug(f"Fetched kubeadm-config ConfigMap: {config_map_yaml}")

# Convert the yaml string into a Python dictionary
config_map = yaml.safe_load(config_map_yaml)

# Extract the ClusterConfiguration field
cluster_configuration = config_map['data'].get('ClusterConfiguration', "")

# Parse the ClusterConfiguration to modify it
cluster_config_dict = yaml.safe_load(cluster_configuration)

# Ensure apiServer.extraArgs exists and is a list
if 'apiServer' not in cluster_config_dict:
cluster_config_dict['apiServer'] = {}

if 'extraArgs' not in cluster_config_dict['apiServer']:
cluster_config_dict['apiServer']['extraArgs'] = []

# Check if feature-gates already exists, and update or add it
feature_gates_exists = False
for arg in cluster_config_dict['apiServer']['extraArgs']:
if arg.get('name') == 'feature-gates':
arg['value'] = 'ControlPlaneKubeletLocalMode=true'
feature_gates_exists = True
break

if not feature_gates_exists:
# Add the feature-gates setting
cluster_config_dict['apiServer']['extraArgs'].append({
'name': 'feature-gates',
'value': 'ControlPlaneKubeletLocalMode=true'
})

# Convert the modified ClusterConfiguration back to a string
updated_cluster_config_yaml = yaml.dump(cluster_config_dict)

# Update the ConfigMap with the new ClusterConfiguration
config_map['data']['ClusterConfiguration'] = updated_cluster_config_yaml

# Convert the entire ConfigMap back to yaml for patching
updated_config_map_yaml = yaml.dump(config_map)

# Apply the updated ConfigMap
control_plane_group.sudo(f"kubectl apply -f - <<EOF\n{updated_config_map_yaml}\nEOF")

cluster.log.debug("ControlPlaneKubeletLocalMode feature gate enabled on control-plane nodes.")


def kubernetes_upgrade(cluster: KubernetesCluster) -> None:
initial_kubernetes_version = kubernetes.get_kubernetes_version(cluster.previous_inventory)

Expand Down
14 changes: 14 additions & 0 deletions kubemarine/resources/configurations/compatibility/internal/kubernetes_images.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ kube-apiserver:
version: v1.30.1
v1.30.3:
version: v1.30.3
v1.31.0:
version: v1.31.0
kube-controller-manager:
v1.26.3:
version: v1.26.3
Expand Down Expand Up @@ -82,6 +84,8 @@ kube-controller-manager:
version: v1.30.1
v1.30.3:
version: v1.30.3
v1.31.0:
version: v1.31.0
kube-scheduler:
v1.26.3:
version: v1.26.3
Expand Down Expand Up @@ -123,6 +127,8 @@ kube-scheduler:
version: v1.30.1
v1.30.3:
version: v1.30.3
v1.31.0:
version: v1.31.0
kube-proxy:
v1.26.3:
version: v1.26.3
Expand Down Expand Up @@ -164,6 +170,8 @@ kube-proxy:
version: v1.30.1
v1.30.3:
version: v1.30.3
v1.31.0:
version: v1.31.0
pause:
v1.26.3:
version: '3.9'
Expand Down Expand Up @@ -205,6 +213,8 @@ pause:
version: '3.9'
v1.30.3:
version: '3.9'
v1.31.0:
version: '3.10'
etcd:
v1.26.3:
version: 3.5.6-0
Expand Down Expand Up @@ -246,6 +256,8 @@ etcd:
version: 3.5.12-0
v1.30.3:
version: 3.5.12-0
v1.31.0:
version: 3.5.15-0
coredns/coredns:
v1.26.3:
version: v1.9.3
Expand Down Expand Up @@ -287,3 +299,5 @@ coredns/coredns:
version: v1.11.1
v1.30.3:
version: v1.11.1
v1.31.0:
version: v1.11.1
6 changes: 6 additions & 0 deletions kubemarine/resources/configurations/compatibility/internal/packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ containerd:
version_debian: 1.6.*
v1.30.3:
version_debian: 1.6.*
v1.31.0:
version_debian: 1.6.*
containerdio:
v1.26.3:
version_rhel: 1.6*
Expand Down Expand Up @@ -126,6 +128,10 @@ containerdio:
version_rhel: 1.6*
version_rhel8: 1.6*
version_rhel9: 1.6*
v1.31.0:
version_rhel: 1.6*
version_rhel8: 1.6*
version_rhel9: 1.6*
haproxy:
version_rhel: 1.8*
version_rhel8: 1.8*
Expand Down
11 changes: 11 additions & 0 deletions kubemarine/resources/configurations/compatibility/internal/plugins.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ calico:
version: v3.27.3
v1.30.3:
version: v3.28.1
v1.31.0:
version: v3.28.1
nginx-ingress-controller:
v1.26.3:
version: v1.4.0
Expand Down Expand Up @@ -106,6 +108,9 @@ nginx-ingress-controller:
v1.30.3:
version: v1.11.1
webhook-version: v1.4.1
v1.31.0:
version: v1.11.1
webhook-version: v1.4.1
kubernetes-dashboard:
v1.26.3:
version: v2.7.0
Expand Down Expand Up @@ -167,6 +172,9 @@ kubernetes-dashboard:
v1.30.3:
version: v2.7.0
metrics-scraper-version: v1.0.8
v1.31.0:
version: v2.7.0
metrics-scraper-version: v1.0.8
local-path-provisioner:
v1.26.3:
version: v0.0.23
Expand Down Expand Up @@ -228,3 +236,6 @@ local-path-provisioner:
v1.30.3:
version: v0.0.27
busybox-version: 1.34.1
v1.31.0:
version: v0.0.27
busybox-version: 1.34.1
12 changes: 12 additions & 0 deletions kubemarine/resources/configurations/compatibility/internal/thirdparties.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ kubeadm:
sha1: 85385f4b78b79bd1cd4d1ad5244422d372d5f758
v1.30.3:
sha1: f840e75f5dc1001ebdd7e286c0e87e1090df011b
v1.31.0:
sha1: 3ac8ce4f0f17a21e5531213025b0090c42310d08
kubelet:
v1.26.3:
sha1: 5fe320fedaabb91d3770da19135412b7454bb28b
Expand Down Expand Up @@ -83,6 +85,8 @@ kubelet:
sha1: c62da6ab918b8e56d7c9b77e642ffc73ffdbffac
v1.30.3:
sha1: fbae53efc43ec715a45b05415294ab991ea087a2
v1.31.0:
sha1: b5cb543e21faaa600a5aac54a2316a9fd5e5b8c6
kubectl:
v1.26.3:
sha1: 56916d87c3caef05489db932fd9e48d32ebdf634
Expand Down Expand Up @@ -124,6 +128,8 @@ kubectl:
sha1: bced94239f1dbdb04d3a661a067bf9587865b6e8
v1.30.3:
sha1: 097d6b02fabb284418a9c95ea81fa86fc3c85bb7
v1.31.0:
sha1: 34214a4983223aa4e8d29063990a265b220bef5b
calicoctl:
# calicoctl version is duplicated from kubemarine/resources/configurations/compatibility/kubernetes_versions.yaml
# It also corresponds to the plugin version in kubemarine/resources/configurations/compatibility/internal/plugins.yaml
Expand Down Expand Up @@ -187,6 +193,9 @@ calicoctl:
v1.30.3:
version: v3.28.1
sha1: 80f164a8248b8fa501f35f12cbcac0b059e665e2
v1.31.0:
version: v3.28.1
sha1: 80f164a8248b8fa501f35f12cbcac0b059e665e2
crictl:
# crictl version is duplicated from kubemarine/resources/configurations/compatibility/kubernetes_versions.yaml
# for backward compatibility with clusters in a private environment.
Expand Down Expand Up @@ -250,3 +259,6 @@ crictl:
v1.30.3:
version: v1.30.0
sha1: c81e76d5d4bf64d6b513485490722d2fc0a9a83b
v1.31.0:
version: v1.30.0
sha1: c81e76d5d4bf64d6b513485490722d2fc0a9a83b
9 changes: 8 additions & 1 deletion kubemarine/resources/configurations/compatibility/kubernetes_versions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ kubernetes_versions:
supported: true
v1.30:
supported: true
v1.31:
supported: true
compatibility_map:
# This section should be changed manually.
v1.26.3:
Expand Down Expand Up @@ -132,7 +134,12 @@ compatibility_map:
kubernetes-dashboard: v2.7.0
local-path-provisioner: v0.0.27
crictl: v1.30.0

v1.31.0:
calico: v3.28.1
nginx-ingress-controller: v1.11.1
kubernetes-dashboard: v2.7.0
local-path-provisioner: v0.0.27
crictl: v1.30.0
# After any change, please run scripts/thirdparties/sync.py

# The following optional keys are supported in addition to the 5 required software keys:
Expand Down
Loading