Looking into Changing kubectl provider for k8s 1.27

[timckt]

Background

On the k8s 1.27 live like test cluster, when we would like to run terraform apply to bump starter_pack_count up, it comes with the following kubectl_manifest error.

It is probably related to gavinbunney/terraform-provider-kubectl is abandoned and we need to consider switching to the alekc/terraform-provider-kubectl.

Reference:

gavinbunney/terraform-provider-kubectl#270
aws-samples/karpenter-blueprints#5
https://github.com/alekc/terraform-provider-kubectl

module.tigera_calico.kubectl_manifest.calico_global_policies: Creating...
╷
│ Error: gatekeeper-system/config failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.kubectl_manifest.config_sync,
│   on .terraform/modules/gatekeeper/main.tf line 91, in resource "kubectl_manifest" "config_sync":
│   91: resource "kubectl_manifest" "config_sync" {
│ 
╵
╷
│ Error: deny-aws-imds failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.tigera_calico.kubectl_manifest.calico_global_policies,
│   on .terraform/modules/tigera_calico/calico.tf line 6, in resource "kubectl_manifest" "calico_global_policies":
│    6: resource "kubectl_manifest" "calico_global_policies" {
│ 
╵
╷
│ Error: lockprivcapabilities failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["lock_priv_capabilities.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8sservicetypeloadbalancer failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["service_type.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8smodsecnginxclass failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["modsec_nginx_class.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8shostnamelength failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["ingress_hostname_length.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8singressclash failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["ingress_clash.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8smodsecsnippetnginxclass failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["modsec_snippet_nginx_class.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8svalidhostname failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["ingress_valid_hostname.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: warnkubectlserviceaccount failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["warn_sa.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8ssnippetallowlist failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["snippet_allowlist.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8srequiredlabels failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["required_labels.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8sexternaldnsweight failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["ingress_external_dns_weight.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8swarnserviceaccountsecretdelete failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["warn_service_account_secret_delete.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: k8sexternaldnsidentifier failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["ingress_external_dns_identifier.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: verifydeprecatedapi failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.constraint_templates.kubectl_manifest.constraint_templates["verify_deprecated_api.yaml"],
│   on .terraform/modules/gatekeeper/constraint_templates/templates.tf line 2, in resource "kubectl_manifest" "constraint_templates":
│    2: resource "kubectl_manifest" "constraint_templates" {
│ 
╵
╷
│ Error: default-supplemental-groups failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["default_supplemental_groups.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: default-fs-group failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["default_fs_group.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: deny-privilege-escalation-init failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["deny_privilege_escalation_init.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: annotate-seccomp-pod-runtime failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["annotate_seccomp.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: deny-privilege-escalation failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["deny_privilege_escalation.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: run-as-non-root-eph failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["run_as_non_root_eph.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: deny-privilege-escalation-eph failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["deny_privilege_escalation_eph.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: drop-all-cap failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["drop_all_cap.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: run-as-non-root failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["run_as_non_root.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: default-seccomp-profile failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["default_seccomp_profile.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: drop-all-cap-eph failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["drop_all_cap_eph.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: run-as-non-root-init failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["run_as_non_root_init.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {
│ 
╵
╷
│ Error: drop-all-cap-init failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.gatekeeper.module.mutations.kubectl_manifest.mutations["drop_all_cap_init.yaml"],
│   on .terraform/modules/gatekeeper/mutations/mutations.tf line 2, in resource "kubectl_manifest" "mutations":
│    2: resource "kubectl_manifest" "mutations" {

Proposed user journey

Approach

Which part of the user docs does this impact

Communicate changes

• post for #cloud-platform-update
• Weeknotes item
• Show the Thing/P&A All Hands/User CoP
• Announcements channel

Questions / Assumptions

Definition of done

• readme has been updated
• user docs have been updated
• another team member has reviewed
• smoke tests are green
• prepare demo for the team

Reference

How to write good user stories

[sj-williams]

Have confirmed that this issue can be traced to the bug which manifests for gavinbunney/kubectl at kubernetes v1.27.

Issue arises from the requestURI for the obects API endpoints is broken:

https://mojdt.slack.com/archives/C514ETYJX/p1710159268589039

Testing switchover for all components modules to forked provider maintainer:
https://registry.terraform.io/providers/alekc/kubectl/latest/docs

https://github.com/alekc/terraform-provider-kubectl

Fix in this version of the provider:
alekc/terraform-provider-kubectl@a0cf1de