Pass custom kubectl binary

[steeve]

Description of the problem / feature request:

Pass custom kubectl binary to k8s_deploy commands.

Feature requests: what underlying problem are you trying to solve with this feature?

We use k3s in Docker for development, and up until now we wrapped kubectl with docker run k3s kubectl, since the k3s image ships in the container.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 3.4.1

I'm thinking of adding a kubectl option to k8s_deploy, but I saw a bunch of stuff related to custom kubectl binaries so I'm not sure.

[kzadorozhny]

In rules_gitops we assume that host kubectl is available and has appropriate configuration to access the cluster. It is the same assumption made by Bazel rules_k8s.

Is docker run k3s kubectl a literally the command you want to run instead of host kubectl or it is just a way of saying "run some other command"?

Does the Bazel downloaded kubectl binary is an acceptable solution?

[steeve]

In rules_k8s, it is possible to override the kubectl binary https://github.com/bazelbuild/rules_k8s/tree/master/toolchains/kubectl#download-a-custom-kubectl-binary:

load("@io_bazel_rules_k8s//toolchains/kubectl:kubectl_configure.bzl", "kubectl_configure")

kubectl_configure(
    name = "k8s_config",
    kubectl_path = "@myrepo//my:kubectl",
)

This has the advantage for easy swapping of the kubectl binary, either by downloading or wrapping (as we do).

Actually, I was wrong, it's not run but exec: we run kubectl in the same container as the k3s node.

Here is our wrapper:

#!/bin/bash
set -eu
use_tty=""
if tty -s; then
    use_tty="-t"
fi

docker exec -i ${use_tty} k3s kubectl "${@}"

The added advantage over a downloaded kubectl binary is that kubectl is always compatible with the k3s node.

That said, it is of course incompatible with deploying on a remote node.

Finally, we used to extract of the kubeconfig.yaml from the container, but the only way was to inject the container id as stamp, so that the genrule would get re-run when the container restarts.