Create a docker context with a buildkit remote daemon in kubernetes

[Momotoculteur]

Hello all !

I have a kubernetes which run a Docker buildkit rootless daemon, without TLS (See that docker buildx, but currently i made my own chart here)

I want to have that scenario in my Jenkins pipeline :

1. Build a docker image via Buildx cli (contact the daemon and send build instructions, which actually works )
2. Scan that image via Trivy (which not work)
3. Push my image if i have no CVE to my registry (which works)

To have Trivy working with image, i need to be able to load my previous builded image or to be able to make a docker pull command

I tried to load the image when building :

docker buildx create --name testBuilder --driver=remote tcp://buildkitd.docker-buildkit.svc.cluster.local:1234 --use --bootstrap
docker buildx build -f Dockerfile -o type=docker,dest=- --load -t image:1.0.0

But this seem to not work.

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I'm not a pro with Buildkit & buildx and other. In my opinion i need to create a docker context connected to my Buildkit daemon ? cause when i check image i can see layers & images in my daemon.

I tried to create via :

docker context create myctx --docker host=tcp://buildkitd.docker-buildkit.svc.cluster.local:1234

But that gave me this following error :

Head "http://buildkitd.docker-buildkit:1234/_ping": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00".
* Are you trying to connect to a TLS-enabled daemon without TLS?

I also tried to force to skip the TLS verify via :

docker context create myctx --docker host=tcp://buildkitd.docker-buildkit.svc.cluster.local:1234,skip-tls-verify=true

But that gave me another weird error :

$ docker image ls
error during connect: Head "https://buildkitd.docker-buildkit:1234/_ping": tls: first record does not look like a TLS handshake

A possible solution would be to build my image in local in a tar.gz format, scan it via trivy, and to re-build & push to my registry. Or to copy it via a tool like Skopeo, but i don't find this solution to be optimal.

[AkihiroSuda]

This one might be easier to set up, as it uses kubectl exec stdio streams instead of TCP

buildkit/examples/kubernetes/README.md

Lines 16 to 21 in 7c025bf

	```console
	$ kubectl apply -f pod.rootless.yaml
	$ buildctl \
	--addr kube-pod://buildkitd \
	build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
	```

The buildctl build CLI is slightly different from the docker buildx CLI, though.

[AkihiroSuda]

DOCKER_HOST must refer to a Docker daemon, not to a BuildKit daemon. These daemons speak different APIs and not interchangeable.

[AkihiroSuda]

In your usecase , probably you should just emit an OCI archive (--output type=oci,dest=FILE.tar) without using --load and let Trivy consume it via trivy image --input FILE.tar.

[Momotoculteur]

Seem to have exactly that issue
docker/cli#5193

I could make your scenario and use local instead of loading in docker context. But as i said in my first message, i need to build only, scan it, and push it. With your method, i need an extra tool (crane, skopeo,....) to be able to push as i already build in local my image. That not sounds to be the optimal choice. Furthermore, people from Trivy's team are able to do the way i want, but only in the same kubernetes namespace, which i can't.

[AkihiroSuda]

--output type=image,name=docker.io/username/image,push=true may work for you?
If docker buildx does not work as expected you can just use buildctl CLI.

[Momotoculteur]

The probleme is not the building part, that work nice with buildx. The problem is around the docker context, which can't docker pull any image, with a a buildkit deploy+service in k8s, receive traffic from pod located in another namespace.

I'm participating here also