[image-rs] failed to get stream processor for application/vnd.oci.image.layer.v1.tar+gzip+encrypted: exec: \"ctd-decoder\": executable file not found in $PATH: unknown

[fu-ju]

Hello everyone, I'm trying to launch a Kata container using an encrypted container image, but I'm encountering errors during the image pull and decryption process. I'll provide more details below and would appreciate any help in identifying the issue. Thank you.

What I did

1. I encrypted the container image using the following Skopeo command and pushed it to the local registry
   skopeo --debug copy --src-tls-verify=false --dest-tls-verify=false --encryption-key provider:attestation-agent:keypath=${ENC_KEY_FILE}::keyid=kbs:///default/key/key_id1 docker://localhost:5000/nginx:latest docker://localhost:5000/nginx_enc:latest
2. I compiled and ran the Attestation Agent(Automatically started by the Kata agent.) in offline_fs_kbc mode
   This means there is no need to rely on remote attestation to obtain the decryption key.
3. K8s starts the Kata container with the following configuration

apiVersion: v1
kind: Pod
metadata:
  name: nginx-enc
  annotations:
    io.katacontainers.config_path: "localhost:5000/nginx_enc"
spec:
  runtimeClassName: kata
  containers:
  - name: nginx-test
    image: localhost:5000/nginx_enc
    command:
      - top
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
     - name: docker-reg-secret
  restartPolicy: Never

What I expected

When starting the Kata container, the Kata agent pulls and decrypts the image using image-rs, and the container starts successfully.

What I encountered

What seems to be happening is that containerd is pulling the encrypted image and attempting to decrypt it using ctd-decoder, rather than the expected image-rs.

The error message I received is as follows:
level=error msg="PullImage \"localhost:5000/nginx_enc:latest\" failed" error="failed to pull and unpack image \"localhost:5000/nginx_enc:latest\": failed to extract layer sha256:ceb5f16d1891b2ef1eb0f93e47ede3b1dbfa0aaa8e8a6b5852efce6ade6f4d09: failed to get stream processor for application/vnd.oci.image.layer.v1.tar+gzip+encrypted: exec: \"ctd-decoder\": executable file not found in $PATH: unknown"

Is it related to the version of containerd? I'm using the one installed from the docker-ce.repo, but the link （https://github.com/kata-containers/kata-containers/blob/main/versions.yaml）seems to require a modified version of containerd from the cc-main branch.

[fitzthum]

This error tends to happen when you pull the image on the host inadvertently. There are a few different things that could cause this. First, try changing your runtime class to kata-qemu-coco-dev. In the most recent releases the kata runtime classes points to standard kata, which pulls the image on the host.