Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use --local=false on amd64, to workaround containerd image loading #3805

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Use --local=false on amd64, to workaround containerd image loading #3805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions pkg/build/nodeimage/imageimporter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package nodeimage

import (
"io"
"runtime"

"sigs.k8s.io/kind/pkg/exec"
)
Expand Down Expand Up @@ -53,9 +54,16 @@ func (c *containerdImporter) Pull(image, platform string) error {
}

func (c *containerdImporter) LoadCommand() exec.Cmd {
return c.containerCmder.Command(
var opt string
// TODO: Hack to workaround #3795.
if runtime.GOARCH == "amd64" {
opt = "--local=false"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate a bit on what this is intended to do and why only amd64?

it will be a bit before I can dig into this

Copy link
Contributor Author

@dgl dgl Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dug a bit more into the actual issue on the issue thread, which will hopefully answer how to actually fix this in containerd / docker (which I do believe it is actually a bug in one of).

The reason for the arm64 / amd64 split is confusing and what makes this a hack:

  • This is happening with packages where there is an amd64 / 386 mix, hence why targeting amd64 for --local=false.
  • On the version of containerd kind is currently using --all-platforms with --local=false doesn't work.

So the bug has been seen on amd64, people using arm64 do want to mix platforms. It's also possible for arm64 to mix v8 or v9, but I think that's far less common and unlikely to break anyone.

Clearly this should be fixed in containerd (or docker, if it should actually include more blobs in its images), but given a kind release is tied to a containerd release I think it would be safe to include this hack until kind is on a containerd with a fix.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like much the idea of we carry patching compensation for known bugs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should do this only on a specific arch, people could mix platforms on other host arches?

but given a kind release is tied to a containerd release I think it would be safe to include this hack until kind is on a containerd with a fix.

... that's not strictly true, users ignore our docs and use node images across kind releases all the time, so we need to at least warn that some minimum version will be required because we require this new flag that isn't present in images from prior releases.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general rule we actually permit using them across releases and notify when a release has a compatibility change that requires images built after a certain release or something like that. It hasn't happened recently as well.

} else {
// TODO: ideally we do not need this in the future. we have fixed at least one image
"ctr", "--namespace=k8s.io", "images", "import", "--label=io.cri-containerd.pinned=pinned", "--all-platforms", "--no-unpack", "--digests", "-",
opt = "--all-platforms"
}
return c.containerCmder.Command(
"ctr", "--namespace=k8s.io", "images", "import", "--label=io.cri-containerd.pinned=pinned", opt, "--no-unpack", "--digests", "-",
)
}

Expand Down
11 changes: 10 additions & 1 deletion pkg/cluster/nodeutils/util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"encoding/json"
"io"
"path"
"runtime"
"strings"

"github.com/pelletier/go-toml"
Expand Down Expand Up @@ -82,7 +83,15 @@ func LoadImageArchive(n nodes.Node, image io.Reader) error {
if err != nil {
return err
}
cmd := n.Command("ctr", "--namespace=k8s.io", "images", "import", "--all-platforms", "--digests", "--snapshotter="+snapshotter, "-").SetStdin(image)
var opt string
if runtime.GOARCH == "amd64" {
// TODO: Hack to workaround #3795.
opt = "--local=false"
} else {
// Needed on e.g. ARM machines to also pull amd64 images (#2957).
opt = "--all-platforms"
}
cmd := n.Command("ctr", "--namespace=k8s.io", "images", "import", opt, "--digests", "--snapshotter="+snapshotter, "-").SetStdin(image)
if err := cmd.Run(); err != nil {
return errors.Wrap(err, "failed to load image")
}
Expand Down