max depth exceeded when trying to load images that have many apt dependencies

[juanzolotoochin]

To repro:

• Add a few larger apt dependencies to main/examples/apt/bullseye.yaml. I can repro this with just adding python3-minimal
• Remove the if statement here:

  rules_distroless/apt/private/package_resolution.bzl

  Line 105 in 789bf1d

  if in_lock(package["Package"], package["Version"]):

  . This is important because it fixes Transitive dependencies of apt packages are affected by the order in which they appear on the manifest yaml.file #29 which is causing a lot of tars to not be added to the image.
• Run bazel run @bullseye//:lock
• Add the dependencies to the image target here https://github.com/GoogleContainerTools/rules_distroless/blob/main/examples/apt/BUILD.bazel#L49
• Run the tests: bazel test //examples/apt/... --test_output=all

Expected output: test passing

Actual output: a bunch of tests failing with Error: error setting env vars: Error creating container: no such image

From my conversations with @thesayyn , the underlying error is {"errorDetail":{"message":"max depth exceeded"},"error":"max depth exceeded"}

Googling for that error, seems to be caused by too many docker layers. One workaround is to combine those layers into a single tar:

pkg_tar(
    name = "linux_layers.tar",
    deps = select({
        "@platforms//cpu:arm64": [
            "%s/arm64" % package
            for package in PACKAGES
        ],
        "@platforms//cpu:x86_64": [
            "%s/amd64" % package
            for package in PACKAGES
        ],
    }),
)

Then using that tar on the image instead of each debian package tar.

Presumably, the same should be achievable using rules_distroless flatten rule. However this:

flatten(
    name = "linux_layers_flatten.tar",
    tars = select({
        "@platforms//cpu:arm64": [
            "%s/arm64" % package
            for package in PACKAGES
        ],
        "@platforms//cpu:x86_64": [
            "%s/amd64" % package
            for package in PACKAGES
        ],
    }),
)

does not resolve it. There seems to be an issue with duplicate file paths in flatten:

$ bazel build //examples/apt:tarball

$ docker load < bazel-bin/examples/apt/tarball/tarball.tar
duplicates of file paths not supported

[thesayyn]

Thank you for filing this!

[lazcamus]

WRT flatten() and "duplicates of file paths not supported" from docker:

• it appears to be a problem only with older versions of docker? I've seen it with docker server version 24.0.4 (gcp COS) but not with server version 26.1.4 (docker desktop on osx)
• I'm not very seasoned with bazel rules, but I have a workaround rule to launder your tar file. Unfortunately, you have to explode the tar temporarily to dedupe:

load("@aspect_bazel_lib//lib:tar.bzl", "mtree_spec", "tar", "tar_lib")

def _dedupe_tar_impl(ctx):
    """
    Remove duplicate entries in a tarball

    This is kinda a hack: extract the mtree file, explode the tarball (handled
    overwriting the files), then recreate the tarball from the exploded tree.

    This has the effect of removing duplicate entries from the tarball
    """
    bsdtar = ctx.toolchains[tar_lib.toolchain_type]
    bsdtar_bin = bsdtar.template_variables.variables["BSDTAR_BIN"]
    src = ctx.file.src

    output_tar = ctx.actions.declare_file("%s.tar.gz" % ctx.label.name)

    ctx.actions.run_shell(
        outputs = [output_tar],
        inputs = [src],
        command = """
            set -e
            export TMP=$(mktemp -d || mktemp -d -t bazel-tmp)
            trap "rm -rf $TMP" EXIT
            mkdir $TMP/extracted

            {bsdtar} --format=mtree -cf - @{src} | egrep -v '^/. ' | sed -e 's|^./||' > {mtree}
            {bsdtar} -xf {src} -C $TMP/extracted
            {bsdtar} -C $TMP/extracted -caf {output} @{mtree}
        """.format(
            bsdtar = bsdtar_bin,
            src = src.path,
            output = output_tar.path,
            mtree = "$TMP/mtree.txt",
        ),
        tools = [bsdtar.default.files],
    )

    return DefaultInfo(files = depset([output_tar]))

dedupe_tar = rule(
    implementation = _dedupe_tar_impl,
    attrs = {
        "src": attr.label(mandatory = True, allow_single_file = True),
    },
    # XXX: side effect: gzipping the output 
    outputs = {"output_tar": "%{name}.tar.gz"},
    toolchains = [tar_lib.toolchain_type],
)

From the above example, I think this would probably work:

flatten(
    name = "linux_layers_flatten.tar",
    tars = select({
        "@platforms//cpu:arm64": [
            "%s/arm64" % package
            for package in PACKAGES
        ],
        "@platforms//cpu:x86_64": [
            "%s/amd64" % package
            for package in PACKAGES
        ],
    }),
)

dedupe_tar(
    name = "linux_layers_deduped.tar",
    src = "linux_layers_flatten.tar",
)

[bd-jczarnowski]

@lazcamus Thank you for the workaround! I'm hitting exactly the same issue while trying to introduce rules_distroless + rules_oci into our builds.

pkg_tar does deduplication but prints a warning for each occurence (and there's a lot..)

Aside from not being able to load so many layers into docker, is creating separate layers for all the debs by default efficient? (re: the "default" behavior from the example")