[Bug]: mtree_spec line order affects tar directory structure

[bmyers-ozette]

What happened?

I am trying to create a tar using the mtree_spec and tar helpers, but having an issue where the resulting tar has files in the incorrect locations.

For example, in a directory structure like this

app
├── nginx
│   ├── conf
│   │   ├── BUILD
│   │   ├── conf.d
│   │   │   └── http.conf
│   │   └── nginx-template-based.conf

With BUILD file:

filegroup(
    name = "foo_files",
    srcs = glob(
        ["**"],
    ),
)

mtree_spec(
    name = "foo_mtree",
    srcs = [":foo_files"],
)

genrule(
    name = "strip_prefix",
    srcs = ["foo_mtree"],
    outs = ["foo_mtree.stripped"],
    cmd = "sed '{}' <$< | sed '/^\\ /d' > $@".format(
        "; ".join(reversed([
            "s#^{s}/##; s#^{s}##".format(s = "/".join(package_name().split("/")[:i]))
            for (i, _) in enumerate(
                package_name().split("/"),
                1,
            )
        ])),
    ),
)

tar(
    name = "foo",
    srcs = [":foo_files"],
    mtree = "foo_mtree.stripped",
)

The resulting mtree_spec file looks like :

conf.d uid=0 gid=0 time=1672560000 mode=0755 type=dir
conf.d/http.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/conf.d/http.conf
nginx-template-based.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/nginx-template-based.conf

But foo.tar has the nginx-template-based.conf file in the conf.d folder instead of at the top level. If I manually supply a mspec sorted by depth instead of alphabetically, it works as expected.

tar(
    name = "bar",
    srcs = [":foo_files"],
    mtree = [
        "nginx-template-based.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/nginx-template-based.conf",
        "conf.d uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "conf.d/http.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/conf.d/http.conf",
    ],
)

Version

Development (host) and target OS/architectures:

Output of bazel --version:
bazel 6.5.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
bazel-lib-v2.4.1

Language(s) and/or frameworks involved:

How to reproduce

No response

Any other information?

No response

[alexeagle]

Thanks, can you isolate this a bit more? Is that genrule that does the manipulation at fault, or can you reproduce a bug without it?

I think it's probably intentional that the order of mtree lines affects the tar. mtree has support for state, like you can effectively "cd somedir" and then subsequent lines are affected.

[bmyers-ozette]

Thanks for the reply and for taking a look! The issue seems to only reproduce for files the are in the root directory of the tar according to the mtree spec.

Here's a repro case without the genrule to illustrate. Resulting tar has nginx-template-based.conf under conf.d instead of at the root:

tar(
    name = "foo",
    srcs = [":foo_files"],
    mtree = [
        "conf.d uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "conf.d/http.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/conf.d/http.conf",
        "nginx-template-based.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/nginx-template-based.conf",
    ],
)

$ tar -tvf foo.tar
drwxr-xr-x 0/0               0 2023-01-01 02:00 conf.d/
-rwxr-xr-x 0/0            1426 2023-01-01 02:00 conf.d/http.conf
-rwxr-xr-x 0/0             636 2023-01-01 02:00 conf.d/nginx-template-based.conf

Reordered mtree has nginx-template-based.conf in the root as expected:

tar(
    name = "bar",
    srcs = [":foo_files"],
    mtree = [
        "nginx-template-based.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/nginx-template-based.conf",
        "conf.d uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "conf.d/http.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/conf.d/http.conf",
    ],
)

$ tar -tvf bar.tar
-rwxr-xr-x 0/0             636 2023-01-01 02:00 nginx-template-based.conf
drwxr-xr-x 0/0               0 2023-01-01 02:00 conf.d/
-rwxr-xr-x 0/0            1426 2023-01-01 02:00 conf.d/http.conf

If the files are nested under a prefix, it also seems to work as I would expect:

tar(
    name = "foobar",
    srcs = [":foo_files"],
    mtree = [
        "app uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "app/nginx uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "app/nginx/conf uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "app/nginx/conf/conf.d uid=0 gid=0 time=1672560000 mode=0755 type=dir",
        "app/nginx/conf/conf.d/http.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/conf.d/http.conf",
        "app/nginx/conf/nginx-template-based.conf uid=0 gid=0 time=1672560000 mode=0755 type=file content=app/nginx/conf/nginx-template-based.conf",
    ],
)

$ tar -tvf foobar.tar
drwxr-xr-x 0/0               0 2023-01-01 02:00 app/
drwxr-xr-x 0/0               0 2023-01-01 02:00 app/nginx/
drwxr-xr-x 0/0               0 2023-01-01 02:00 app/nginx/conf/
drwxr-xr-x 0/0               0 2023-01-01 02:00 app/nginx/conf/conf.d/
-rwxr-xr-x 0/0            1426 2023-01-01 02:00 app/nginx/conf/conf.d/http.conf
-rwxr-xr-x 0/0             636 2023-01-01 02:00 app/nginx/conf/nginx-template-based.conf

Hope that helps clarify the issue. If it's intentional that the order of mtree lines affects the tar, seems like creating an mtree_spec from a globbed filegroup will give different behavior just depending on the alphabetical ordering of the directories and files contained.

[thesayyn]

this is how mtrees work, you either have to add ./ to the dir entry, or have / as the suffix for the dir entry.
This is working as expected.

[bmyers-ozette]

Thanks @thesayyn,

Seems like this is a nuance to the mtrees that is easy to trip over if you naively try to strip off a prefix from the mtree_spec output. I believe the genrule in the tests that is referenced from the tar documentation would be susceptible to this issue: https://github.com/aspect-build/bazel-lib/blob/main/lib/tests/tar/BUILD.bazel#L128-L145

Anyway, adding the / suffix did the trick so I think this can be closed. Thanks again for the reply and explanation.