permission issue when mounting directory inside pool in Docker container

[snyssen]

Hi,
I am using a mergerfs pool for storing most of the files on my server. It is composed of 4 HDD formatted with ext4 and mounted on /mnt/disk{1,..,4}, then pooled with the following fstab entry:

/mnt/disk* /mnt/storage fuse.mergerfs defaults,nonempty,allow_other,use_ino,cache.files=off,moveonenospc=true,category.create=mfs,minfreespace=20G,fsname=mergerfs-storage 0 0

On this pool, I have a directory /mnt/storage/nextcloud mounted as the /var/www/html volume inside a nextcloud container. It is owned by 33:1000, 33 being the user id for www-data, the user running the Nextcloud server inside the container. It used to work great, but I have recently had to re-install the OS (Fedora) after the root drive failed, and trying to start the container using the same configuration now fails. More specifically, the www-data user inside the container is unable to access the /var/www/html directory (running ls on it returns No such file or directory) while the root user can. There is no issue listing the files from the host and all permissions seem correct (I did a sudo chown -R 33:33 /mnt/storage/nextcloud && sudo chmod -R 770 /mnt/storage/nextcloud as well as later 777 permissions juste to be sure).

I also did confirm it was not a simple permission issue by creating a fresh install of Nextcloud but mounting /mnt/disk1/nextcloud-debug instead, which worked just fine, and then doing the same but using /mnt/storage/nextcloud-debug, which did not.

I am now stuck, unsure of how I could debug this issue further, and thus asking for help. It should also be noted that other containers seem to work without issues using similar configuration, such as Photoprism which mounts /mnt/storage/pictures.

[trapexit]

Can you replicate the issue with minimal example using a random OS container? What exactly is your config for the container? Are you using uid namespaces? What the volume mount options? A reproducable example is the best for me to diagnose.

[trapexit]

Is it?

sudo -u www-data ls -lh /mnt/disk2/nextcloud-debug
ls: cannot access '/mnt/disk2/nextcloud-debug': Permission denied

Unless I'm missing something isn't this the uid you expect to be working getting a perm denied outside the container? As I mentioned error handling inside mergerfs for this could be improved and I believe that due to subtle differences in how mergerfs reads dirs it isn't returning the proper perm denied error but the fact you get a perm denied as www-data on the host matches with what mergerfs is getting.

[pid 772] 22:43:25.410912 openat(AT_FDCWD, "/mnt/disk2/nextcloud-debug", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 EACCES (Permission denied) <0.000008>

www-data user only needs to read and write inside the nextcloud directory, so I don't see why it would need access to /mnt/disk2, especially since the upper directory should be invisible to the container.

A filesystem is a tree. A process needs appropriate perms to the entire hierarchy of nodes. To / then /mnt/ then /mnt/disk2/ then /mnt/disk2/nextcloud-debug/. Need at least execute down the chain to be able to access the files. There is more subtly around perms due to a number of factors but that's the basics: https://www.redhat.com/sysadmin/linux-file-permissions-explained

[trapexit]

And remember... mergerfs is outside the container reading the paths you configured. Your binding of that path into the container (which effectively changes the permission hierarchy) doesn't affect mergerfs. mergerfs is still just reading /mnt/disk2/nextcloud-debug/ (not /var/tmp/html) as the user in question. I don't know all the perms as I need to reproduce (full hierarchy) but my guess now is that you've disallowed the user access to the full hierarchy and so mergerfs is properly denied access.

Is this a arguably incorrect behavior? Yes. I believe I could work around this by always opening the base of the branch as root and then using that instead as the string path root. That was always something I considered but isn't without its consequences. If I kept the file open all the time then users wouldn't be able to umount branches without first ensuring mergerfs had it removed from the pool. A "feature" that at least I have used many times. It might even lead to drives not being able to spin down due to being in use. I'm not positive about that though. The alternative would be to always open branch bases as root first and then use it temporarily but then that adds more syscalls which isn't ideal. And making such a thing optional just isn't practical. It is too invasive of a change. It fundamentally changes the way everything works.

For those curious it would be moving all regular functions to *at functions. IE... open -> openat.

The easiest workaround for you is just to give execute access at least to "other" for /mnt and /mnt/disk*

[snyssen]

Oh wow, thank you for explaining! I didn't know or forgot that the execute permission was also needed for directory listing, and you can see exactly where I shot myself in the foot by setting the permission on all disks and the mergerfs mount itself to "740"... I'll update that and I'm pretty sure it should work then. Such a dumb mistake... 🤦Will update you as soon as I have tried the fix.
As for the way mergerfs handled the error, I'm not sure I understand all the implication but the way you did seems like a very reasonable trade-off. Opening the base branch as root seems like it would also hinder performances?

[snyssen]

Okay I can confirm it was the issue: a directory higher in the tree (/mnt/storage) was owned by 1000:1000 with permissions 744 and was thus blocking access to /mnt/storage/nextcloud for user with id 33. Giving permissions 755 to `/mnt/storage' solved the issue.

This issue had nothing to do with mergerfs then, sorry for the disturbance and thank you a lot for helping me debug this issue!

[trapexit]

I wouldn't say nothing but... as I mentioned above a somewhat complex and slightly niche situation.

thank you a lot for helping me debug this issue!

No problem. I'll look at making the error handling in readdir a bit better so at least it returns something more appropriate when all branches fail.

[trapexit]

Opening the base branch as root seems like it would also hinder performances?

Actually it would likely be slightly faster depending on use though I'd need to run some micro benchmarks again as I did years ago.

The difference is this:

fd = open("/full/path/to/branch" + "/relative/path/requested")
do_something_with_fd(fd)
close(fd)

or

do_something_with_path("/full/path/to/branch" + "/relative/path/requested")

Which in both cases the full path has to be walked and perm checks and everything done. That's how most things work today.

or

fd = open("/branch/base/path") // This happens once at startup or when configuring mergerfs

newfd = openat(fd, "relative/path")
do_thing(newfd)

// or

do_thing(fd,"relative/path")

In which case their is less path traversal and can be slightly faster / less overhead. But if I had to do open (as root), change to user, openat, do_thing each time rather than change user, open, do_thing. it would be slower.

[snyssen]

Ah yes that makes sense, thanks for the explanation