Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use OverflayFS instead of bind mounts #164

Closed
Infinoid opened this issue May 21, 2024 · 5 comments · Fixed by #215
Closed

Use OverflayFS instead of bind mounts #164

Infinoid opened this issue May 21, 2024 · 5 comments · Fixed by #215

Comments

@Infinoid
Copy link
Contributor

PR #162 fixed a bug where executing a command in a Node failed to bind-mount a nonexistent file. However, it fixes this by creating an empty file and then bind-mounting those. After this, iproute2 will see the empty file and behave differently, running on a Node or running natively.

The iproute2 package recently (v6.5.0) changed to not install config files in /etc/iproute2. Instead, it puts default configs in /usr/share/iproute2, leaving /etc/iproute2 for system administrators who want to override those defaults.

[Note: The path /usr/share/iproute2 depends on the build configuration. That is where the Debian package puts it; other builds may have different paths.]

Gont tries to bind-mount the /etc/iproute2/group file into a mount namespace. If it can't find the file, it creates an empty one. This effectively overrides the defaults in /usr/share/iproute2.

This function in the iproute2 sources tries to open the file in /etc. If it doesn't find one, it tries to open the default file in /usr instead. The file in /etc prevents it from looking in /usr.

So, on a machine which has never run gont:

  1. iproute2 is run
  2. it tries to read the group config
  3. it finds the file in /usr/share/iproute2/group
  4. it parses the file and finds one group: 0 default

On that same machine, after gont ran:

  1. iproute2 is run
  2. it tries to read the group config
  3. it finds the file in /etc/iproute2/group
  4. it parses the file and finds no groups

I don't know exactly what happens when iproute2 has no groups. But its configuration has changed, and that's the bug.

What do you think about leaving it unmounted, instead? If a native process doesn't see the file, it uses the default instead. I think the same process running in a mount namespace will also use the default, and that's what we want.
@Infinoid Infinoid mentioned this issue May 21, 2024
Closed
@stv0g
Copy link
Collaborator

stv0g commented May 21, 2024

@Infinoid thanks for reporting this. Yes, I indeed added the creation of empty files/directories in order to have a target for the bind mount.

I think we have two options:

  • Delete the files after the Gont network has been torn down
  • Use a completely different approach like OverlayFS to mount the network/host specific network files.
    • OverlayFS sounds nice initially. But I think it also has its disadvantages: modifications to the lowerdir (the original rootfs or /etc) are not allowed.

@Infinoid
Copy link
Contributor Author

I like the overlayfs idea. Another option would be to make the file in /tmp, and then bind-mount that.

I've been very focused on this one /etc/iproute2/group file, because that's the problem in front of me, but I don't know what else you are modifying. So maybe there's a good reason that I've missed. Could you give me an example of when it is necessary to modify the original /etc folder?

@Infinoid
Copy link
Contributor Author

I think there are two problems with making changes to /etc, even temporarily:

  • gont is not necessarily the only thing running on the system, and temporary changes may confuse something else
  • lose power at the wrong moment and your temporary changes become permanent

@stv0g
Copy link
Collaborator

stv0g commented May 26, 2024

I think there are two problems with making changes to /etc, even temporarily:

I agree. We should not touch /etc/.

Another option would be to make the file in /tmp, and then bind-mount that.

No, that wouldnt work. As we need to have a file/directory in place at the target site (/etc) for the bind mount. We can not bind mount to a non-existing file/dir.

I would like to focus on the OverlayFS approach. I made a simple test on my machine with promising results.

@stv0g stv0g changed the title Don't create empty files in /etc/iproute2 Use OverflayFS instead of bind mounts Jun 17, 2024
@stv0g
Copy link
Collaborator

stv0g commented Oct 27, 2024

My initial PoC for using OverlayFS has stalled as I ran into two blocking issues:

  • Modifications to the underlying (lowerdir) filesystem of an OverlayFS mount are not allowed and result in undefined behaviour.
  • Changes performed by the executed process are not written back to the underlying filesystem but instead to the upperdir / workdir. Hence, they will be lost once the process terminates and we deconstruct the mountpoint.

For these reasons, I have decided that I will stick with the the bind mount points and revert my previous changes which created empty targets directories & files.
Instead, we will throw a more descriptive error and ask the user to create those.

By Gont's current defaults, we are only attempting to bind mount the following (patched) files:

  • /etc/hosts
  • /etc/iproute2/group
  • /etc/nsswitch.conf

We may even skip the bind mounts and just show a small warning? Or nothing at all.

Using an OverlayFS might still be benefitial in some use cases. However, those are probably better addressed by using a proper container runtime. Supporting those better is now on the agenda (see #5).

@stv0g stv0g mentioned this issue Oct 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix bind mounts cunicu/gont
Don't create files in /etc/iproute2 out of thin air Infinoid/gont
2 participants
@Infinoid @stv0g