Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port mapping not working #3318

Closed
4 tasks done
tuefue opened this issue Sep 19, 2024 · 5 comments · Fixed by #3444
Closed
4 tasks done

Port mapping not working #3318

tuefue opened this issue Sep 19, 2024 · 5 comments · Fixed by #3444
Labels
bug Something isn't working

Comments

@tuefue
Copy link

tuefue commented Sep 19, 2024

Before you report an issue...

Version of Singularity

singularity-ce version 4.2.1-noble

Describe the bug

Port mapping causes errors.

ERROR:   could not delete networks: plugin type="portmap" failed (delete): neither iptables nor ip6tables is usable, (iptables) could not get iptables version: exit status 111, (ip6tables) could not get iptables version: exit status 111
FATAL:   container creation failed: plugin type="bridge" failed (add): failed to locate iptables: could not get iptables version: exit status 111

To Reproduce

Run command:

singularity run --net --network-args "portmap=8000:8000/tcp" img.sif

OS / Linux Distribution

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

up-to-date

Installation Method

DEB from GitHub (https://github.com/sylabs/singularity/releases/download/v4.2.1/singularity-ce_4.2.1-noble_amd64.deb)

Additional context

iptables command output:

iptables v1.8.10 (nf_tables): no command specified
Try `iptables -h' or 'iptables --help' for more information.

ip6tables command output:

ip6tables v1.8.10 (nf_tables): no command specified
Try `ip6tables -h' or 'ip6tables --help' for more information.
@tuefue tuefue added the bug Something isn't working label Sep 19, 2024
@dtrudg
Copy link
Member

dtrudg commented Oct 4, 2024

Hi @tuefue - thanks for the report.

I believe that this issue is due to a lack of full support for nf_tables (which has replaced iptables), when using the CNI ipmasq / portmap plugins. Newer Linux distributions, which have moved to nf_tables, will hit issues with certain CNI configurations.

Luckily, there is a potential solution on the horizon. The upstream CNI plugins project has recently merged a change that adds support for nftables on ipmasq / portmap plugins:

containernetworking/plugins#935

This project is what we import into SingularityCE to gain CNI networking functionality. There is no release yet with the fix in it, but as soon as there is then we can update the dependency and support for portmap with nf_tables should then be working. We'll monitor the status of the CNI plugins project until that happens.

@tuefue
Copy link
Author

tuefue commented Oct 16, 2024

It's released https://github.com/containernetworking/plugins/releases/tag/v1.6.0.

@dtrudg
Copy link
Member

dtrudg commented Oct 18, 2024

Dependabot has opened some pull requests to bring in the dependency update onto our branches.

Unforunately, the update CNI plugins dependency requires Go 1.23. Our current documented policy is to support the current (1.23) and previous (1.22) Go version for stable releases. We'll have to have a think about the policy (the issue has also come up with other dependencies recently).

Chiefly the issue is with building packages for EPEL etc. where the Go version may lag behind upstream.

There will be a delay in getting a release out with this due to the issue above.

@sukim96
Copy link

sukim96 commented Dec 4, 2024

Hi. Is there any update on this issue?

I have same problem using fakeroot.
In my Rocky9 machine, I updated containernetworking-plugin as v1.6.1, but it still doesn't work.
Is there an upcoming patch scheduled to use this new plugin?

Also I wonder if it's problem on newer iptables library.
My Ubuntu 22.04 machine uses iptables v1.8.7, and works well.
My Rocky9 machine uses iptables v1.8.10, and it has same error.

After a few googlings, I found that iptables v1.8.8 has changelog as below:

libxtables: exit if called by setuid executeable

And for manpage of iptables v1.8.9, such message is inserted:

BUGS
       Bugs?   What's  this? ;-) Well, you might want to have a look at https://bugzilla.netfilter.org/ iptables will exit immediately with an error code of 111 if it finds that it was called as a setuid-to-root program.

So I doubt that it's because of newer iptables library...
Is there any way to portmap without using iptables?

@dtrudg
Copy link
Member

dtrudg commented Dec 4, 2024

Ahh okay - the iptables command from v1.8.8 onward is blocking execution from a setuid executable, via a real vs effective uid comparison, to avoid issues in the case where the setuid executable doesn't sanitize the environment before calling iptables.

Singularity does clear the environment before calling the CNI plugin, which will in turn call iptables ... but the iptables binary no longer allows execution from any setuid context.

This will require a bit of thought. Likely we can set the real & effective uid to be the same, avoiding the error.... but this needs consideration of what else is using the escalation code shared by CNI plugin execution.

dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Pick sylabs#3444

Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
dtrudg added a commit to dtrudg/singularity that referenced this issue Dec 20, 2024
Newer versions of the `iptables` command use a real vs effective uid
check whether they are being called from a setuid script, and exit if
that's the case.

This check was added because iptables can call out to binaries /
libraries on `PATH` / `LD_LIBRARY_PATH`, and these are generally under
control of the user - allowing privilege escalation attackes.

Singularity sanitizes the environment before running CNI plugins, which
will call `iptables`, so we can set both real and effective uid to 0 to
avoid the error.

While we are here, make `PATH` sanitization the default in the network
code, rather than relying on the caller applying it. Add some tests
around the priv escalation / drop code.

Fixes sylabs#3318
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants