Port mapping not working

[tuefue]

Before you report an issue...

• Have you searched for a duplicate report?
• Have you replicated the problem on the latest release?
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Version of Singularity

singularity-ce version 4.2.1-noble

Describe the bug

Port mapping causes errors.

ERROR:   could not delete networks: plugin type="portmap" failed (delete): neither iptables nor ip6tables is usable, (iptables) could not get iptables version: exit status 111, (ip6tables) could not get iptables version: exit status 111
FATAL:   container creation failed: plugin type="bridge" failed (add): failed to locate iptables: could not get iptables version: exit status 111

To Reproduce

Run command:

singularity run --net --network-args "portmap=8000:8000/tcp" img.sif

OS / Linux Distribution

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

up-to-date

Installation Method

DEB from GitHub (https://github.com/sylabs/singularity/releases/download/v4.2.1/singularity-ce_4.2.1-noble_amd64.deb)

Additional context

iptables command output:

iptables v1.8.10 (nf_tables): no command specified
Try `iptables -h' or 'iptables --help' for more information.

ip6tables command output:

ip6tables v1.8.10 (nf_tables): no command specified
Try `ip6tables -h' or 'ip6tables --help' for more information.

[dtrudg]

Hi @tuefue - thanks for the report.

I believe that this issue is due to a lack of full support for nf_tables (which has replaced iptables), when using the CNI ipmasq / portmap plugins. Newer Linux distributions, which have moved to nf_tables, will hit issues with certain CNI configurations.

Luckily, there is a potential solution on the horizon. The upstream CNI plugins project has recently merged a change that adds support for nftables on ipmasq / portmap plugins:

containernetworking/plugins#935

This project is what we import into SingularityCE to gain CNI networking functionality. There is no release yet with the fix in it, but as soon as there is then we can update the dependency and support for portmap with nf_tables should then be working. We'll monitor the status of the CNI plugins project until that happens.

[tuefue]

It's released https://github.com/containernetworking/plugins/releases/tag/v1.6.0.

• nftables support for ipmasq and portmap by @danwinship in nftables support for ipmasq and portmap containernetworking/plugins#935

[dtrudg]

Dependabot has opened some pull requests to bring in the dependency update onto our branches.

Unforunately, the update CNI plugins dependency requires Go 1.23. Our current documented policy is to support the current (1.23) and previous (1.22) Go version for stable releases. We'll have to have a think about the policy (the issue has also come up with other dependencies recently).

Chiefly the issue is with building packages for EPEL etc. where the Go version may lag behind upstream.

There will be a delay in getting a release out with this due to the issue above.

[sukim96]

Hi. Is there any update on this issue?

I have same problem using fakeroot.
In my Rocky9 machine, I updated containernetworking-plugin as v1.6.1, but it still doesn't work.
Is there an upcoming patch scheduled to use this new plugin?

Also I wonder if it's problem on newer iptables library.
My Ubuntu 22.04 machine uses iptables v1.8.7, and works well.
My Rocky9 machine uses iptables v1.8.10, and it has same error.

After a few googlings, I found that iptables v1.8.8 has changelog as below:

libxtables: exit if called by setuid executeable

And for manpage of iptables v1.8.9, such message is inserted:

BUGS
       Bugs?   What's  this? ;-) Well, you might want to have a look at https://bugzilla.netfilter.org/ iptables will exit immediately with an error code of 111 if it finds that it was called as a setuid-to-root program.

So I doubt that it's because of newer iptables library...
Is there any way to portmap without using iptables?

[dtrudg]

Ahh okay - the iptables command from v1.8.8 onward is blocking execution from a setuid executable, via a real vs effective uid comparison, to avoid issues in the case where the setuid executable doesn't sanitize the environment before calling iptables.

Singularity does clear the environment before calling the CNI plugin, which will in turn call iptables ... but the iptables binary no longer allows execution from any setuid context.

This will require a bit of thought. Likely we can set the real & effective uid to be the same, avoiding the error.... but this needs consideration of what else is using the escalation code shared by CNI plugin execution.