wrong --net settings and cni subnet config ignored

[scicco]

Hello,

I'm having trouble with IP lease.

Here are the relevant configuration files:

whoami

root

cat /etc/issue

Ubuntu 22.04.1 LTS \n \l

$> singularity-compose -v

0.1.18

$> cat /usr/local/etc/singularity/singularity.conf | grep "allow net"

allow net networks = bridge, fakeroot
allow net users = deployer
allow net groups = deployer

$> cat /usr/local/etc/singularity/network/00_bridge.conflist (the same IP range for /usr/local/etc/singularity/network/40_fakeroot.conflist)

{
    "cniVersion": "0.4.0",
    "name": "bridge",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "sbr0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.20.0/24",
                "routes": [
                    { "dst": "0.0.0.0/0" }
                ]
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

#singularity-compose.yml
version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
      options:
        - "env-file=./env-file.sh"
      command: redis-server --bind 0.0.0.0 --requirepass ${REDIS_PASSWORD}
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: true
      allocate_ip: true
      ports:
        - "6379:6379"

#redis.def
BootStrap:docker
From:redis:latest

%runscript

    exec /usr/local/bin/docker-entrypoint.sh "$@"

#env-file.sh
export REDIS_PASSWORD=ThisIsNotAStrongPassword!

# running output of:
singularity-compose --debug up
Creating redis1
DEBUG singularity instance start --bind /home/deployer/cgp/singularity-test/env-file.sh:/.singularity.d/env/env-file.sh --bind /home/deployer/cgp/singularity-test/resolv.conf:/etc/resolv.conf --bind /home/deployer/cgp/singularity-test/etc.hosts:/etc/hosts --net --network none --network-args "IP=10.22.0.2" --env-file=./env-file.sh --hostname redis1 --writable-tmpfs /home/deployer/cgp/singularity-test/redis.sif redis1
DEBUG singularity run  instance://redis1
singularity run instance://redis1

$> singularity instance list

INSTANCE NAME    PID      IP    IMAGE
redis1           19524          /home/deployer/cgp/singularity-test/redis.sif

$> singularity-compose ps

INSTANCES  NAME         PID     IP              IMAGE
1         redis1	19524		redis.sif

I think --net --network none --network-args "IP=10.22.0.2" inside singularity-compose debug message
is causing the issue because:

1. it doesn't use bridge
2. it doesn't use my bridge network config: 192.168.20.0/24 instead is using default one (10.22.0.x)

What I'm doing wrong?

Thank you in advance

[vsoch]

We probably need to add support to be able to detect that if singularity doesn’t do it. And you’ve tried with sudo?

I’m not super experienced with custom networking - @pierlauro @PauloMigAlmeida do you have any insights?

[PauloMigAlmeida]

I agree with @vsoch, we need to add support for other variations of network configs if we want to cover these use cases.

@scicco One workaround I had to recently do and I believe it might work for you too (remains to be seen) is to "disable the network" on singularity-compose so it doesn't try add the --net *** params and potentially singularity can pick up the hosting OS network configs (assuming that the bridge config is the default one).

  redis:
    image: cvatredis.sif
    start:
      options:
        - containall
    network:
      enable: false

PS.: if you do that, you won't be able to list the IP addresses when you run singularity instance list as a side-effect of the workaround. Give that a spin to see if that works.

[vsoch]

Thank you @PauloMigAlmeida ! And if there is something extra we can do to make this easier, let's chat about that too.

[scicco]

We probably need to add support to be able to detect that if singularity doesn’t do it. And you’ve tried with sudo?

I've tried with sudo, unfortunately I got the same result

[vsoch]

okay, perhaps try the workaround suggested by @PauloMigAlmeida then?

[scicco]

okay, perhaps try the workaround suggested by @PauloMigAlmeida then?

It does not work either, here is the step done:

$> cat singularity-compose.yml

version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
      options:
        - "env-file=./env-file.sh"
      command: redis-server --bind 0.0.0.0 --requirepass ${REDIS_PASSWORD}
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: false

$> singularity-compose --debug up -d
DEBUG singularity instance start --bind /home/deployer/cgp/singularity-test2/env-file.sh:/.singularity.d/env/env-file.sh --bind /home/deployer/cgp/singularity-test2/resolv.conf:/etc/resolv.conf --bind /home/deployer/cgp/singularity-test2/etc.hosts:/etc/hosts --env-file=./env-file.sh --hostname redis1 --writable-tmpfs /home/deployer/cgp/singularity-test2/redis.sif redis1
DEBUG singularity run  instance://redis1
singularity run instance://redis1

$> singularity instance list
INSTANCE NAME    PID      IP    IMAGE
redis1           15001          /home/deployer/cgp/singularity-test2/redis.sif

[scicco]

What I'm trying to achieve with singularity-compose is the same as the following script steps:

#!/bin/bash

set -eo pipefail

echo "[1/6] stopping existing instance"
singularity instance stop redis* || true

# (docker-compose build --no-cache equivalent) #OPTIONAL
echo "[2/6] cleaning cache and old image"
singularity cache clean -f || true

echo "[3/6] removing old sif image" #OPTIONAL
rm -f redis.sif

echo "[4/6] building sif image"
singularity build redis.sif redis.def || true

echo "[5/6] spinning up a new redis instance"
singularity instance start --hostname redis --writable-tmpfs --net --network=static-redis --network-args "portmap=6379:6379/tcp" $PWD/redis.sif redis

echo "[6/6] launching redis server in background"
singularity exec instance://redis redis-server --bind 0.0.0.0 --requirepass ThisIsNotAStrongPassword! &

$> cat singularity-compose.yml

version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
      options:
        - "env-file=./env-file.sh"
      command: redis-server --bind 0.0.0.0 --requirepass ${REDIS_PASSWORD}
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: true
      allocate_ip: true
      ports:
        - 6379:6379

I also needed to add the following static cni config file to have a specific IP but using default bridge with 192.168.20.0/24 will work and add IP to the instance:

$> cat /usr/local/etc/singularity/network/50_static-redis.conflist

{
  "cniVersion": "0.4.0",
  "name": "static-redis",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "sbr0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "static",
        "addresses": [
          {
            "address": "192.168.20.10/24",
            "gateway": "192.168.20.254"
          }
        ],
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      },
      "snat": true
    }
  ]
}

if I run all commands I got this result:

$> singularity instance list

INSTANCE NAME    PID      IP               IMAGE
redis            15354    192.168.20.10    /home/deployer/cgp/singularity-test/redis.sif

and now it's reachable from redis-cli on localhost (thanks to port forwarding) or either with

redis-cli -h 192.168.20.10

and auth <PASSWORD> must be used to access db (for both commands)

[scicco]

I've created a test repo to simplify debug process. Hope this helps to clarify better my issue

[vsoch]

This is perfect! I'll make some time soon to work on this for you, stay tuned!

[vsoch]

@scicco how does the above work for you without any sudo? It simply does not work for me without sudo - if you add fakeroot, fakeroot isn't allowed to use the custom bridge.

[scicco]

@scicco how does the above work for you without any sudo? It simply does not work for me without sudo - if you add fakeroot, fakeroot isn't allowed to use the custom bridge.

I'm not using fakeroot and running singularity as root user to avoid any permission related issue. If you check inside files of my test repo you should not see any usage of fakeroot. Anyway to use a network I think you have to add it inside singularity.conf. In the test repo I'm launching it with root.

Anyway the following command (according to guide and this one) should allow you to use fakeroot and bridge:

#as root user or using sudo
echo "allow net networks = bridge, fakeroot" >> /etc/singularity/singularity.conf

I'll double check again my test repo to be sure

[vsoch]

Ok gotcha. Since most users don’t by default run as root I’ll modify your example files to have it. I’ll have more time to look at this again tomorrow evening.

[vsoch]

Gotcha - I think I should be able to finish this up and do a PR tonight. Stay tuned!

[vsoch]

See #61