krun can't pass -- correctly

[karuboniru]

Issue Description

podman run --runtime=krun --entrypoint echo -it --rm fedora:41 -- test don't display everything while it should give -- test

Steps to reproduce the issue

Steps to reproduce the issue

1. podman run --runtime=krun --entrypoint echo -it --rm fedora:41 -- test
2. podman run --runtime=crun --entrypoint echo -it --rm fedora:41 -- test

Describe the results you received

in step 1 no output is produced, but in step 2 I get -- test

Describe the results you expected

The behavior should be consistent and the result should be equivlent to running echo -- test on the host.

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 94.77
    systemPercent: 1.02
    userPercent: 4.21
  cpus: 96
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: server
    version: "41"
  eventLogger: journald
  freeLocks: 2039
  hostname: epyc-server
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.13.5-200.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 725938176
  memTotal: 134346657792
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250217.ga1e48a0-2.fc41.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 8577871872
  swapTotal: 8589930496
  uptime: 8h 4m 58.00s (Approximately 0.33 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/yan/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /var/home/yan/.local/share/containers/storage
  graphRootAllocated: 2000398934016
  graphRootUsed: 674461384704
  graphStatus:
    Build Version: Btrfs v6.12
    Library Version: "104"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 15
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /run/media/cache/yan/containers/storage/volumes
version:
  APIVersion: 5.4.0
  BuildOrigin: Fedora Project
  Built: 1739232000
  BuiltTime: Tue Feb 11 08:00:00 2025
  GitCommit: ""
  GoVersion: go1.23.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

I know -- is very special in command line phrasing, maybe there is some program stopped phrasing incorrectly when seeing the -- in the chain of passing the full command?

[eriksjolund]

Just a guess without checking it: Could it be related to this issue?

• parsed_entrypoint typo in init/init.c #265

[Luap99]

Yes this should be filled against crun or krun even, podman passe the same spec regardless of runtime so this is not a podman issue. I move it to crun.
cc @giuseppe @slp

[giuseppe]

@slp the --, test arguments are configured with krun_set_exec().

Should be reassigned to libkrun?

[giuseppe]

https://github.com/containers/libkrun/blob/main/src/libkrun/src/lib.rs#L1166 seems to get confused with another --.

Reassigning

EDIT: the issue seems to be in the cmdline handling, these arguments seem to get lost once in the kernel.

@slp is there any reason to not always use the config.json file when it is available?

[slp]

@slp is there any reason to not always use the config.json file when it is available?

No, in fact we intend to deprecate krun_set_exec() in libkrun 2.0 (which will redesign the API). We should probably just stop using krun_set_exec() in crun.

[giuseppe]

thanks for confirming it. Opened a PR: containers/crun#1689

[giuseppe]

returning to the original issue, the kernel seems to call twice parse_args():

        if (!IS_ERR_OR_NULL(after_dashes))
                parse_args("Setting init args", after_dashes, NULL, 0, -1, -1,
                           NULL, set_init_arg);

after_dashes is what it is obtained the first time, so it ends up discarding everything after and including the second --.

A quick hack like the following patch seems to fix it, but I've not tested it carefully:

➜ diff -Naur linux-6.12.12/kernel/params.c.old linux-6.12.12/kernel/params.c
--- linux-6.12.12/kernel/params.c.old   2025-03-07 11:24:19.438764863 +0100
+++ linux-6.12.12/kernel/params.c       2025-03-07 11:23:52.109603366 +0100
@@ -180,7 +180,7 @@

                args = next_arg(args, &param, &val);
                /* Stop at -- */
-               if (!val && strcmp(param, "--") == 0) {
+               if (!val && min_level >= 0 && strcmp(param, "--") == 0) {
                        return err ?: args;
                 }
                irq_was_disabled = irqs_disabled();

For crun, I think just avoiding krun_set_exec() is enough

[mtjhrc]

Regarding fixing krun_set_exec:
I think the limit for kernel command line is also too small. If the limit of the kernel command line could be increased, we could pass in something like a base64 encoded json with the arguments and decode in the init. Or we could pass it using vsock (or a virtio-console pipe).

In it's current state, you probably shouldn't use krun_set_exec and use the json config instead.