Symlinks don't work on podman machine on macOS Monterey when using volumes feature

[shadowusr]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Symlinks don't seem to work on podman machine on macOs Monterey with volumes. Any attempt to interact with symlinks ends with the following error:

ls: cannot access '/Users/user/some_directory': Timer expired

This bug prevents me from using podman, because my project uses pnpm which extensively uses symlinks.

Steps to reproduce the issue:

1. brew install podman

2. podman machine init --volume /Users

3. podman machine start

4. Create some symlink on host machine e.g. ln -s ~/some_dir/ /Users/my_symlink

5. podman ssh

6. (inside podman machine VM): ls /Users/my_symlink => results in an error

Describe the results you received:
Error ls: cannot access '/Users/user/some_directory': Timer expired

Describe the results you expected:
Symlinks work exactly as they do on the host machine.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.0.3
API Version:  4.0.3
Go Version:   go1.18
Built:        Fri Apr  1 18:28:59 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.16.14
Built:        Thu Mar  3 17:56:56 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 738473217
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.15.18-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1346039808
  memTotal: 2061381632
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/738473217/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 37m 24.82s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 0
    stopped: 8
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/738473217/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319416
  BuiltTime: Thu Mar  3 17:56:56 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

brew info podman

podman: stable 4.0.3 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/4.0.3 (172 files, 47.6MB) *
  Poured from bottle on 2022-04-05 at 15:49:03
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Analytics
install: 20,716 (30 days), 47,460 (90 days), 118,586 (365 days)
install-on-request: 20,709 (30 days), 47,440 (90 days), 118,562 (365 days)
build-error: 9 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
physical, MacBook

[mheon]

@baude PTAL

[baude]

Podman 4.0.3 is not a supported release imho. Where did it come from?

[shadowusr]

@baude as I mentioned in my original post, it came from brew install podman.

[willcohen]

Symlink issues noticed with lima as well: lima-vm/lima#726

[acdha]

I ran into the same problem using a Python Poetry project with the VS Code remote containers feature. I haven't looked in depth into the volume configuration but I had a .venv file created natively on macOS which had the usual bin/python symlink. Deleting the .venv and reinstalling inside the container fixed that problem, which in my case would have been necessary in any case, but feels somewhat unsatisfying.

[zwjzxh520]

same issue with use pnpm

/var/workdir # ls -l
total 0
ls: ./gulp: cannot read link: Device timeout
lrwxrwxrwx    1 501      root            36 Oct 24  2019 gulp
ls: ./pnpm: cannot read link: Device timeout
lrwxr-xr-x    1 501      root            36 Mar  5  2021 pnpm

symlink is:

lrwxrwxrwx  1 xxx  xxx    36B 10 24  2019 gulp -> ../lib/node_modules/gulp/bin/gulp.js
lrwxr-xr-x  1 xxx  xxx    36B  3  5  2021 pnpm -> ../lib/node_modules/pnpm/bin/pnpm.js

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ncdc]

@baude @mheon I'm just now running into this as well - podman 4.1.1. Trying to do some work with the openshift/release repo, which has some symlinks. Any updates on this?

[rhatdan]

Is the symlinked directory mounted into the VM at the same location?

[ncdc]

Yes

[rhatdan]

If you podman machine ssh, does it look like the symlinks are correct?

[ncdc]

No / not possible to tell

stat ipi-install-install-stableinitial-commands.sh
  File: ipi-install-install-stableinitial-commands.shstat: cannot read symbolic link 'ipi-install-install-stableinitial-commands.sh': Timer expired

  Size: 34              Blocks: 0          IO Block: 126976 symbolic link
Device: 0,49    Inode: 32954698    Links: 1
Access: (0755/lrwxr-xr-x)  Uid: (  501/    core)   Gid: (   20/   games)
Context: system_u:object_r:nfs_t:s0
Access: 2022-06-06 15:08:43.511551861 -0400
Modify: 2022-06-06 15:08:43.511551861 -0400
Change: 2022-06-06 15:08:43.511551861 -0400
 Birth: -

It should be pointing to a file in the parent dir (here's the output from my Mac)

lrwxr-xr-x  1 angoldst  staff    34 Jun  6 15:08 ipi-install-install-stableinitial-commands.sh -> ../ipi-install-install-commands.sh

[bugfood]

This seems to be due to the choice of 9p security model.
https://github.com/containers/podman/blob/d7121b0/pkg/machine/qemu/machine.go#L339

If you change mapped-xattr to none, then symlinks work as expected (though possibly with other ramifications).

The documentation of the purpose and behavior of the different security models seems to be rather light:
https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly

From testing, it appears that the mapped-xattr security model intends to manage symlinks such that the guest can see the symlinks but the host only sees regular files (with extended attributes). As far as I can tell, this behavior only makes sense when the guest is the only thing that ever needs to create and read symlinks. Otherwise, symlinks created on the host are unusable on the guest, and vice versa.

This is consistent with the information noted here:
https://gitlab.com/qemu-project/qemu/-/issues/173#note_662730194

For the podman machine, unless anybody has a better idea, I suggest making the security model be configurable. Using none might be a better default, but I don't know what use cases that would break, if any.

[rhatdan]

I think the default of none does make sense.

@gbraad @baude WDYT?

[gbraad]

I am not sure, but I remember 9p not mapping correctly attributes when no authentication was used. Although I can be wrong. At the moment I was testing something similar with py9p though no conclusion yet.

So yes, none might be the way to go.

[bugfood]

Sorry for the noise above. I wish github wouldn't do that until there's a PR. This is work in progress.

[bugfood]

I have posted a PR for this. The change makes the security model configurable but does not change the default. I'd vote for changing to default value to none, but that's just a reflection of my own use case. Changing the default might break other use cases and may be best reserved for a major release.

[rhatdan]

SGTM, we can look at changing the default in 5.0, We should also have this default exposed in containers.conf.

[afbjorklund]

As far as I can tell, this behavior only makes sense when the guest is the only thing that ever needs to create and read symlinks.

That was the design, indeed.

[gabriel-rh]

I hit this problem as well with podman on mac, trying to build the OpenShift docs - they use symlinks extensively.

For anyone who lands here with the same issue, I followed the instructions here as a workaround:

https://medium.com/intuit-engineering/how-to-getting-podman-to-correctly-mount-a-native-folder-on-a-mac-73eb2a4ee317

How-to: Getting Podman to Correctly Mount a Native Folder on a Mac

[afbjorklund]

Originally changing from virtfs to sshfs was a feature (like in lima), but it was dropped during the integration...

• Implement virtfs volumes for podman machine #11454

• Implement sshfs volumes for podman machine #12584

The recommendation is still to use a local filesystem (named volume), instead of a remote filesystem (mount)

[bugfood]

Recent podman git has a better way to do this; I presume that will be included in the next release.
03ee820

[afbjorklund]

@bugfood that only changes the virtfs options, it doesn't really change driver (like in that Medium article).
(you might want to add similar options for the others as well, such as protocol version or msize and cache ?)

There was a comment:

Also remove misleading statement about /mnt. By my observation, this line is incorrect. If the intended meaning is different, then I don't understand.

It was a change in CoreOS, to allow empty directories (as mountpoints) in the (read-only) root.

Previously (when the code was written), you were not allowed to change anything but /var
And /mnt was actually a symlink into /var/mnt, so that was an allowed read-write path to use...

Mounting /tmp could still have interesting side-efffects, many assume that to be local (or tmpfs)

[bugfood]

that only changes the virtfs options, it doesn't really change driver (like in that Medium article).

What I mean is that the change makes 9p usable for this use case (and requires much less configuration than sshfs). Is there an advantage to using sshfs still?

It was a change in CoreOS, to allow empty directories (as mountpoints) in the (read-only) root.

Thanks, that makes sense in retrospect.

[afbjorklund]

I think that sshfs is mostly a fallback, where virtfs is not available (such as CentOS 7 guests or Windows hosts). The way forward will be virtiofs, anyway...

Just that your virtfs comment was after a sshfs post.

[graywolf-at-work]

Workaround for those waiting for the next release with built-in support for this:

1. Create the machine with podman machine init -v ...
2. Manually change the security model in ~/.config/containers/podman/machine/qemu/podman-machine-default.json
3. Start the machine

Symlinks now correctly resolve.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[graywolf-at-work]

Thank you for the reminder. Anyway I think this can be closed?

[ssbarnea]

I think we might have to reopen this bug because as of today (4.2.1), symlinks are broken. I wrote more about the issue on #16102

Hacking the vm to make them work is not really what I would expect from users. Is there a way to create the machine from start with correct security_model?

On my machine, i see security_model=mapped-xattr, so now I am going to test with none, and hope it would work better.

[avatarneil]

I'm experiencing a similar issue when attempting to build a NodeJS project which uses a pnpm monorepo.

Error: invalid symlink "/var/tmp/libpod_builder404645871/build/node_modules/@org-monorepo/some-package" -> "../../../../packages/some-package

The error occurs with both security_model=mapped-xattr and security_model=none VMs.

[nettybun]

This seems like a bad user experience on macos... Initially the container mount seems fine but you can't do any real work: find, yarn, jest, and chroot all fail with Too many levels of symbolic links.

Using security_model=none fixes the issue, but it's really unintuitive to debug...

[rhatdan]

I believe the default is being changed to that.