Why is no nonvolatile (overlayed) container possible when using podman run --rootfs /path/to/rootfs:O

[maikelmeyers79]

Feature request description

First I was happy to have the :O flag of the --rootfs option of podman run, in order to not modify contents of in the rootfs path itself. An overlayfs for the container is useful to have the same advantages as when using an image store. But then I noticed, that this container is volatile. The container still exists after a reboot, but changed contents vanished. Why is it not possible to make them nonvolatile over reboots? I did not find any option to do this.

Suggest potential solution

Provide an option to make this containers changed contents non-volatile.

Have you considered any alternatives?

Only alternative is to use an image store, but this is not an option for us in an embedded linux device, because images can not be preinstalled.

Additional context

Add any other context or screenshots about the feature request here.

[Luap99]

For volumes we document:

For advanced users, the overlay option also supports custom non-volatile
upperdir and workdir for the overlay mount. Custom upperdir and
workdir can be fully managed by the users themselves, and Podman does not
remove it on lifecycle completion.
Example :O,upperdir=/some/upper,workdir=/some/work

But that does not seem to be support with --rootfs so I guess it would make sense to allow that there as well

[maikelmeyers79]

I wonder why it was decided to make this volatile? A normal container of an image is not volatile. It's kind of inconsistent. The container is created somethere in /var/lib/containers and after a reboot it is still there, but all changes made to the rootfs of the container are lost. Who needs this behaviour? If I want to delete all changes, I will delete the container. Who needs a non-volatile container with volatile data?

[rhatdan]

I don't recall if there was a reason for this. But I think your arguments are correct. @giuseppe and I worked on this many years ago, so perhaps he recalls.

I was thinking there could be some conflict if the underlying rootfs is changeing, which is undefined in OverlayFS.

So sharing / as Overlay and then changing content underneath / could cause issues.

[giuseppe]

can you please share what commands you are using and all the information requested in the issue template?

I tried the following test locally, and the changes persist:

$ podman run -i --name test --rootfs ~/path/to/rootfs:O sh -c 'wc -l /hello; echo 1 >> /hello'
wc: /hello: No such file or directory
$ podman start -a test
1 /hello
$ podman start -a test
2 /hello
$ podman start -a test
3 /hello
$ podman start -a test
4 /hello

So every time the container is restarted, it can see the changes performed before.

[rhatdan]

I thought he was talking about surviving a reboot.

[giuseppe]

We don't do anything special on a reboot, unless --transient-store is used. I've tested the same command after a reboot, and it works in the same way for me.

[maikelmeyers79]

can you please share what commands you are using and all the information requested in the issue template?

I tried the following test locally, and the changes persist:

$ podman run -i --name test --rootfs ~/path/to/rootfs:O sh -c 'wc -l /hello; echo 1 >> /hello'
wc: /hello: No such file or directory
$ podman start -a test
1 /hello
$ podman start -a test
2 /hello
$ podman start -a test
3 /hello
$ podman start -a test
4 /hello

So every time the container is restarted, it can see the changes performed before.

Not working here:

root@raspberrypi4-64 # podman --version
podman version 4.0.3-dev
~
root@raspberrypi4-64 # podman run -i --name test --rootfs /path/to/rootfs/:O sh -c 'wc -l /hello; echo 1 >> /hello'
wc: /hello: No such file or directory
~
root@raspberrypi4-64 # podman start -a test
1 /hello
~
root@raspberrypi4-64 # podman start -a test
2 /hello
~
root@raspberrypi4-64 # podman start -a test
3 /hello
~
root@raspberrypi4-64 # reboot

.
.
.

~
root@raspberrypi4-64 # podman start -a test
wc: /hello: No such file or directory
~
root@raspberrypi4-64 # podman start -a test
1 /hello
~
root@raspberrypi4-64 # podman start -a test
2 /hello
~
root@raspberrypi4-64 # cat /etc/containers/storage.conf | grep graphroot
graphroot = "/user-data/containers"
~
root@raspberrypi4-64 # mount | grep user-data
/dev/mmcblk0p7 on /user-data type ext4 (rw,noatime,nodiratime,discard)

As you can see, the container still exists, but changed contents vanished. The documentation says: "Modifications to the mount point are destroyed when the container finishes executing, similar to a tmpfs mount point being unmounted."

[maikelmeyers79]

It seems that all changes go into /run/containers/storage which is tmpfs. In storage.conf it seems to be this setting:

runroot = "/run/containers/storage"

But the documentation of storage.conf says:

# Temporary storage location
runroot = "/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

So what is the Temporary storage location for? I thought the directory of graphroot is used for the non-volatile overlayfs directorys.

[maikelmeyers79]

Maybe this is a special issue with the podman version of Yocto

[Luap99]

We only support the lastest upstream release. This seems to work since 0e949bf v4.5

[maikelmeyers79]

We only support the lastest upstream release. This seems to work since 0e949bf v4.5

Now it's clear 👍
Unfortunately, we are tied to Yocto kirkstone, which uses an old version of podman. So the workaround is to adapt runroot to a non-volatile location.

I don't really understand the difference between these two paths. Is it possible to set them to the same location?

[Luap99]

No do not set them to the same dir and in fact is very much expected for rundir to be on a tmpfs as this should not persist. In case of unexpeted reboots there might be stuff in there that we expect to be gone.

If you cannot change our podman version that maybe just mount the overlay on your own and pass the mount point as rootfs to podman.

[maikelmeyers79]

No do not set them to the same dir and in fact is very much expected for rundir to be on a tmpfs as this should not persist. In case of unexpeted reboots there might be stuff in there that we expect to be gone.

If you cannot change our podman version that maybe just mount the overlay on your own and pass the mount point as rootfs to podman.

Good idea. What I try to do first, is to apply your commit as a patch to the old version.
Thank you all for the excellent help 👍