Save BuildKit state on client for cache support

[crazy-max]

fixes #122

Signed-off-by: CrazyMax [email protected]

[underyx]

This is the most wonderful PR to ever have graced GitHub with its existence 🥳

[crazy-max]

@underyx

This is the most wonderful PR to ever have graced GitHub with its existence 🥳

Unfortunately there are some issues to copy back from the volume to a local folder for the GitHub Action cache.

One of them is some snapshots have device based permissions: 8/work/work/#b0

    │       │   └── [drwx--x--x root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/8/work
    │       │       └── [d--------- root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/8/work/work
    │       │           └── [c--------- root     root        0 Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/8/work/work/#b0
    │       └── [drwx------ root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/9
    │           ├── [drwxr-xr-x root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/9/fs

Others are perms on runc-overlayfs dir and cache.db:

├── [-rw------- root     root      64K Apr 22 11:42]  /tmp/.buildkit-state/cache.db
└── [drwx------ root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs
    ├── [drwx------ root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/cachemounts
    ├── [-rw-r--r-- root     root     512K Apr 22 11:43]  /tmp/.buildkit-state/runc-overlayfs/containerdmeta.db
    ├── [drwxrwxrwx root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/content

So cache fails to be uploaded:

Post job cleanup.
/usr/bin/tar --posix --use-compress-program zstd -T0 -cf cache.tzst -P -C /home/runner/work/test-docker-action/test-docker-action --files-from manifest.txt
/usr/bin/tar: ../../../../../tmp/.buildkit-state/runc-overlayfs: Cannot open: Permission denied
/usr/bin/tar: ../../../../../tmp/.buildkit-state/cache.db: Cannot open: Permission denied
/usr/bin/tar: Exiting with failure status due to previous errors
Warning: Tar failed with error: The process '/usr/bin/tar' failed with exit code 2

As every actions are executed with a standard user it seems that's something we can't workaround atm.

But even after succeeding to fix the perms issues, loading the cache back and forth from the official cache action that does tar comp/uncomp, it takes more time than the build itself 😞.

After discussing with @tonistiigi about this our best shot is to be able to export cache mounts. See moby/buildkit#1512

[rarecrumb]

@crazy-max anything we can do to get this released?