layer created implicitly by WORKDIR is not cached

[mzihlmann]

Actual behavior
When WORKDIR is called on a non-existent directory, kaniko is kind enough to create that directory for you, resulting in a layer being added. However, kaniko does not cache that layer, which means that on every invocation a completely new image is emitted from that point onwards. Inside the same stage this is non-obvious as caching mechanism still pulls, so you get a 100% cache hitrate thereafter, but the image is completely new. In multistage builds or builds that depend on the newly emitted image, this is catastrophic, as they do consider the entire image's sha when determining whether a cache is hit or not, so this will invalidate the entire cache.

Workaround is simple but silly, we just need to make sure that the directory exists before calling WORKDIR

RUN mkdir /app
WORKDIR /app

Expected behavior

When WORKDIR is called on a non-existent directory, that directory is implicitly created and the resulting layer is pushed into cache.

To Reproduce

FROM ubuntu:latest
WORKDIR /app
RUN echo "whatever"

run this build multiple times, with cache, and observe that the layers change every time

docker inspect <image>

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[mzihlmann]

this is the underlying cause of #3246

[mzihlmann]

problem is here https://github.com/GoogleContainerTools/kaniko/blob/main/pkg/commands/workdir.go#L74
we add to the snapshot but don't cache it.

[nogweii]

Just ran into this issue. Would love to see 3341 merged!