Skip to content

How g3 cache works

Jump to bottom
accetto edited this page Mar 11, 2023 · 2 revisions

How g3-cache works

Version: G3v2

Updated: 2023-03-11

Introduction

The custom g3-cache has been introduced in the second version (G3v2) of the project.

The local g3-cache is an additional cache used by the building pipeline and it should not be confused with the Docker builder cache maintained by the Docker Build itself.

The g3-cache stores the selected pre-downloaded packages used by the Dockerfiles, that would be otherwise repeatedly downloaded from the external sources by each build.

It results in a significantly higher performance by building sets of images or by repeated builds.

You can learn more about the concept on the Wiki page "Concepts of g3-cache".

Ensure wget utility

If you are on Windows, you can encounter the problem of missing wget utility. It is used by refreshing the g3-cache and it's available on Linux by default.

On Windows you have generally two choices. You can build your images inside the WSL environment or you can download the wget.exe application for Windows. Make sure to update also the PATH environment variable appropriately.

Local g3-cache

The local g3-cache is actually used by the Dockerfiles and the Docker builder.

Therefore it must be always placed inside the Docker build context, because the Docker builder has no access to the files outside the context.

The root of the Docker build context by this project is the docker/ subfolder of the projects root folder.

The root of the local g3-cache is the hidden subfolder docker/.g3-cache and underneath it are the section folders (e.g. tigervnc or chromium).

After starting the building pipeline, the hook script pre_build executes the cache helper script, which refreshes the local g3-cache. If the local g3-cache doesn't exist, then it's created.

Because the build hook script can be called also stand-alone, it also executes the cache helper script.

That way it's ensured, that the local g3-cache exists and it's up-to-date before the actual building of the image begins.

The provided Dockerfiles preferably use the packages from the local g3-cache. However, they also fall back to the ad-hoc downloading, if the local g3-cache or the particular package are not available.

Remarks:

The local g3-cache folder docker/.g3-cache is excluded from the commits into the git repository by the means of the .gitignore file.

You can delete the local g3-cache folder any time, because it will be re-created each time you build an image.

Helper script cache

The actual g3-cache refreshing is done by the helper script cache, which is stored in the folder docker/hooks/. Therefore it's sometimes referred also as a hook script.

It starts by sourcing the source file docker/hooks/env.rc, so it's able to update only the packages that are required for building the current particular image.

Then it checks, if the hidden folder docker/.g3-cache exists and if not, then it creates it.

If the folder already existed, then it checks if all the required packages are available.

If the folder did not exist or some packages were not available, then it tries to refresh the local g3-cache content by copying it from the shared g3-cache.

However, the shared g3-cache will be skipped, if the environment variable SHARED_G3_CACHE_PATH is not set or the defined folder is not accessible.

If the required package could not be copied from the shared g3-cache, the it will be downloaded ad-hoc from the external sources on the Internet.

If a fresh package has been downloaded from the Internet and the shared g3-cache is accessible, then the new package will be copied also there.

The cache helper script can be executed also stand-alone:

### PWD = the project's root folder

./builder.sh latest-chromium cache
### or also
./docker/hooks/cache dev latest-chromium

This can be also used as a quick check if some of the packages require a refresh.

Remark: The current implementation of the cache refreshing code is not thread safe and it is not intended for parallel building of multiple images.

Shared g3-cache

The shared g3-cache is placed outside the Docker build context of any of the projects, that use it.

Its root folder can have any name, but the absolute path to it should be set as the value of the environment variable SHARED_G3_CACHE_PATH.

Underneath this root folder are the same section folders as by the local g3-cache.

The content of the shared g3-cache is refreshed by the cache helper script described above.

Clone this wiki locally