-
Notifications
You must be signed in to change notification settings - Fork 66
How building works
Version: G3v2
Updated: 2022-11-04
This page describes the actual implementation of the second version (G3v2) of the building pipeline.
Before reading this page it can be helpful to read the pages Concepts of Dockerfiles, Concepts of building, Concepts of README files and Building stages first.
The building pipeline originally began as a mirror of the standard Docker Hub auto-building pipeline as it is described on the page Advanced automated builds. However, it has evolved a little bit since then.
The Docker Hub has removed the auto-building feature from the free plan and therefore the building on the Docker Hub stage has been abandoned.
The local stage is the default place for building the images now. Also a CI/CD building in a local GitLab installation in a Docker container on a Linux machine has already been tested.
It should be noticed, that the version G3v2 of the building pipeline offers a significantly higher performance by building sets of images or by repeated builds. It's achieved by utilizing the local g3-cache, which is described separately.
The complete building pipeline consists of
- hook scripts
- source files
- helper scripts
They are all stored in the folder docker/hooks/.
Also the Dockerfiles stored in the docker/ folder are part of the pipeline.
The docker/ folder itself is also referred as a docker building context or simply a building context.
The hook scripts pre_build, build, push and post_push make the actual building pipeline.
The source files are the helper files that contain additional source code, that is sourced by the hook scripts.
The helper scripts are used as external utilities.
The pipeline file structure looks like this:
-
docker/-
hooks/buildcacheenv.rcpost_pushpre_buildpushrelease_ofutil.rc
Dockerfile.xfce
-
It should be noted that some of the hook scripts use, or even require, some specific environment variables to be set.
The building pipeline can be executed completely or per parts.
All hook scripts require the following two inputs arguments
branchblend
These are explained on the page Concepts of building (section Other building concepts).
The hook scripts pre_build and build can take also additional Docker CLI arguments (e.g --no-cache), that will be passed to the docker build command used internally.
The common usage pattern:
<hook-script> <branch> <blend> [[--no-cache] [<other-docker-cli-options>]]
Additionally to the input arguments, also some environment variables play a role in the building process. Those are described on the page Building stages.
The source files contain additional source code, that is sourced by the hook scripts.
This file prepares the building environment and is conceptually close to a configuration file. This file is source by all hook scripts.
The code in env.rc processes the input arguments branch and blend and sets all the variables, that are necessary for building a particular image.
The best description is probably an example of a possible source code. Note that this is not the actual code, but an illustration of the concept.
case "${_branch}" in
# default (master), developer (dev, dev-*) and release (v*) builds
master | dev | dev-* | v* )
# feature blends in this building branch
case "${_blend}" in
vnc )
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-g3"
_deploy_tags=( "vnc" )
_readme_context="docker/xfce"
;;
latest | vnc-novnc )
FEATURES_NOVNC=1
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-g3"
_deploy_tags=( "latest" "vnc-novnc" )
_readme_context="docker/xfce"
;;
vnc-fugo )
FEATURES_USER_GROUP_OVERRIDE=1
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-g3"
_deploy_tags=( "vnc-fugo" )
_readme_context="docker/xfce"
;;
vnc-chromium )
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce.chromium"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-chromium-g3"
_deploy_tags=( "vnc" )
_readme_context="docker/xfce-chromium"
CHROMIUM_VERSION=$( "${_mydir}"/release_of 'chromium-browser' )
;;
*)
echo "Unsupported blend '${_blend}'"
exit 1
;;
esac
;;
# experimental builds
experimental | exp-* )
# feature blends in the experimental branch
case "${_blend}" in
rdp )
# set some variables here
;;
nomachine )
# set some variables here
;;
*)
echo "Unsupported blend '${_blend}'"
exit 1
;;
esac
;;
*)
echo "Unsupported branch '${_branch}'"
exit 1
;;
esac
This file contains some utility procedures and is also sourced by all hook scripts.
Examples of the utility procedures:
get_gist_file()update_gist()
This file is used for setting the environment variables.
If the source file env.rc finds the file secrets.rc in the folder docker/hooks/, then it sources it automatically, so you don't need to do it manually.
You can use the provided file example-secrets.rc as a template.
Be careful to exclude the file from your public commits, because it probably contains the real secrets. Adjust the .gitignore file correctly.
The helper scripts are used as external utilities.
This helper script returns the current release versions of the products, that are installed explicitly by their versions (e.g. Chromium Browser).
This helper script refreshes the local g3-cache, which must be always located inside the docker building context.
Note that this helper script is often referenced also as a hook script.
It is always executed by the pre_build and build hook scripts. However, it can be used also stand-alone. For example for checking, if there are newer versions of the packages that are normally downloaded from the external sources.
The g3-cache and the rules for its refreshing are described separately.
The hook scripts would override the default auto-building handlers on the Docker Hub stage, as it is described on the page Advanced automated builds.
However, since the version G3v2 the local stage took the role of the default building stage. Nevertheless, the original naming of the hook scripts has not been changed.
This script builds a temporary helper image, gets the current verbose version sticker value of it and stores the value into the temporary helper file scrap-version_sticker-verbose_current.tmp.
Then it gets the verbose version sticker value of the previous image release from the GitHub Gist belonging to the builder repository and stores the value into the temporary helper file scrap-version_sticker-verbose_previous.tmp.
The contents of the both helper files are then compared and if they are the same, then an another temporary helper file named scrap-demand-stop-building is created. The file is actually empty, but its sheer presence has the meaning of a demand building stop command.
The comparison itself can be overridden and the image re-building can be forced by setting the environment variable FORCE_BUILDING=1. The environment variable PROHIBIT_BUILDING=1 has the opposite effect.
If the building process should continue, then the short version sticker value is got using the helper image. The value is stored into the temporary helper file scrap-version_sticker_current.tmp.
After that the temporary helper image is not needed any more and it is deleted.
Executing the pre_build hook script is not necessary in all scenarios.
It is required only if the up-to-date version sticker values are needed. This is usually the case when the images are to be published to the Docker Hub.
As it can be seen from the actual source code, the pre_build hook script can process also additional Docker CLI arguments, that will be passed to the internally used docker build command (notice the part docker build $@). Probably the most usual case is for refreshing the Docker builder cache:
./docker/hooks/pre_build dev vnc --no-cacheThe Docker builder cache should not be confused with the local g3-cache, which is described separately.
The pre_build hook script always refreshes the local g3-cache before it starts building the temporary helper image. It uses the helper script cache for that.
It should be noticed, that the temporary helper image name has the name and the tag of the builder repository. Additionally, its name gets the suffix _helper by default.
TIP: The pre_build hook script is the only script you need to execute, if you only want to check, if the image needs a refresh.
This script builds the persistent final image and it also sets its created and version-sticker labels to the up-to-date values.
The docker build command gets a number of arguments that are partially dependent on the feature variables set by the env.rc source file.
Before building the image, the build hook script refreshes the local g3-cache by executing the cache helper script and checks if the temporary helper file scrap-demand-stop-building with the meaning of a demand building stop command is present. If the file is found, then the script does not continue and it exits gracefully.
Similar to the previously described pre_build hook script, the build hook script can also get additional Docker CLI arguments (e.g. --no-cache), that will be passed to the internally used docker build command.
It should be noticed, that the final image has the name and the tag of the builder repository.
TIP: The build hook script is the only script you have to execute, if you want to build an image only for a local use and you are not interested in the up-to-date values of the image's labels.
This script pushes the persistent final image into its target deployment repository on the Docker Hub.
The image gets the name and the tag of the deployment repository.
The same final image can be pushed several times into the same deployment repository, each time with a different deployment tag.
Before building the image, the script checks, if the temporary helper file scrap-demand-stop-building with the meaning of a demand building stop command is present. If it is found, the script does not continue and it exits gracefully.
You don't need to execute this script in all scenarios. Only if you want to push the image to its deployment repository on the Docker Hub.
It should be noticed, that if the target repository has the reserved name void, then the push will be skipped.
This script first checks, if the temporary helper file scrap-demand-stop-building with the meaning of a demand building stop command is present. If it is found, then the script does nothing and it exits gracefully.
In its normal course the post_push hook script begins with getting the values of the labels created and version-sticker (the short version sticker) from the persistent final image. The current value of the verbose version sticker is got from the temporary helper file created by the pre_build hook script.
All three values are then published to the GitHub Gist belonging to the builder repository.
Then the same three values are published also to the GitHub Gist belonging to the deployment repository. This is done for each deployment tag.
The other published data include the JSON endpoints for the badges that are embedded in the README files.
At the end this hook script removes the temporary helper files, unless the environment variable KEEP_HELPER_FILES is not set to 1.
You don't need to execute this script in all scenarios. Only if you want to update the GitHub Gists.
It should be noticed, that since the version G3v2 this hook script does not publish the README file to the Docker Hub. It has to be done separately as it is described bellow.
If you want to execute the complete building pipeline, then you don't need to execute the hook scripts individually.
Since the version G3v2 you can use the provided helper scripts builder.sh and ci-builder.sh that can build the individual images or also sets of images. They are described on the page How CI works.
The scripts also create the log files scrap_builder.log and scrap_ci-builder.log in the project's root directory.
The automatic publishing of the README files to the Docker Hub had to be removed, because it was not working properly any more.
However, the README files for the Docker Hub can still be prepared with the provided utility util-readme.sh.
The content of the generated file scrap-readme.md can then be copy-and-pasted to the Docker Hub manually.
The Wiki page "Utility util-readme.sh" describes how to use the utility.