Skip to content

Latest commit

 

History

History
328 lines (256 loc) · 14.7 KB

release.md

File metadata and controls

328 lines (256 loc) · 14.7 KB
Raw

libjxl release process

This guide documents the release process for the libjxl project.

libjxl follows the semantic versioning specification for released versions. Releases are distributed as tags in the git repository with the semantic version prefixed by the letter "v". For example, release version "0.3.7" will have a git tag "v0.3.7".

The public API is explicitly defined as C headers in the lib/include directory, normally installed in your include path. All other headers are internal API and are not covered by the versioning rules.

Development and release workflow

New code development is performed on the main branch of the git repository. Pre-submit checks enforce minimum build and test requirements for new patches that balance impact and test latency, but not all checks are performed before pull requests are merged. Several slower checks only run after the code has been merged to main, resulting in some errors being detected hours after the code is merged or even days after in the case of fuzzer-detected bugs.

Release tags are cut from release branches. Each MAJOR.MINOR version has its own release branch, for example releases 0.7.0, 0.7.1, 0.7.2, ... would have tags v0.7.0, v0.7.1, v0.7.2, ... on commits from the v0.7.x branch. v0.7.x is a branch name, not a tag name, and doesn't represent a released version since semantic versioning requires that the PATCH is a non-negative number. Released tags don't each one have their own release branch, all releases from the same MAJOR.MINOR version will share the same branch. The first commit after the branch-off points between the main branch and the release branch should be tagged with the suffix -snapshot and the name of the next MAJOR.MINOR version, in order to get meaningful output for git describe.

The main purpose of the release branch is to stabilize the code before a release. This involves including fixes to existing bugs but not including new features. New features often come with new bugs which take time to fix, so having a release branch allows us to cherry-pick bug fixes from the main branch into the release branch without including the new features from main. For this reason it is important to make small commits in main and separate bug fixes from new features.

After the initial minor release (MAJOR.MINOR.PATCH, for example 0.5.0) the release branch is used to continue to cherry-pick fixes to be included in a patch release, for example a version 0.5.1 release. Patch fixes are only meant to fix security bugs or other critical bugs that can't wait until the next major or minor release.

Release branches may continue to be maintained even after the next minor or major version has been released to support users that can't update to a newer minor release. In that case, the same process applies to all the maintained release branches.

A release branch with specific cherry-picks from main means that the release code is actually a version of the code that never existed in the main branch, so it needs to be tested independently. Pre-submit and post-submit tests run on release branches (branches matching v*.*.x) but extra manual checks should be performed before a release, specially if multiple bug fixes interact with each other. Take this into account when selecting which commits to include in a release. The objective is to have a stable version that can be used without problems for months. Having the latest improvements at the time the release tag is created is a non-goal.

Creating a release branch

A new release branch is needed before creating a new major or minor release, that is, a new release where the MAJOR or MINOR numbers are increased. Patch releases, where only the PATCH number is increased, reuse the branch from the previous release of the same MAJOR and MINOR numbers.

The following instructions assume that you followed the recommended libjxl git setup where origin points to the upstream libjxl/libjxl project, otherwise use the name of your upstream remote repository instead of origin.

The release branch is normally created from the latest work in main at the time the branch is created, but it is possible to create the branch from an older commit if the current main is particularly unstable or includes commits that were not intended to be included in the release. The following example creates the branch v0.5.x from the latest commit in main (origin/main), if a different commit is to be used then replace origin/main with the SHA of that commit. Change the v0.5.x branch name to the one you are creating.

git fetch origin main
git push [email protected]:libjxl/libjxl.git origin/main:refs/heads/v0.5.x

Here we use the SSH URL explicitly since you are pushing to the libjxl/libjxl project directly to a branch there. If you followed the guide origin will have the HTTPS URL which wouldn't normally let you push since you wouldn't be authenticated. The v*.*.x branches are GitHub protected branches in our repository, however you can push to a protected branch when creating it but you can't directly push to it after it is created. To include more changes in the release branch see the "Cherry-picking fixes to a release" section below.

Creating a merge label

We use GitHub labels in Pull Requests to keep track of the changes that should be merged into a given release branch. For this purpose create a new label for each new MAJOR.MINOR release branch called merge-MAJOR.MINOR, for example, merge-0.5.

In the edit labels page, click on "New label" and create the label. Pick your favorite color.

Labels are a GitHub-only concept and are not represented in git. You can add the label to a Pull Request even after it was merged, whenever it is decided that the Pull Request should be included in the given release branch. Adding the label doesn't automatically merge it to the release branch.

Update the versioning number

The version number (as returned by JxlDecoderVersion) in the source code in main must match the semantic versioning of a release. After the release branch is created the code in main will only be included in the next major or minor release. Right after a release branch update the version targeting the next release. Artifacts from main should include the new (unreleased) version, so it is important to update it. For example, after the v0.5.x branch is created from main, you should update the version on main to 0.6.0.

To help update it, run this helper command (in a Debian-based system):

./ci.sh bump_version 0.6.0

This will update the version in the following files:

  • lib/CMakeLists.txt
  • lib/lib.gni, automatically updated with tools/scripts/build_cleaner.py --update.
  • debian/changelog to create the Debian package release with the new version. Debian changelog shouldn't repeat the library changelog, instead it should include changes to the packaging scripts.
  • .github/workflows/conformance.yml

If there were incompatible API/ABI changes, make sure to also adapt the corresponding section in CMakeLists.txt.

Cherry-pick fixes to a release

After a Pull Request that should be included in a release branch has been merged to main it can be cherry-picked to the release branch. Before cherry-picking a change to a release branch it is important to check that it doesn't introduce more problems, in particular it should run for some time in main to make sure post-submit tests and the fuzzers run on it. Waiting for a day is a good idea.

Most of the testing is done on the main branch, so be careful with what commits are cherry-picked to a branch. Refactoring code is often not a good candidate to cherry-pick.

To cherry-pick a single commit to a release branch (in this example to v0.5.x) you can run:

git fetch origin
git checkout origin/v0.5.x -b merge_to_release
git cherry-pick -x SHA_OF_MAIN_COMMIT
# -x will annotate the cherry-pick with the original SHA_OF_MAIN_COMMIT value.
# If not already mentioned in the original commit, add the original PR number to
# the commit, for example add "(cherry picked from PR #NNNN)".
git commit --amend

The SHA_OF_MAIN_COMMIT is the hash of the commit as it landed in main. Use git log origin/main to list the recent main commits and their hashes.

Making sure that the commit message on the cherry-picked commit contains a reference to the original pull request (like #NNNN) is important. It creates an automatic comment in the original pull request notifying that it was mentioned in another commit, helping keep track of the merged pull requests. If the original commit was merged with the "Squash and merge" policy it will automatically contain the pull request number on the first line, if this is not the case you can amend the commit message of the cherry-pick to include a reference.

Multiple commits can be cherry-picked and tested at once to save time. Continue running git cherry-pick and git commit --amend multiple times for all the commits you need to cherry-pick, ideally in the same order they were merged on the main branch. At the end you will have a local branch with multiple commits on top of the release branch.

To update the version number, for example from v0.8.0 to v0.8.1 run this helper command (in a Debian-based system):

./ci.sh bump_version 0.8.1

as described above and commit the changes.

Finally, upload your changes to your fork like normal, except that when creating a pull request select the desired release branch as a target:

git push myfork merge_to_release

If you used the guide myfork would be origin in that example. Click on the URL displayed, which will be something like

https://github.com/mygithubusername/libjxl/pull/new/merge_to_release

In the "Open a pull request" page, change the drop-down base branch from "base: main" (the default) to the release branch you are targeting.

The pull request approval and pre-submit rules apply as with normal pull requests to the main branch.

Important: When merging multiple cherry-picks use "Rebase and merge" policy, not the squash one since otherwise you would discard the individual commit message references from the git history in the release branch.

Publishing a release

Once a release tag is created it must not be modified, so you need to prepare the changes before creating the release. Make sure you checked the following:

  • The semantic version number in the release branch (see lib/CMakeLists.txt) matches the number you intend to release, all three MAJOR, MINOR and PATCH should match. Otherwise send a pull request to the release branch to update them.

  • The GitHub Actions checks pass on the release branch. Look for the green tick next to the last commit on the release branch. This should be visible on the branch page, for example: https://github.com/libjxl/libjxl/tree/v0.5.x

  • There no open fuzzer-found bugs for the release branch. The most effective way is to run the fuzzer on the release branch for a while. You can seed the fuzzer with corpus generated by oss-fuzz by downloading it, for example djxl_fuzzer with libFuzzer will use: gs://libjxl-corpus.clusterfuzz-external.appspot.com/libFuzzer/libjxl_djxl_fuzzer

  • Manually check that images encode/decode ok.

  • Manually check that downstream projects compile with our code. Sometimes bugs on build scripts are only detected when other projects try to use our library. For example, test compiling imagemagick and Chrome.

A GitHub "release" consists of two different concepts:

  • a git "tag": this is a name (v plus the semantic version number) with a commit hash associated, defined in the git repository. Most external projects will use git tags or HTTP URLs to these tags to fetch the code.

  • a GitHub "release": this is a GitHub-only concept and is not represented in git other than by having a git tag associated with the release. A GitHub release has a given source code commit SHA associated (through the tag) but it also contains release notes and optional binary files attached to the release.

Releases from the older GitLab repository only have a git tag in GitHub, while newer releases have both a git tag and a release entry in GitHub.

To publish a release open the New Release page and follow these instructions:

  • Set the "Tag version" as "v" plus the semantic version number.

  • Select the "Target" as your release branch. For example for a "v0.7.1" release tag you should use the "v0.7.x" branch.

  • Use the version number as the release title.

  • Copy-paste the relevant section of the CHANGELOG.md to the release notes into the release notes. Add any other information pertaining the release itself that are not included in the CHANGELOG.md, although prefer to include those in the CHANGELOG.md file. You can switch to the Preview tab to see the results.

  • Finally click "Publish release" and go celebrate with the team. 🎉

  • The branch v0.7.x will be pushed to gitlab automatically, but make sure to manually push the tag of the release also to https://gitlab.com/wg1/jpeg-xl, by doing

git push gitlab v0.7.1

where gitlab is the remote [email protected]:wg1/jpeg-xl.git.

How to build downstream projects

docker run -it debian:bookworm /bin/bash

apt update
apt install -y clang cmake git libbrotli-dev nasm pkg-config ninja-build
export CC=clang
export CXX=clang++

mkdir -p /src
cd /src

git clone --recurse-submodules --depth 1 -b v0.9.x \
  https://github.com/libjxl/libjxl.git
git clone --recurse-submodules --depth 1 \
  https://github.com/ImageMagick/ImageMagick.git
git clone --recurse-submodules --depth 1 \
  https://github.com/FFmpeg/FFmpeg.git

cd /src/libjxl
cmake -B build -G Ninja .
cmake --build build -j`nproc`
cmake --install build --prefix="/usr"

cd /src/ImageMagick
./configure --with-jxl=yes
# check for "JPEG XL --with-jxl=yes yes"
make -j `nproc`
./utilities/magick -version

cd /src/FFmpeg
./configure --disable-all --disable-debug --enable-avcodec --enable-avfilter \
  --enable-avformat --enable-libjxl --enable-encoder=libjxl \
  --enable-decoder=libjxl --enable-ffmpeg
# check for libjxl decoder/encoder support
make -j `nproc`
ldd ./ffmpeg
./ffmpeg -version