[CI] Pulsar CI will stop working after November 30th, 2024 unless custom uploading actions are updated

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

GitHub is deprecating existing v1, v2, and v3 upload actions. https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/ explains more.

[Deprecation notice: v1, v2, and v3 of the artifact actions]
...
Please update your workflow to use v4 of the artifact actions.
Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

Solution

Pulsar CI uses a custom "GitHub Actions Artifact client" . Instead of using the actual action, it uses the GitHub provided javascript library to create a custom client that can be used in shell scripts to upload and download any files to be cached during a workflow run. This is used in Pulsar CI to build the docker images once so that they can be used in the integration test and system test jobs. It's also used to share the binary build results. This action will need to be updated to use the v4 client libraries.

Here's how the docker image gets uploaded:

pulsar/.github/workflows/pulsar-ci.yaml

Lines 531 to 534 in 590e133

	- name: save docker image apachepulsar/java-test-image:latest to Github artifact cache
	if: ${{ matrix.platform == 'linux/amd64' }}
	run: |
	$GITHUB_WORKSPACE/build/pulsar_ci_tool.sh docker_save_image_to_github_actions_artifacts apachepulsar/java-test-image:latest pulsar-java-test-image

And restored:

pulsar/.github/workflows/pulsar-ci.yaml

Lines 647 to 649 in 590e133

	- name: Load docker image apachepulsar/java-test-image:latest from Github artifact cache
	run: |
	$GITHUB_WORKSPACE/build/pulsar_ci_tool.sh docker_load_image_from_github_actions_artifacts pulsar-java-test-image

This is how the underlying shell script uses the custom gh-actions-artifact-client:

pulsar/build/pulsar_ci_tool.sh

Lines 82 to 101 in 590e133

	# saves a given image (1st parameter) to the GitHub Actions Artifacts with the given name (2nd parameter)
	function ci_docker_save_image_to_github_actions_artifacts() {
	local image=$1
	local artifactname="${2}.zst"
	ci_install_tool pv
	echo "::group::Saving docker image ${image} with name ${artifactname} in GitHub Actions Artifacts"
	# delete possible previous artifact that might exist when re-running
	timeout 1m gh-actions-artifact-client.js delete "${artifactname}" &>/dev/null || true
	docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | timeout 20m gh-actions-artifact-client.js upload --retentionDays=$ARTIFACT_RETENTION_DAYS "${artifactname}"
	echo "::endgroup::"
	}
	# loads a docker image from the GitHub Actions Artifacts with the given name (1st parameter)
	function ci_docker_load_image_from_github_actions_artifacts() {
	local artifactname="${1}.zst"
	ci_install_tool pv
	echo "::group::Loading docker image from name ${artifactname} in GitHub Actions Artifacts"
	timeout 20m gh-actions-artifact-client.js download "${artifactname}" | pv -batf -i 5 | unzstd | docker load
	echo "::endgroup::"
	}

Alternatives

Anything else?

It's simply about updating the custom client that is written in Javascript. The model in the new v4 is slightly different, but it is possible to find a way which works for the new backend.

Are you willing to submit a PR?

• I'm willing to submit a PR!