Server: buildx for server images

[redrathnure]

Switching to Docker BuildX toolset. It's faster, have more benefits for development process (e.g. caching mount points) and open possibility to build multi platform images.

[laurent22]

Do you know how difficult it would be to add ARM64 support if this pull request is merged? It seems to be particularly challenging to add since several people (including myself) unsuccessfully tried.

[redrathnure]

Most likely one line to Docker descriptor and a line to place from were you trigger image building.
Will try to do at least first part today.

[redrathnure]

Oh, BTW I just realized that I have not checked your GH actions. Perhaps we should adjust this workflow and think about proper builder for arm (sorry I knows nothing about your workers infrastructure).

[laurent22]

Indeed any new dependencies should be added there, but I guess the check would tell us that - if it's still passing it's probably that the new dependency is already included as part of something else, and so we don't need to change anything.

[redrathnure]

OK, server related checks - pass. Means we built at least linux\amd64 image:)

However there are few more questions/ideas around this topic:

1. Do you have arm64 based workers? If yes, it's possible to configure ServerDockerImage like step but for amr64 images.
2. If you have only intel based worker but still would like to build/check arm64 server images, it's possible to run docker together with QEMU. It works but extremely slow. For me amd64 image takes 5-10 minutes, but arm64 with emulation it's more than 1hour.
3. And I have no idea how you publish server images. If you build it manually (not via GH actions): please try a new cli arg --platform linux/amd64,platform/arm64 (build multiform image(s) without publishing). And now you may exec just yarn buildServerDocker to check all available args with human friendly help :)

[laurent22]

Do you have arm64 based workers? If yes, it's possible to configure ServerDockerImage like step but for amr64 images.

Yes, probably, the image is built on GitHub Action

If you have only intel based worker but still would like to build/check arm64 server images, it's possible to run docker together with QEMU. It works but extremely slow. For me amd64 image takes 5-10 minutes, but arm64 with emulation it's more than 1hour.

Same, "we" can have everything as long as it works on GitHub Action and most things work.

The image is built from GitHub Action by running this script:

joplin/.github/scripts/run_ci.sh

Line 280 in c6956df

yarn buildServerDocker --tag-name $GIT_TAG_NAME --push-images --repository $SERVER_REPOSITORY

which calls buildServerDockerImage and the image is then pushed here:

joplin/packages/tools/buildServerDocker.ts

Line 70 in c6956df

if (pushImages) await execCommand(`docker push ${repository}:${tag}`);

But I guess as long as the way the script is called doesn't change it should work the same?

[redrathnure]

But I guess as long as the way the script is called doesn't change it should work the same?

Yes, current behavior should be the same as previously.

If I got it right, easiest way is to install QEMU and build amr64 part using it (need to specify --platform linux/amd64,linux/arm64 too).

[laurent22]

I thought GitHub supported ARM64 but apparently that's very recent: https://discourse.joplinapp.org/t/github-actions-now-provides-arm64-for-free/43173

Does that make a difference for your pull request? Maybe we don't need QEMU anymore?

[redrathnure]

Probably. Arm native builder should be way faster then emulation. In any case switching to BuildX (this PR) and building multiplatform images arr pretty non related steps and should be done in separated PRs.