-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release TerriaMap using create-docker-context #681
Draft
steve9164
wants to merge
15
commits into
main
Choose a base branch
from
docker-context
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
bdf304f
Flatten terria helm chart
steve9164 6afaf94
Add metadata option supporting docker/metadata-action tags and labels
steve9164 5c1f0e0
Update release process to use create-docker-context for smaller image…
steve9164 98c0b9d
Use env var for metadata json
steve9164 09e528b
Use slim node image
steve9164 9279cee
Use annotations instead of labels and stop passing json as an argument
steve9164 b25a1c3
Bump version for release
steve9164 f975d57
Correct annotations handling
steve9164 bb02963
Bump version for release
steve9164 ce2b318
Try annotations on index as well to see if those are picked up by Git…
steve9164 a7da33b
Bump version for release
steve9164 b49e225
Update ADRs and related documentation
steve9164 3641a36
Change path and run as user node
steve9164 3123a7a
Merge remote-tracking branch 'origin/flatten-chart' into docker-context
steve9164 b64cdd2
Improve wording and fix typos
steve9164 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
# 2. Docker multi-architecture build using create-docker-context | ||
|
||
Date: 2024-07-18 | ||
|
||
## Status | ||
|
||
Proposed | ||
|
||
## Context | ||
|
||
I (crispy) consider a multi-stage dockerfile to be the gold standard of | ||
reproducible, mutli-architecture builds. Something like the following: | ||
|
||
- Build container copies workspace, installs development dependencies and builds | ||
the app. | ||
- Production container copies build artifacts and installs only production | ||
dependencies. | ||
|
||
is ideal. This ensures only production dependencies are present, and you can run | ||
this process on every architecture to create a multi-arch docker image. Binaries | ||
downloaded during dependency installation will fetch the correct architecture | ||
binary as depencies are installed separately on each architecture. But | ||
installing dependencies and building JS is extremely slow on emulated | ||
architectures, such as docker buildx on GitHub Actions (this can take 2.5 hours | ||
to build the image). | ||
|
||
If instead we can (on the build machine/VM): | ||
|
||
1. install all dependencies | ||
2. build the app | ||
3. copy build artifacts and only production dependencies to the multi-arch | ||
docker image | ||
|
||
then **as long as production dependencies are portable**, we have a working | ||
multi-arch docker image with very little computation being run on emulated | ||
architectures. The `create-docker-context.js` script allows us to do this, | ||
copying build artifacts and only production dependencies to an intermediate | ||
"context" folder which is then used to create the image. | ||
|
||
Currently none of our production dependencies install non-portable binaries. | ||
|
||
## Decision | ||
|
||
While all production dependencies remain portable, we will build | ||
multi-architecture docker images by building TerriaMap on the VM and copying | ||
only production-necessary files and dependencies to the final docker image. | ||
|
||
## Consequences | ||
|
||
- We will replace current GitHub Actions release process with one using | ||
`create-docker-context.js`. | ||
- Our GitHub Actions TerriaMap release time will reduce from 2.5 hours to less | ||
than 10 minutes. | ||
- If in future TerriaMap uses a binary installed by side effect during JS | ||
dependency installation and this binary cannot be run on an architecture for | ||
which an image is created, that image will fail when run on that architecture. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,12 @@ | ||
# Docker image for the primary terria map application server | ||
FROM node:16 | ||
# Intended for use only with a "context" directory created by create-docker-context.js | ||
FROM node:16-slim | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This Dockerfile runs the node process as |
||
|
||
RUN mkdir -p /usr/src/app && mkdir -p /etc/config/client | ||
WORKDIR /usr/src/app/component | ||
COPY . /usr/src/app | ||
RUN mkdir -p /etc/config/client | ||
|
||
USER node | ||
WORKDIR /usr/src/app | ||
COPY --chown=node:node component /usr/src/app | ||
|
||
EXPOSE 3001 | ||
ENV NODE_ENV=production | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,16 @@ | ||
#!/usr/bin/env node | ||
|
||
// MAJOR ASSSUMPTION: build artifacts and node_modules content for all production | ||
// dependencies is cross-platform, or care is taken to only install dependencies, | ||
// run create-docker-context.js and create docker images on a compatible platform | ||
// See architecture/0002-docker-multi-arch-build.md | ||
|
||
// Based off @magda/[email protected] create-docker-context-for-node-component | ||
// Changes made: | ||
// - The Dockerfile path is configurable in package.json | ||
// - The Dockerfile path is configurable in package.json (I don't want a dockerfile | ||
// intended to be used only through a script to be in the top level directory) | ||
// - Can parse metadata from GitHub Action docker/metadata-action@v5 and add this | ||
// to the created image | ||
|
||
const childProcess = require("child_process"); | ||
const fse = require("fs-extra"); | ||
|
@@ -42,7 +50,7 @@ const argv = yargs | |
}, | ||
name: { | ||
description: | ||
"The package name to use in auto tag generation. Will default to ''. Used to override the docker nanme config in package.json during the auto tagging. Requires --tag=auto", | ||
"The package name to use in auto tag generation. Will default to ''. Used to override the docker name config in package.json during the auto tagging. Requires --tag=auto", | ||
type: "string", | ||
default: process.env.MAGDA_DOCKER_NAME | ||
}, | ||
|
@@ -86,6 +94,12 @@ const argv = yargs | |
description: | ||
"Version to cache from when building, using the --cache-from field in docker. Will use the same repository and name. Using this options causes the image to be pulled before build.", | ||
type: "string" | ||
}, | ||
metadata: { | ||
description: | ||
"Use tags and annotations from https://github.com/docker/metadata-action v5. Utilises env.DOCKER_METADATA_OUTPUT_JSON. Overrides --tag", | ||
type: "boolean", | ||
default: false | ||
} | ||
}) | ||
.help().argv; | ||
|
@@ -166,16 +180,22 @@ if (argv.build) { | |
} | ||
); | ||
|
||
const tags = getTags( | ||
argv.tag, | ||
argv.local, | ||
argv.repository, | ||
argv.version, | ||
argv.name | ||
); | ||
const tagArgs = tags | ||
.map((tag) => ["-t", tag]) | ||
.reduce((soFar, tagArgs) => soFar.concat(tagArgs), []); | ||
// metadata json from GitHub Action docker/metadata-action@v5 | ||
const metadata = | ||
argv.metadata && env.DOCKER_METADATA_OUTPUT_JSON | ||
? JSON.parse(env.DOCKER_METADATA_OUTPUT_JSON) | ||
: undefined; | ||
|
||
const tags = metadata | ||
? metadata.tags | ||
: getTags(argv.tag, argv.local, argv.repository, argv.version, argv.name); | ||
const tagArgs = tags.flatMap((tag) => ["-t", tag]); | ||
|
||
const annotationArgs = | ||
metadata?.annotations?.flatMap((annotation) => [ | ||
"--annotation", | ||
annotation | ||
]) ?? []; | ||
|
||
const cacheFromArgs = cacheFromImage ? ["--cache-from", cacheFromImage] : []; | ||
|
||
|
@@ -186,6 +206,7 @@ if (argv.build) { | |
...(argv.platform ? ["buildx"] : []), | ||
"build", | ||
...tagArgs, | ||
...annotationArgs, | ||
...cacheFromArgs, | ||
...(argv.noCache ? ["--no-cache"] : []), | ||
...(argv.platform ? ["--platform", argv.platform, "--push"] : []), | ||
|
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
1 change: 1 addition & 0 deletions
1
deploy/helm/terria/.helmignore → deploy/helm/terriamap/.helmignore
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
apiVersion: v1 | ||
description: A Helm chart for Kubernetes | ||
name: terriamap | ||
version: 0.1.2 | ||
home: https://github.com/TerriaJS/terriamap | ||
sources: ["https://github.com/TerriaJS/terriamap"] |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
12 changes: 10 additions & 2 deletions
12
.../helm/terria/charts/terriamap/values.yaml → deploy/helm/terriamap/values.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is a big deal, Github provides ARM runners now so if this becomes a problem in the future it's "easy" to rectify.