Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(workflows) add GitHub Actions workflow for preview builds #2446

Merged
merged 32 commits into from
Sep 20, 2024
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
target/
.git/
.github/
.idea/
.vscode/
.devcontainer/
.cargo/
artifacts/
45 changes: 45 additions & 0 deletions .github/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
FROM rust:slim as chef

# Install libclang and other necessary tools
RUN apt-get update && \
apt-get install -y clang llvm-dev libclang-dev git libtool automake autoconf make curl protobuf-compiler && \
rm -rf /var/lib/apt/lists/*

# Verify and set LIBCLANG_PATH environment variable
RUN find /usr -name "libclang.so*" && \
export LIBCLANG_PATH=$(find /usr -name "libclang.so*" | head -n 1 | xargs dirname)

COPY rust-toolchain.toml .
RUN rustup install $(cat rust-toolchain.toml | grep channel | cut -d' ' -f3 | tr -d '"')
RUN rustup component add cargo clippy rust-docs rust-std rustc rustfmt

RUN cargo install cargo-chef

FROM chef AS planner

WORKDIR /app
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder

WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json
RUN cargo build --release --bins # pre-cache some stuff

# Build application
COPY . .
ENV PATH="/root/.cargo/bin:${PATH}"

RUN cargo build --release --bins
steebchen marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +24 to +36
Copy link

@coderabbitai coderabbitai bot Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sensei, there are a few things that can be improved in the builder stage.

The pre-caching step on line 30 is redundant since the same command is already executed on line 36. Consider removing the redundant command to simplify the Dockerfile and reduce build time.

Also, copying the application code again on line 33 is unnecessary since it was already copied in the planner stage. Setting the PATH environment variable on line 34 is also unnecessary since it is not used in the final image.

Here's how you can refactor it:

FROM chef AS builder

WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json

RUN cargo build --release --bins

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this still caches building the deps, so it certainly speeds up the build when dependencies don't change

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steebchen Ohayo, sensei! Thank you for the clarification. I understand now that the pre-caching step helps speed up the build when dependencies don't change. I'll keep this in mind for future reviews.

(・ェ・)


Learnings added
Learnt from: steebchen
PR: dojoengine/dojo#2446
File: .github/Dockerfile:24-36
Timestamp: 2024-09-19T19:02:02.095Z
Learning: In the Dockerfile, the pre-caching step in the `builder` stage is important for caching dependencies and speeding up the build when dependencies don't change.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


FROM rust:1-alpine

WORKDIR /

COPY --from=builder /app/target/release/katana /app/artifacts/
COPY --from=builder /app/target/release/torii /app/artifacts/
COPY --from=builder /app/target/release/sozo /app/artifacts/
COPY --from=builder /app/target/release/saya /app/artifacts/
glihm marked this conversation as resolved.
Show resolved Hide resolved
89 changes: 89 additions & 0 deletions .github/workflows/preview.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: preview

on:
workflow_dispatch:
push:
branches:
- ci-preview
glihm marked this conversation as resolved.
Show resolved Hide resolved

jobs:
publish:
runs-on: ubuntu-latest-32-cores

steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Cache Docker layers
uses: actions/cache@v4
with:
path: /tmp/.buildx-cache/prebuild
key: ${{ runner.os }}-buildx-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-${{ github.ref_name }}
${{ runner.os }}-buildx-

- name: Cache Docker layers
uses: actions/cache@v4
with:
path: /tmp/.buildx-cache/build
key: ${{ runner.os }}-buildx-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-${{ github.ref_name }}
${{ runner.os }}-buildx-

- name: Checkout repository
uses: actions/checkout@v4

- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Set outputs
id: vars
run: |
set -eux
git config --global --add safe.directory "${{ github.workspace }}"
echo "sha_short=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT"

- name: Build Docker image
uses: docker/build-push-action@v5
with:
context: .
file: .github/Dockerfile
load: true
tags: build-local:latest
cache-from: type=local,src=/tmp/.buildx-cache/prebuild
cache-to: type=local,dest=/tmp/.buildx-cache-new/prebuild,mode=max
platforms: linux/amd64

- name: Build local binaries
run: |
set -eux
docker run --rm -v "$(pwd)/artifacts/$PLATFORM:/artifacts" build-local:latest /bin/sh -c "cp -r /app/artifacts/* /artifacts/"
env:
PLATFORM: linux/amd64

- name: Build and push docker image
uses: docker/build-push-action@v3
with:
push: true
tags: ghcr.io/${{ github.repository }}:preview--${{ steps.vars.outputs.sha_short }}
cache-from: type=local,src=/tmp/.buildx-cache/build
cache-to: type=local,dest=/tmp/.buildx-cache-new/build,mode=max
platforms: linux/amd64
build-contexts: |
artifacts=artifacts

- # Temp fix
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
name: Move cache
run: |
rm -rf /tmp/.buildx-cache/prebuild
mv /tmp/.buildx-cache-new/prebuild /tmp/.buildx-cache/prebuild
rm -rf /tmp/.buildx-cache/build
mv /tmp/.buildx-cache-new/build /tmp/.buildx-cache/build
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,5 @@ justfile
spawn-and-move-db
types-test-db
examples/spawn-and-move/manifests/saya/**

artifacts/
Loading