Add docker deploy instructions in dev.md and multi-platform build (#…

…753)

.dockerignore

@@ -0,0 +1,13 @@
+.git/
+.github/
+.ruff_cache/
+.vscode/
+.mypy_cache/
+.pytest_cache/
+.venv/
+__pycache__/
+web/
+lilac/
+notebooks/
+docs/
+scripts/

.gcloudignore

@@ -0,0 +1,2 @@
+#!include:.gitignore
+#!include:.dockerignore

README.md

@@ -70,11 +70,8 @@ explore them.
 
 ### Run via Docker
 
-Build the image after cloning the repo:
-
-```sh
-docker build -t lilac .
-```
+We publish images for `linux/amd64` and `linux/arm64` on Docker Hub under
+[lilacai](https://hub.docker.com/u/lilacai).
 
 The container runs on the virtual port `8000`, this command maps it to the host machine port `5432`.
 
@@ -86,7 +83,14 @@ docker run -it \
   -p 5432:8000 \
   --volume /host/path/to/data:/data \
   -e LILAC_PROJECT_DIR="/data" \
-  lilac
+  --gpus all \ # Remove if you don't have a GPU, or on MacOS.
+  lilacai/lilac
+```
+
+To build your own custom image run the following command, otherwise skip to the next step.
+
+```sh
+docker build -t lilac .
 ```
 
 ### 📊 Load data

cloudbuild.yml

@@ -0,0 +1,36 @@
+steps:
+  - name: 'gcr.io/cloud-builders/docker'
+    entrypoint: 'bash'
+    args: ['-c', 'docker login --username=$_HUB_REPO --password=$$PASSWORD']
+    secretEnv: ['PASSWORD']
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - buildx
+      - create
+      - '--name'
+      - buildxbuilder
+      - '--use'
+      - '--bootstrap'
+    id: create-builder
+  - name: 'gcr.io/cloud-builders/docker'
+    entrypoint: 'bash'
+    args: ['-c', 'docker pull docker.io/$_HUB_REPO/lilac:latest || exit 0']
+  - name: 'gcr.io/cloud-builders/docker'
+    entrypoint: 'bash'
+    args:
+      [
+        '-c',
+        'docker buildx build --platform $_DOCKER_BUILDX_PLATFORMS -t $_HUB_REPO/lilac:latest -t
+        $_HUB_REPO/lilac:$_VERSION --cache-from docker.io/$_HUB_REPO/lilac:latest . --push'
+      ]
+options:
+  env:
+    - DOCKER_CLI_EXPERIMENTAL=enabled
+substitutions:
+  _PROJECT_ID: 'lilac-386213'
+  _DOCKER_BUILDX_PLATFORMS: 'linux/amd64,linux/arm64'
+  _HUB_REPO: 'lilacai'
+availableSecrets:
+  secretManager:
+    - versionName: projects/$_PROJECT_ID/secrets/DOCKER_PASSWORD_SECRET_NAME/versions/1
+      env: 'PASSWORD'

development.md

@@ -68,18 +68,38 @@ Details can be found at
    # --hf_username and --hf_space are optional and can override the ENV for local uploading.
    ```
 
-#### Deployment
+#### Publishing docker images
 
-To build the docker image:
+All docker images are published under the [lilacai](https://hub.docker.com/u/lilacai) account on
+Docker Hub. We build docker images for two platforms `linux/amd64` and `linux/arm64`.
+
+> NOTE: `./scripts/publish_pip.sh` will do this automatically.
+
+**Building on Google Cloud**
+
+```sh
+gcloud builds submit \
+  --config cloudbuild.yml \
+  --substitutions=_VERSION=$(poetry version -s) \
+  --async .
+```
+
+**Building locally**
+
+To build the image for both platforms, as a one time setup do:
 
 ```sh
-./scripts/build_docker.sh
+docker buildx create --name mybuilder --node mybuilder0 --bootstrap --use
 ```
 
-To run the docker image locally:
+Make sure you have Docker Desktop running and you are logged as the lilacai account. To build and
+push the image:
 
 ```sh
-docker run -p 5432:5432 lilac_blueprint
+docker buildx build --platform linux/amd64,linux/arm64 \
+  -t lilacai/lilac \
+  -t lilacai/lilac:$(poetry version -s) \
+  --push .
 ```
 
 #### Authentication

scripts/publish_pip.sh

@@ -25,6 +25,9 @@ poetry version --dry-run $VERSION_TYPE
 # Make sure the user is logged into github and has `gh` installed.
 gh auth status || exit 1
 
+# Make sure the user is logged in and can see the cloud builds.
+gcloud builds list --project lilac-386213 || exit 1
+
 set -o allexport
 source .env.local
 set +o allexport
@@ -94,3 +97,9 @@ gh release create "$TAG_ID" ./dist/*.whl \
   --latest \
   --title "$TAG_ID" \
   --verify-tag
+
+# Building the docker images.
+gcloud builds submit \
+  --config cloudbuild.yml \
+  --substitutions=_VERSION=$(poetry version -s) \
+  --async .