Split docker (#392)

* Initial docker container split.

Rework Dockerfiles.

Make Gradle server as optional. Cleanup imports.

Read template file from the jar.
Cleanup unnecessary annotations.
Add dockerfile for base application.

Import cleanup.

Add profiles for different services in docker compose file.
Add services for integration testing.
Modify tests to run with current dev engine version (fix it later).
Update scripts to start/stop test server (rework to run docker compose).
Cleanup build.gradle a bit (remote all docker related parts).
Add service definitions to run tests locally in docker-compose.yml.

[ci skip] Remove application dockerfile.
Pass DM_PACKAGES_URL as secret during docker build.
Remove python2.7 from base image.

Add workflow to build docker imaes with environment.

[ci skip] Add workflow for jar release.
Adopt changes in Dockerfile naming.
[WIP] Docker compose rework.
Pass jar's version as gradle properties.
Add publishing tasks.

Add micrometrics Stackdriver adapter.

Prepare CI for testing.

Fix pass secret env to docker build.

Fix maven auth.

Rework docker compose file.
Add mout directories with jars.
Add copying jars to /app directories to further mounting.
Update .gitignore.
Fix extender's tests.
Fix remote builder mappings.

User Path context instead of Git context when building Docker image.

Fix tag pattern for manifest merge tool.

Add missed registry address.

Remove unnessary maven repo ref.
Add disk cleanup action for Extender build.

Fix gradle task for Extender build.

Update registries names. Update service accounts according to new project.

Pin exact image versions.

Fix andorid docker image.

Update registries urls.

Fix encoding in base Docker image.
Update IntegrationTest.
Get rid of docker warning about ENV.

Update dependencies. (#394)

Upgrade spring boot to latest 2.x version.
Update to Spring boot 3.x. Update java dependencies to more fresh versions.
Remove InfluxDB dependencies and related code.
Fix warning about several common-logging jars in classpath.

Update java distro to 17.

Exclude dependencies that not necessary for manifestmergetool.

Fix tests after upgrade.

Fix metrics units.

Update new server documentation.

* Fix instance network aliases in docker compose.

* Add .Net package to base image. Cleanup CSBuilder.

* Update sdk version for AuthTest.

* Add dependabot configuration.

* Add standalone jetifier. Run jetifier when resolve gradle dependencies.

* Rework extender's metrics. Replace Stackdriver with Influx. (#404)

* Rework extender's metrics. Replace Stackdriver with Influx.

* Made metrics connected to operation duration Timer instaed of Gauge.

* Remake some metrics as distribution summary instead of gauge. Make gauge metric about gradle cache size works.

* Add configuration for accessing gcp bucket as Spring boot resource. (#406)

* Add configuration for accessing gcp bucket as Spring boot resource.

* Add GCPDataCache implementation. (#402)

* Add GCPDataCache implementation.
Remove S3DataCache implementation.

* Add prefix configuration for gcp cache bucket.

* Temporary disable cache test.
Add metrics services in docker-compose.yml.
Add influx configuration example.

* Health report (#407)

* Add health report endpoint. Add HealthReporterService.

* Add status page.

* Fix CrossOrigin settigns for health report endpoint. Fix status page.

* Remove excessive cache operations.

* Add connection timeout in HealthReporterService. Fix Nan in status page.

* Move Artifact registry to other region.

* Change base image for console's environment.

* Add GCPInstanceService which used to monitor and suspend gcp instances if instance not used for a while. (#410)

* Add GCPInstanceService which used to monitor and suspend gcp instances if instance not used for a while.

* Add option to exclude instance from resume/suspend control. Fix initialization issues.

* Improve logging.

* Add thread count limit in GCPInstanceService.
Add maintance endpoint to touch all VMs before deployment.

* Fix typos.

* Revert "Add standalone jetifier. Run jetifier when resolve gradle dependencies."

This reverts commit f4a4741.

* Fix download cache size metric value.
Unify information about download/upload caches.
Add metrics to track download/upload cache file count.

* Fix merge issue.

* Update consoles sdks.

* Improve health reporting. Now is instance is suspended it's marked as Operational.
Remove status page because it moved to separate repo.

* Add exception handling for cache upload operation.

.github/dependabot.yml

@@ -0,0 +1,19 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+  - package-ecosystem: "gradle"
+    directories:
+      - "/"
+      - "/client"
+      - "/server"
+      - "/server/manifestmergetool"
+    schedule:
+      interval: "weekly"

.github/workflows/application-build.yml

@@ -0,0 +1,213 @@
+name: Build jars
+
+on:
+  workflow_dispatch:
+  push:
+    tags:
+      - "extender-v[0-9]+.[0-9]+.[0-9]+"
+      - "client-v[0-9]+.[0-9]+.[0-9]+"
+      - "manifestmergetool-v[0-9]+.[0-9]+.[0-9]+"
+
+env:
+  GOOGLE_ARTIFACT_REGISTRY: europe-west1-docker.pkg.dev
+  MAVEN_SERVICE_ACCOUNT: [email protected]
+
+jobs:
+  client:
+    if: ${{ startsWith(github.ref_name, 'client-') }}
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout repo
+      uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332
+    - name: Generate version
+      id: version
+      run: |
+          SHORT_SHA="$(echo ${GITHUB_SHA} | cut -c1-8)"
+          TAG_NAME=${{ github.ref_name }}
+          TAG_VERSION="${TAG_NAME#*-v}"
+
+          echo "tag_version=$TAG_VERSION" >> $GITHUB_OUTPUT
+    - name: Login to Google Artifact registry (Maven)
+      uses: google-github-actions/auth@71fee32a0bb7e97b4d33d548e7d957010649d8fa
+      with:
+        service_account: ${{ env.MAVEN_SERVICE_ACCOUNT }}
+        credentials_json: ${{ secrets.MAVEN_REGISTRY_JSON_KEY }}
+        export_environment_variables: true
+        create_credentials_file: true
+    - name: Install Java
+      uses: actions/setup-java@99b8673ff64fbf99d8d325f52d9a5bdedb8483e9
+      with:
+        java-version: '17'
+        distribution: 'temurin'
+    - name: Setup Gradle
+      uses: gradle/actions/setup-gradle@31ae3562f68c96d481c31bc1a8a55cc1be162f83
+    - name: Build the extender-client.jar
+      run: ./gradlew -PclientVersion=${{ steps.version.outputs.tag_version }} client:jar
+    - name: Test the extender-client.jar
+      run: ./gradlew -PclientVersion=${{ steps.version.outputs.tag_version }} client:test
+    - name: Publish to Google artifact registry
+      run: ./gradlew -PclientVersion=${{ steps.version.outputs.tag_version }} publishExtenderClientPublicationToExtenderRepoRepository
+    - name: Prepare jar for GH release
+      run: |
+        mkdir -p ${{ github.workspace }}/gh_release
+        cp ${{ github.workspace }}/client/build/libs/extender-client-${{ steps.version.outputs.tag_version }}.jar \
+          ${{ github.workspace }}/gh_release/extender-client-${{ steps.version.outputs.tag_version }}.jar
+    - name: Publish to Github release
+      run: |
+        chmod 0444 ${{ github.workspace }}/gh_release/*.jar
+          echo ${{ github.token }} | gh auth login --with-token
+          gh release create "${GITHUB_REF_NAME}"  \
+            --draft \
+            --generate-notes \
+            --target ${{ github.sha }} \
+            ${{ github.workspace }}/gh_release/*.jar
+    - name: Notify if tests failed
+      uses: homoluctus/slatify@cd4b4a1158cfb3e26fe1ee35c1cd4f0247dfbf96
+      if: failure()
+      with:
+        type: ${{ job.status }}
+        job_name: 'Extender cleint release'
+        channel: '#defold-alarms-build'
+        url: ${{ secrets.SLACK_WEBHOOK }}
+
+  manifestmerger:
+    if: ${{ startsWith(github.ref_name, 'manifestmergetool-') }}
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout repo
+      uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332
+    - name: Generate version
+      id: version
+      run: |
+          SHORT_SHA="$(echo ${GITHUB_SHA} | cut -c1-8)"
+          TAG_NAME=${{ github.ref_name }}
+          TAG_VERSION="${TAG_NAME#*-v}"
+
+          echo "tag_version=$TAG_VERSION" >> $GITHUB_OUTPUT
+    - name: Login to Google Artifact registry (Maven)
+      uses: google-github-actions/auth@71fee32a0bb7e97b4d33d548e7d957010649d8fa
+      with:
+        service_account: ${{ env.MAVEN_SERVICE_ACCOUNT }}
+        credentials_json: ${{ secrets.MAVEN_REGISTRY_JSON_KEY }}
+        export_environment_variables: true
+        create_credentials_file: true
+    - name: Install Java
+      uses: actions/setup-java@99b8673ff64fbf99d8d325f52d9a5bdedb8483e9
+      with:
+        java-version: '17'
+        distribution: 'temurin'
+    - name: Setup Gradle
+      uses: gradle/actions/setup-gradle@31ae3562f68c96d481c31bc1a8a55cc1be162f83
+    - name: Build manifestmerger
+      run: ./gradlew -PmanifestMergetoolVersion=${{ steps.version.outputs.tag_version }} :manifestmergetool:mainJar
+    - name: Test manifestmerger
+      run: ./gradlew -PmanifestMergetoolVersion=${{ steps.version.outputs.tag_version }} :manifestmergetool:test
+    - name: Publish to Google artifact registry
+      run: ./gradlew -PmanifestMergetoolVersion=${{ steps.version.outputs.tag_version }} publishManifestMergeToolPublicationToExtenderRepoRepository
+    - name: Prepare jar for GH release
+      run: |
+        mkdir -p ${{ github.workspace }}/gh_release
+        cp ${{ github.workspace }}/server/manifestmergetool/build/libs/manifestmergetool-${{ steps.version.outputs.tag_version }}.jar \
+          ${{ github.workspace }}/gh_release/manifestmergetool-${{ steps.version.outputs.tag_version }}.jar
+    - name: Publish to Github release
+      run: |
+        chmod 0444 ${{ github.workspace }}/gh_release/*.jar
+          echo ${{ github.token }} | gh auth login --with-token
+          gh release create "${GITHUB_REF_NAME}"  \
+            --draft \
+            --generate-notes \
+            --target ${{ github.sha }} \
+            ${{ github.workspace }}/gh_release/*.jar
+    - name: Notify if tests failed
+      uses: homoluctus/slatify@cd4b4a1158cfb3e26fe1ee35c1cd4f0247dfbf96
+      if: failure()
+      with:
+        type: ${{ job.status }}
+        job_name: 'Manifest merge tool release'
+        channel: '#defold-alarms-build'
+        url: ${{ secrets.SLACK_WEBHOOK }}
+
+  extender:
+    if: ${{ startsWith(github.ref_name, 'extender-') }}
+    runs-on: ubuntu-latest
+    steps:
+    # cleanup disk space because we use lots of disk space for Docker image during extender tests
+    - name: Free Disk Space (Ubuntu)
+      uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be
+      with:
+        # this might remove tools that are actually needed,
+        # if set to "true" but frees about 6 GB
+        tool-cache: false
+
+        # all of these default to true, but feel free to set to
+        # "false" if necessary for your workflow
+        android: true
+        dotnet: true
+        haskell: true
+        large-packages: true
+        docker-images: true
+        swap-storage: true
+    - name: Checkout repo
+      uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332
+    - name: Generate version
+      id: version
+      run: |
+          SHORT_SHA="$(echo ${GITHUB_SHA} | cut -c1-8)"
+          TAG_NAME=${{ github.ref_name }}
+          TAG_VERSION="${TAG_NAME#*-v}"
+
+          echo "tag_version=$TAG_VERSION" >> $GITHUB_OUTPUT
+    - name: Install Java
+      uses: actions/setup-java@99b8673ff64fbf99d8d325f52d9a5bdedb8483e9
+      with:
+        java-version: '17'
+        distribution: 'temurin'
+    - name: Setup Gradle
+      uses: gradle/actions/setup-gradle@31ae3562f68c96d481c31bc1a8a55cc1be162f83
+    - name: Build extender
+      run: ./gradlew -PextenderVersion=${{ steps.version.outputs.tag_version }} server:bootJar
+    - name: Login to Google Artifact registry (Docker)
+      uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446
+      with:
+        registry: ${{ env.GOOGLE_ARTIFACT_REGISTRY }}
+        username: _json_key
+        password: ${{ secrets.DOCKER_REGISTRY_JSON_KEY }}
+    - name: Pull necessary docker images
+      run: |
+        # pull all containers under profile 'test'. Part of them is used in 'auth-test'
+        # if you need some new container under other profiles - need to add new statement with different profile
+        docker compose --file ./server/docker/docker-compose.yml --profile test pull
+    - name: Run extender tests
+      run: ./gradlew -PextenderVersion=${{ steps.version.outputs.tag_version }} server:test -i
+    - name: Login to Google Artifact registry (Maven)
+      uses: google-github-actions/auth@71fee32a0bb7e97b4d33d548e7d957010649d8fa
+      with:
+        service_account: ${{ env.MAVEN_SERVICE_ACCOUNT }}
+        credentials_json: ${{ secrets.MAVEN_REGISTRY_JSON_KEY }}
+        export_environment_variables: true
+        create_credentials_file: true
+    - name: Publish to Google artifact registry
+      run: ./gradlew -PextenderVersion=${{ steps.version.outputs.tag_version }} publishExtenderPublicationToExtenderRepoRepository
+    - name: Prepare jar for GH release
+      run: |
+        mkdir -p ${{ github.workspace }}/gh_release
+        cp ${{ github.workspace }}/server/build/libs/extender-${{ steps.version.outputs.tag_version }}.jar \
+          ${{ github.workspace }}/gh_release/extender-${{ steps.version.outputs.tag_version }}.jar
+    - name: Publish to Github release
+      run: |
+        chmod 0444 ${{ github.workspace }}/gh_release/*.jar
+          echo ${{ github.token }} | gh auth login --with-token
+          gh release create "${GITHUB_REF_NAME}"  \
+            --draft \
+            --generate-notes \
+            --target ${{ github.sha }} \
+            ${{ github.workspace }}/gh_release/*.jar
+    - name: Notify if tests failed
+      uses: homoluctus/slatify@cd4b4a1158cfb3e26fe1ee35c1cd4f0247dfbf96
+      if: failure()
+      with:
+        type: ${{ job.status }}
+        job_name: 'Extender release'
+        channel: '#defold-alarms-build'
+        url: ${{ secrets.SLACK_WEBHOOK }}
+

.github/workflows/docker-env-build.yml

@@ -0,0 +1,94 @@
+name: Build Docker environment container
+
+on:
+  workflow_dispatch:
+  push:
+    tags:
+      - "android*"
+      - "base*"
+      - "linux*"
+      - "emsdk*"
+      - "wine*"
+      - "winsdk*"
+      - "nssdk*"
+      - "ps4*"
+      - "ps5*"
+
+
+
+env:
+  GOOGLE_ARTIFACT_REGISTRY: europe-west1-docker.pkg.dev
+  PUBLIC_REGISTRY: europe-west1-docker.pkg.dev/extender-426409/extender-public-registry
+  PRIVATE_REGISTRY: europe-west1-docker.pkg.dev/extender-426409/extender-private-registry
+  PRIVATE_PLATFORMS: "nssdk ps4 ps5"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332
+    - name: Process tag name
+      id: name_postprocess
+      run: |
+          PRIVATE_PLATFORM_LIST="${{ env.PRIVATE_PLATFORMS }}"
+          PUBLIC_REGISTRY="${{ env.PUBLIC_REGISTRY }}"
+          PRIVATE_REGISTRY="${{ env.PRIVATE_REGISTRY }}"
+
+          SHORT_SHA="$(echo ${GITHUB_SHA} | cut -c1-8)"
+          TAG_NAME=${{ github.ref_name }}
+          DOCKERFILE="${TAG_NAME%%-*}"
+          PLATFORM="${DOCKERFILE%%.*}"
+          PLATFORM_VERSION=$(if [[ "$DOCKERFILE" == *"."* ]]; then echo "${DOCKERFILE#*.}"; else echo ""; fi)
+          IMAGE_VERSION="${TAG_NAME#*-}"
+          IMAGE_NAME=$(if [[ "$PLATFORM_VERSION" != "" ]]; then echo "$PLATFORM-$PLATFORM_VERSION"; else echo "$PLATFORM"; fi)
+          TARGET_REGISTRY=$(if [[ $PRIVATE_PLATFORM_LIST == *$PLATFORM* ]]; then echo $PRIVATE_REGISTRY; else echo $PUBLIC_REGISTRY; fi)
+
+          echo "TAG_NAME $TAG_NAME"
+          echo "DOCKERFILE $DOCKERFILE"
+          echo "PLATFORM $PLATFORM"
+          echo "PLATFORM_VERSION $PLATFORM_VERSION"
+          echo "IMAGE_VERSION $IMAGE_VERSION"
+          echo "IMAGE_NAME $IMAGE_NAME"
+          echo "TARGET_REGISTRY $TARGET_REGISTRY"
+
+          echo "short_sha=$SHORT_SHA" >> $GITHUB_OUTPUT
+          echo "dockerfile=$DOCKERFILE" >> $GITHUB_OUTPUT
+          echo "platform=$PLATFORM" >> $GITHUB_OUTPUT
+          echo "platform_version=$PLATFORM_VERSION" >> $GITHUB_OUTPUT
+          echo "image_version=$IMAGE_VERSION" >> $GITHUB_OUTPUT
+          echo "image_name=$IMAGE_NAME" >> $GITHUB_OUTPUT
+          echo "target_registry=$TARGET_REGISTRY" >> $GITHUB_OUTPUT
+    - name: Login to Google Artifact registry
+      uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446
+      with:
+        registry: ${{ env.GOOGLE_ARTIFACT_REGISTRY }}
+        username: _json_key
+        password: ${{ secrets.DOCKER_REGISTRY_JSON_KEY }}
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb
+    - name: Build and push
+      env:
+        DM_PACKAGES_URL:  ${{ secrets.S3_URL }}
+      uses: docker/build-push-action@c382f710d39a5bb4e430307530a720f50c2d3318
+      with:
+        context: ${{ github.workspace }}/server/docker/
+        file: ${{ github.workspace }}/server/docker/Dockerfile.${{ steps.name_postprocess.outputs.dockerfile }}-env
+        pull: true
+        push: true
+        platforms: |
+          linux/amd64
+        secret-envs: |
+          DM_PACKAGES_URL=DM_PACKAGES_URL
+        tags: |
+          ${{ steps.name_postprocess.outputs.target_registry }}/extender-${{ steps.name_postprocess.outputs.image_name }}-env:${{ steps.name_postprocess.outputs.image_version }}
+          ${{ steps.name_postprocess.outputs.target_registry }}/extender-${{ steps.name_postprocess.outputs.image_name }}-env:${{ steps.name_postprocess.outputs.short_sha }}
+          ${{ steps.name_postprocess.outputs.target_registry }}/extender-${{ steps.name_postprocess.outputs.image_name }}-env:latest
+
+    - name: Notify if tests failed
+      uses: homoluctus/slatify@master
+      if: failure()
+      with:
+        type: ${{ job.status }}
+        job_name: 'Extender: Docker environment container'
+        channel: '#defold-alarms-build'
+        url: ${{ secrets.SLACK_WEBHOOK }}