Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Cache postgres volume after first migration #3488

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
c661c01
ci: Cache postgres volume after first migration
BYK Dec 21, 2024
3d6d96c
try hack
BYK Dec 21, 2024
f23d2a4
ofc use sudo
BYK Dec 21, 2024
def10cc
try something softer
BYK Dec 21, 2024
c58ca1c
fix deprecated output setting
BYK Dec 21, 2024
e4fee34
use cache in upgrade test too
BYK Dec 21, 2024
986e419
add ref for hack
BYK Dec 21, 2024
8e8d8c2
debug
BYK Dec 21, 2024
09a9572
some cache key DRYing
BYK Dec 21, 2024
cd2dc30
fix cache path
BYK Dec 21, 2024
cd33fa4
fix cache save condition (can also be null)
BYK Dec 21, 2024
44a8d63
cancel old jobs on the same branch or sha
BYK Dec 21, 2024
e185b11
remove debug line
BYK Dec 21, 2024
b2dac61
simplify
BYK Dec 21, 2024
97848ca
save permissions?
BYK Dec 21, 2024
b9f4337
be more aggressive
BYK Dec 21, 2024
2607b51
sigh...
BYK Dec 21, 2024
470e99a
add forgotten recursive flag
BYK Dec 21, 2024
b9c9708
Merge branch 'master' into byk/ci/postgres-cache
BYK Dec 21, 2024
dd79f7e
become reckless
BYK Dec 21, 2024
00afa95
don't be reckless
BYK Dec 21, 2024
d4fccbb
Merge branch 'master' into byk/ci/postgres-cache
BYK Dec 21, 2024
7eb44f9
recursive perms
BYK Dec 21, 2024
a85cdda
version and bump cache key
BYK Dec 21, 2024
e6cddde
Merge branch 'master' into byk/ci/postgres-cache
BYK Dec 21, 2024
ff8b12b
never give up
BYK Dec 21, 2024
6bd2d90
100th time is the charm?
BYK Dec 21, 2024
4757291
include clickhouse in the cache
BYK Dec 22, 2024
9ac67b1
docker compose up --wait before tests
BYK Dec 22, 2024
f951609
more accurate step names
BYK Dec 22, 2024
35116d7
remove geoipupdate from compose
BYK Dec 22, 2024
d91d9dc
include kafka too
BYK Dec 22, 2024
d48680d
Revert "remove geoipupdate from compose"
BYK Dec 22, 2024
7ee17c8
Revert "docker compose up --wait before tests"
BYK Dec 22, 2024
25135b5
include snuba image sha in cache key
BYK Dec 22, 2024
e1502cc
be additive
BYK Dec 22, 2024
b04ec04
no kafka for now
BYK Dec 22, 2024
f69ac05
try kafka with fresh cache and additive perms
BYK Dec 23, 2024
7c2c748
remove kafka again, bump version
BYK Dec 23, 2024
f41a7e6
Merge branch 'master' into byk/ci/postgres-cache
BYK Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ on:
pull_request:
schedule:
- cron: "0 0,12 * * *"

concurrency:
group: ${{ github.ref_name || github.sha }}
cancel-in-progress: true

defaults:
run:
shell: bash
Expand Down Expand Up @@ -66,9 +71,48 @@ jobs:
sudo curl -L https://github.com/docker/compose/releases/download/v2.26.0/docker-compose-`uname -s`-`uname -m` -o "/usr/local/lib/docker/cli-plugins/docker-compose"
sudo chmod +x "/usr/local/lib/docker/cli-plugins/docker-compose"

- name: Prepare Docker Volume Caching
id: cache_key
run: |
# Set permissions for docker volumes so we can cache and restore
sudo chmod o+x /var/lib/docker
sudo chmod -R o+rwx /var/lib/docker/volumes
source .env
SENTRY_IMAGE_SHA=$(docker buildx imagetools inspect $SENTRY_IMAGE --format "{{println .Manifest.Digest}}")
echo "SENTRY_IMAGE_SHA=$SENTRY_IMAGE_SHA" >> $GITHUB_OUTPUT
SNUBA_IMAGE_SHA=$(docker buildx imagetools inspect $SNUBA_IMAGE --format "{{println .Manifest.Digest}}")
echo "SNUBA_IMAGE_SHA=$SNUBA_IMAGE_SHA" >> $GITHUB_OUTPUT

- name: Restore DB Volumes Cache
id: restore_cache
uses: actions/cache/restore@v4
with:
key: db-volumes-v4-${{ steps.cache_key.outputs.SENTRY_IMAGE_SHA }}-${{ steps.cache_key.outputs.SNUBA_IMAGE_SHA }}
restore-keys: |
db-volumes-v4-${{ steps.cache_key.outputs.SENTRY_IMAGE_SHA }}
db-volumes-v4-
path: |
/var/lib/docker/volumes/sentry-postgres/_data
/var/lib/docker/volumes/sentry-clickhouse/_data

- name: Install ${{ env.LATEST_TAG }}
run: ./install.sh

- name: Prepare Docker Volume Caching
run: |
# Set permissions for docker volumes so we can cache and restore
sudo chmod o+x /var/lib/docker
sudo chmod -R o+rx /var/lib/docker/volumes

- name: Save DB Volumes Cache
if: steps.restore_cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
key: ${{ steps.restore_cache.outputs.cache-primary-key }}
path: |
/var/lib/docker/volumes/sentry-postgres/_data
/var/lib/docker/volumes/sentry-clickhouse/_data

- name: Checkout current ref
uses: actions/checkout@v4

Expand Down Expand Up @@ -125,13 +169,52 @@ jobs:
sudo curl -L https://github.com/docker/compose/releases/download/${{ matrix.compose_version }}/docker-compose-`uname -s`-`uname -m` -o "${{ matrix.compose_path }}/docker-compose"
sudo chmod +x "${{ matrix.compose_path }}/docker-compose"

- name: Prepare Docker Volume Caching
id: cache_key
run: |
# Set permissions for docker volumes so we can cache and restore
sudo chmod o+x /var/lib/docker
sudo chmod -R o+rwx /var/lib/docker/volumes
source .env
SENTRY_IMAGE_SHA=$(docker buildx imagetools inspect $SENTRY_IMAGE --format "{{println .Manifest.Digest}}")
echo "SENTRY_IMAGE_SHA=$SENTRY_IMAGE_SHA" >> $GITHUB_OUTPUT
SNUBA_IMAGE_SHA=$(docker buildx imagetools inspect $SNUBA_IMAGE --format "{{println .Manifest.Digest}}")
echo "SNUBA_IMAGE_SHA=$SNUBA_IMAGE_SHA" >> $GITHUB_OUTPUT

- name: Restore DB Volumes Cache
id: restore_cache
uses: actions/cache/restore@v4
with:
key: db-volumes-v4-${{ steps.cache_key.outputs.SENTRY_IMAGE_SHA }}-${{ steps.cache_key.outputs.SNUBA_IMAGE_SHA }}
restore-keys: |
db-volumes-v4-${{ steps.cache_key.outputs.SENTRY_IMAGE_SHA }}
db-volumes-v4-
path: |
/var/lib/docker/volumes/sentry-postgres/_data
/var/lib/docker/volumes/sentry-clickhouse/_data

- name: Install self-hosted
uses: nick-fields/retry@v3
with:
timeout_minutes: 10
max_attempts: 3
command: ./install.sh

- name: Prepare Docker Volume Caching
run: |
# Set permissions for docker volumes so we can cache and restore
sudo chmod o+x /var/lib/docker
sudo chmod -R o+rx /var/lib/docker/volumes

- name: Save DB Volumes Cache
if: steps.restore_cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
key: ${{ steps.restore_cache.outputs.cache-primary-key }}
path: |
/var/lib/docker/volumes/sentry-postgres/_data
/var/lib/docker/volumes/sentry-clickhouse/_data

- name: Integration Test
run: |
docker compose up --wait
Expand Down
Loading