Skip to content

Commit

Permalink
Initial implementation of in-place upgrades (#7689)
Browse files Browse the repository at this point in the history
This builds on all of the work done to put the standard library into
versioned namespaces.

The components currently are:
* A new `ADMINISTER prepare_upgrade()` command that returns the metadata
about the schema necessary to perform the upgrade. Currently a user
script runs this on every database and aggregates it, and we pass that
to the server.
* An `--inplace-upgrade <json file>` flag is added to the server, which
tells it to do an inplace upgrade using the `prepare_upgrade`
information in the file. It does this by running the a partial standard
library tpldbdump on each database, and then rebuilding the schema
tables for each database.
* Trampoline swapping, which requires some subtlety: we can't drop or
reorder columns on views, so I wrote some pl/pgsql that regenerates a
view preserving the order of previously existing fields and inserting
NULL for removed ones.
* Testing: I've added an initial testing script that runs `inittestdb`,
collects the `prepare_upgrade` results, applies a patch that adds and
removes some stuff to the std schema and bumps the version number,
upgrades the database in place, and then runs the full test suite.

Major progress on #6697.

Remaining TODOs (can largely be in follow ups):
- [ ] Extension upgrade support
- [ ] Parallelize upgrade (by using compiler pool)?
- [ ] Support upgrade while old version is running
- [ ] Better fault tolerance
- [ ] Extension triggers and user functions: regenerate them if a user
type points to schema object that has grown a new subtype
  • Loading branch information
msullivan authored Aug 29, 2024
1 parent a344e1d commit 198a73e
Show file tree
Hide file tree
Showing 16 changed files with 1,300 additions and 115 deletions.
6 changes: 5 additions & 1 deletion .github/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ all: workflows/nightly.yml \
workflows/tests-managed-pg.yml \
workflows/tests-ha.yml \
workflows/tests-pg-versions.yml \
workflows/tests-patches.yml
workflows/tests-patches.yml \
workflows/tests-inplace.yml

workflows/%.yml: workflows.src/%.tpl.yml workflows.src/%.targets.yml workflows.src/build.inc.yml workflows.src/ls-build.inc.yml
$(ROOT)/workflows.src/render.py $* $*.targets.yml
Expand All @@ -34,3 +35,6 @@ workflows.src/tests-pg-versions.tpl.yml: workflows.src/tests.inc.yml

workflows.src/tests-patches.tpl.yml: workflows.src/tests.inc.yml
touch $(ROOT)/workflows.src/tests-patches.tpl.yml

workflows.src/tests-inplace.tpl.yml: workflows.src/tests.inc.yml
touch $(ROOT)/workflows.src/tests-inplace.tpl.yml
1 change: 1 addition & 0 deletions .github/workflows.src/tests-inplace.targets.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data:
55 changes: 55 additions & 0 deletions .github/workflows.src/tests-inplace.tpl.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
<% from "tests.inc.yml" import build, calc_cache_key, restore_cache -%>

name: Tests of in-place upgrades

on:
schedule:
- cron: "0 3 * * *"
workflow_dispatch:
inputs: {}
push:
branches:
- "A-inplace*"

jobs:
build:
runs-on: ubuntu-latest

steps:
<%- call build() -%>
- name: Compute cache keys
run: |
<< calc_cache_key()|indent >>
<%- endcall %>

test:
runs-on: ubuntu-latest
needs: build

steps:
<<- restore_cache() >>

# Run the test
# TODO: Would it be better to split this up into multiple jobs?
- name: Test performing in-place upgrades
run: |
./tests/inplace-testing/test.sh vt
workflow-notifications:
if: failure() && github.event_name != 'pull_request'
name: Notify in Slack on failures
needs:
- build
- test
runs-on: ubuntu-latest
permissions:
actions: 'read'
steps:
- name: Slack Workflow Notification
uses: Gamesight/slack-workflow-status@26a36836c887f260477432e4314ec3490a84f309
with:
repo_token: ${{secrets.GITHUB_TOKEN}}
slack_webhook_url: ${{secrets.ACTIONS_SLACK_WEBHOOK_URL}}
name: 'Workflow notifications'
icon_emoji: ':hammer:'
include_jobs: 'on-failure'
Loading

0 comments on commit 198a73e

Please sign in to comment.