Upgrade checklist updates #967
Conversation
[skip ci]
dacook
force-pushed
the
upgrade-updates
branch
from
32186d7 to
7ba0b7b
Compare
| @@ -17,9 +19,10 @@ Checklist based on general guide https://github.com/openfoodfoundation/ofn-insta | |||
| - [ ] DNS: add temporary domain (eg `prod2.openfoodnetwork.org`) | |||
|
|
|||
| ### config | |||
| - [ ] Add temporary name to `inventory/hosts` | |||
| - [ ] Add temporary name to `inventory/hosts` (suggest doing this on separate branch) | |||
There was a problem hiding this comment.
I'm wondering about this temporary host name. I don't think it's temporary. Don't we want to be able to identify a unique host like prod4.openfoodnetwork.org.au which is providing the site openfoodnetwork.org.au?
| @@ -41,7 +44,7 @@ Then setup new server. Ensure you have the correct secrets (current secrets are | |||
| - [ ] Setup direct ssh access for `ofn-admin` and `openfoodnetwork` as per guide | |||
| @@ -9,6 +9,8 @@ assignees: '' | |||
|
|
|||
| Checklist based on general guide https://github.com/openfoodfoundation/ofn-install/wiki/Migrating-a-Production-Server | |||
|
|
|||
| Tip: find/replace to set up most commands ready to go, eg: `x_prod` -> `ca_prod` | |||
There was a problem hiding this comment.
I wonder if we should use easy to replace example names like old_server and new_server which could be prod or staging. Some of the guide just refers to x instead of x_prod as well. It would be good to make this consistent.
| - [ ] old server: make a tiny data change to verify later (eg add `.` in meta description `/admin/general_settings/edit`) | ||
|
|
||
| ### switchover: old server | ||
| - [ ] 🚧 `maintenance_mode.yml` | ||
| - [ ] 🚧 `ansible-playbook playbooks/maintenance_mode.yml -l x_prod` | ||
| - [ ] `sudo systemctl stop sidekiq redis-jobs puma` |
There was a problem hiding this comment.
Should we add disable as well? Just in case the server gets restarted accidentally. It means that rolling back is also one step more (enable) but it makes the intent of retiring the server at this stage clearer. And it could prevent us from accidentally running a start or restart command on the old server.
| - [ ] `db_transfer.yml` ~3min | ||
| - [ ] `sudo systemctl stop postgres` (ensure other integrations no longer touch it) | ||
| - [ ] `transfer_assets.yml` just in case | ||
| - [ ] `sudo systemctl stop postgresql` (ensure other integrations no longer touch it) |
There was a problem hiding this comment.
Same here. I think that disable would be good as well.
| - [ ] `db_transfer.yml` ~3min | ||
| - [ ] `sudo systemctl stop postgres` (ensure other integrations no longer touch it) | ||
| - [ ] `transfer_assets.yml` just in case | ||
| - [ ] `sudo systemctl stop postgresql` (ensure other integrations no longer touch it) | ||
|
|
||
| ### switchover: new server | ||
| - [ ] `sudo systemctl restart puma; sudo systemctl start sidekiq redis-jobs` |
There was a problem hiding this comment.
Sidekiq is disabled at this point. We have to enable it first before we can start it. But I personally would probably do it staggered:
- Start puma and check with hosts file.
- Start sidekiq and check log file.
- Then install proxy forwarding.
There was a problem hiding this comment.
How much time did you need for testing? If it's short, maybe we could do it during maintenance mode which would enable us to then switch over straight away. It would save us all the work of transferring data twice and resetting in between.
Some more updates to the upgrade checklist and readme.
I'd like to update the process further and simplify it.. but haven't got time for that.