feat: #638 Add cronjob config for database restore

[MCatherine1994]

• added a yaml file for setting up a cronjob for database restore, refs: Test recovery of FOM database #638

Full documentation is at: https://github.com/bcgov/nr-fom/wiki/Database-Restore

---

Thanks for the PR!

Any successful deployments (not always required) will be available below.

• api
• admin
• public

Once merged, code will be promoted and handed off to following workflow run.

• Main Merge Workflow

---

Thanks for the PR!

Any successful deployments (not always required) will be available below.

• api
• admin
• public

Once merged, code will be promoted and handed off to following workflow run.

• Main Merge Workflow

[basilv]

Reading the wiki process: "Do a manual backup of the current database (can do that in the database pod, so this temporary backup will be stored in the database volume)" -> the specific commands to run should be provided. I'm a little concerned by doing this versus the normal backup process as if you need to restore from this special backup you need a special restore process, but on the other hand if the regular restore process fails, this approach should still succeed. Just want to be careful that if this special backup isn't stored within a PVC, then if the DB pod restarts the backup will be lost.

[webgismd]

Team Evergreen has in their backlog to look at DB restore/backup as part of STRA-- goal was to not use PVC but object storage. they had buckets created for this purpose, but I am not sure how far they have gone with it yet. @craigyu or @DerekRoberts may know?

[DerekRoberts]

@webgismd Yup! @RMCampos is working on that. I'll be sidekicking and making it easier to repeat.

[basilv]

@webgismd for FOM at least, the database is small and we store a limited number of backups so storage demands are limited. But there are some nice aspects to using object storage instead of a PVC - better resistance to ransomware-style attacks, especially if you can structure your object storage permissions to be insert-only... I'd be more worried though about every team developing a completely different backup/restore process, versus having a defined method for OpenShift Postgres DBs that all the teams can leverage.

[webgismd]

I 100% agree here @basilv , FDS is essentially on its own to develop a pattern for itself. I don't see alot of tactile support from other teams in this area. Perhaps something to discuss with @RMCampos @jazzgrewal @craigyu @paulushcgcj @abschwenker @DerekRoberts

[DerekRoberts]

@webgismd @basilv Big yes! Let's solve this and get it out there. Our teams, quickstart, developer.gov, stink overflow, etc.

[paulushcgcj]

@webgismd for FOM at least, the database is small and we store a limited number of backups so storage demands are limited. But there are some nice aspects to using object storage instead of a PVC - better resistance to ransomware-style attacks, especially if you can structure your object storage permissions to be insert-only... I'd be more worried though about every team developing a completely different backup/restore process, versus having a defined method for OpenShift Postgres DBs that all the teams can leverage.

The base here was derived from client. In our case, we will keep both a PVC and an S3 backup, as the PVC is a "hot" copy, that will be the more recent one while the S3 will have all past and all current ones, but will be handled as a "cold" copy. This was at least the idea/suggestion I gave and is what we're aiming to do on client.