Complete maestro staging/production standalone deployment

[nuclearcat]

Since a while we have half-completed production/staging deployment for maestro.
We need to complete it, make run by assigned team without sysadmin assistance, and transparent for the rest of the team.

Problem

We have a half-completed production/staging deployment for maestro. We need to complete it, make run by assigned team without sysadmin assistance, and transparent for the rest of the team.

Solution

1. Staging deployment process

The first step is to define the deployment process. This process should include the following steps:

• Create a staging snapshot of source code (merge pending PR, create a new branch, etc.)
• Build and push staging docker images (compiler, runtime images for api, pipeline, lava-callback etc)
• Deploy staging docker images to staging environment (docker-compose setup)
• Run tests on staging environment

2. Production deployment process

Same as above, except step 1 doesn't need merging PRs.
Additional step 0:

• Create full/incremental backups according to the backup policy

3. Production rollback process

• Design a rollback process in case of failure

4. Monitoring

• Setup monitoring for the staging and production environment (API endpoints, CPU, memory, each subsystem feedback, etc.)

5. Logging

• Setup logging for the production environment (k8s)

6. Documentation

• Document the deployment process, rollback process, monitoring, and logging