Deployment of build.opensuse.org

Running the deployment

1. Check get the commit id of the currently deployed code

     osc api /about
   

2. Verify that newly built packages are available on the OBS instance by checking the versions:

     zypper search --details obs-api
   

   Get the commit id of the currently available (published) packages

   NOTE: If new packages don't get published due to running builds, you might want to disable the GitHub / OBS integration.

3. Inspect the code (diff) for migrations and other changes that require attention. You can do this in GitHub with the following url:

   https://github.com/openSUSE/open-build-service/compare/commit1...commit2
   

   i.e using curl curl -L "https://github.com/openSUSE/open-build-service/compare/80cb3011e...3c1e92ea8.diff"

   Go to the "When there is no migration" section if there are not migrations, or to the "When there are migrations" section otherwise.

When there is no migration

1. Check /etc/motd, it lists all pull requests that manually got applied to the deployment. Either merge them and wait for new packages or make sure you manually apply the changes after deployment again. Read here to know how to apply them manually.

2. Update the packages

   To avoid the package being accidentally deployed, we add a zypper lock locks(5) to the obs-api package.

   To be able to update the package, remove the lock:

   zypper rl obs-api
   

   After that you are able to update the package:

   zypper update --best-effort --details obs-api
   (--best-effort is to get the latest version for which dependencies are met)
   

   After deployment, please lock the package again:

   zypper al obs-api

   To check if there is a lock in place:

   zypper ll

3. Check the state of OBS. Especially the areas that have been touched by code you just deployed. Make sure delayed jobs are running.

4. Apply monkey patches if there are any and modify the Monkey Patch section of /etc/motd accordingly.

When there are migrations

If the migrations are just data, you don't need to wait for the maintenance window. You can apply those migrations right away as they won't cause any downtime. You will recognize data migrations when they contain db/data in their path, instead of db/migrate. To do so, apply just points 5 and 6 on this section and then jump to 9.

1. Wait for our maintenance window

   Thursday, 8am to 10am CET / CEST

2. Check /etc/motd, it lists all pull requests that manually got applied to the deployment. Either merge them and wait for new packages or make sure you manually apply the changes after deployment again.

3. Put apache into maintenance mode in /etc/sysconfig/apache2

     APACHE_SERVER_FLAGS="STATUS MAINTENANCE"
   

4. Restart apache

     rcapache2 restart
   

5. Update the packages

     zypper update --best-effort --details obs-api
     (--best-effort is to get the latest version for which dependencies are met)
   

6. Migrate the database

     run_in_api rails db:migrate:with_data
   

7. Put apache out of maintenance mode in /etc/sysconfig/apache2

     APACHE_SERVER_FLAGS="STATUS"
   

8. Restart apache and delayed jobs

     rcapache2 restart
   

9. Check the state of OBS. Especially the areas that have been touched by code you just deployed. Make sure delayed jobs are running.

10. Apply monkey patches if there are any and modify the Monkey Patch section of /etc/motd accordingly.

Merging Pull Requests With Migrations

Pull Requests which introduce migrations (which are expected to raise a downtime) are required to get merged on Wednesday morning (Those pull requests are labeled as Merge on Wednesday) to make sure the migrations can be run in the maintenance window.

The creator of the pull request is responsible to merge those changes.

Troubleshooting

Delayed Jobs

For delayed jobs, clockworkd and indexing sphinx there is a systemd target:

systemctl obs-api-support.target start|stop|status|restart

To make sure everything is running fine:

# The systemd target should display as active (running), in green
systemctl status obs-api-support.target

# All dependant services should display active (running), in green
systemctl list-dependencies obs-api-support.target

If one of this is already running and systemctl obs-api-support.target stop does not stop it, you have to manually kill it (ps aux | grep delayed) otherwise restarting will fail.

To get an overview about remaining jobs, you can run:

run_in_api rails runner script/delayed_job_stats.rb

Assets issues

From time to time we have some issues with the CSS/JS assets. If application.css or application.js are missing (you will notice it, when you see unusual errors in your javascript console, specially a 404 when trying to retrieve it) then there are probably more than one sprocket manifest in production. Go to the public folder, check which one comes from the package and delete the one which doesn't. After that reload the application.

cd public/assets
rpm -qf .sprockets-manifest*
rm .sprockets-manifest-$SOMEHASH.json
cd ../..
touch tmp/restart.txt

RabbitMQ

When you see errors like SSL_connect SYSCALL returned=5 errno=0 state=unknown state in Errbit this usually means that there is some issue with the RabbitMQ server / connection. The maintenance window of the RabbitMQ server is Thursday, 8:00am to 10:00am CET. This can also cause this issue.

Airbrake/Errbit

To make sure that Airbrake is working, run the following command run_in_api rake airbrake:test and it will send an airbrake event to our errbit. It should show up in the OBS frontend App.

Disabling the rpm test suite

In case that the deployment caused some breakage and you might need to build some new packages quickly. In that case you might want to temporarily disable the test suite in our rpm spec. To do that add %disable_obs_test_suite 1 to the project config of OBS:Server:Unstable. Important: It needs to go into the Macros: ... :Macros section.

Downgrading packages

It might happen that a deployment breaks OBS badly and you need to get it to work again quickly. In that case you can check if zypper still has the old packages in it's cache. Just run zypper se --details and verify that the package version you want is still available. If that's the case, run zypper in --oldpackage obs-api-$VERSION.$ARCH to downgrade the package.

Disabling OBS integration in GitHub

OBS packages are build whenever a PR get's merged to master. This might delay publishing of built packages. To prevent this, disable the OBS integration in GitHub:

1. Go to settings of the OBS GitHub project
2. Select the 'Integration & services' tab and click on 'Edit' in the OBS column
3. Uncheck the 'Activate' checkbox and 'Update services'
4. Once the deployment is done activate the checkbox again ;-)

Manual Fixing on Production

As far as possible, when a bug is found we should act as usual: create a Pull Request, wait for it to be reviewed and merged and then wait until the changes can be deployed.

This process usually takes so long that, sometimes, we can not wait for it to finish. For example, when the bug is blocking someone's work or even the whole application.

Only in such cases, we can apply the changes manually on production (monkey patch) following these steps:

• Access production server via SSH.
• Go to the application's directory.
• Apply the fix manually.
• Run touch tmp/restart.txt to restart the server (Passenger).
• Add to /etc/motd the link to the Pull Request that fixes the problem. In the next deployment, that PR is going to be applied and will replace the manual changes.

Deployment of the obs-cloud-uploader package

• ssh to our cloud uploader instance
• Run zypper up obs-cloud-uploader