Deployment of build.opensuse.org

Running the deployment

1. Check get the commit id of the currently deployed code

     osc api /about
   

2. Verify that newly built packages are available on the OBS instance by checking the versions:

     zypper search --details obs-api
   

   Get the commit id of the currently available (published) packages

   NOTE: If new packages don't get published due to running builds, you might want to disable the GitHub / OBS integration.

3. Inspect the code (diff) for migrations and other changes that require attention. You can do this in GitHub with the following url:

   https://github.com/openSUSE/open-build-service/compare/commit1...commit2
   

   i.e using curl curl -L "https://github.com/openSUSE/open-build-service/compare/80cb3011e...3c1e92ea8.diff"

   Go to the "When there is no migration" section if there are not migrations, or to the "When there are migrations" section otherwise.

When there is no migration

1. Check /etc/motd, it lists all pull requests that manually got applied to the deployment. Either merge them and wait for new packages or make sure you manually apply the changes after deployment again.

2. Update the packages

   zypper update --best-effort --details obs-api
   (--best-effort is to get the latest version for which dependencies are met)
   

3. Check the state of OBS. Especially the areas that have been touched by code you just deployed. Make sure delayed jobs are running.

4. Apply monkey patches if there are any and modify the Monkey Patch section of /etc/motd accordingly.

When there are migrations

1. If the migrations are just data, you don't need to wait for the maintenance window. You can apply those migrations right away as they won't cause any downtime. You will recognize data migrations when they contain db/data in their path, instead of db/migrate. Once applied, you can skip the rest of this section.

2. Wait for our maintenance window

   Thursday, 8am to 10am CET / CEST

3. Check /etc/motd, it lists all pull requests that manually got applied to the deployment. Either merge them and wait for new packages or make sure you manually apply the changes after deployment again.

4. Put apache into maintenance mode in /etc/sysconfig/apache2

     APACHE_SERVER_FLAGS="STATUS MAINTENANCE"
   

5. Restart apache

     rcapache2 restart
   

6. Update the packages

     zypper update --best-effort --details obs-api
     (--best-effort is to get the latest version for which dependencies are met)
   

7. Migrate the database

     run_in_api rails db:migrate:with_data
   

8. Put apache out of maintenance mode in /etc/sysconfig/apache2

     APACHE_SERVER_FLAGS="STATUS"
   

9. Restart apache and delayed jobs

     rcapache2 restart
   

10. Check the state of OBS. Especially the areas that have been touched by code you just deployed. Make sure delayed jobs are running.

11. Apply monkey patches if there are any and modify the Monkey Patch section of /etc/motd accordingly.

12. Check that the Factory dashboard still works https://build.opensuse.org/project/dashboard/openSUSE:Factory. If not, inform Ludwig / DimStar / Coolo.

Merging Pull Requests With Migrations

Pull Requests which introduce migrations (which are expected to raise a downtime) are required to get merged on Wednesday morning (Those pull requests are labeled as Merge on Wednesday) to make sure the migrations can be run in the maintenance window.

The creator of the pull request is responsible to merge those changes.

Troubleshooting

Delayed Jobs

For delayed jobs, clockworkd and indexing sphinx there is a systemd target:

systemctl obs-api-support.target start|stop|status|restart

To make sure everything is running fine:

# The systemd target should display as active (running), in green
systemctl status obs-api-support.target

# All dependant services should display active (running), in green
systemctl list-dependencies obs-api-support.target

If one of this is already running and systemctl obs-api-support.target stop does not stop it, you have to manually kill it (ps aux | grep delayed) otherwise restarting will fail.

To get an overview about remaining jobs, you can run:

run_in_api rails runner script/delayed_job_stats.rb

Search

Sometimes, we search in OBS for a package/project name and we don't obtain any results, but we know the package/project exist (we know the direct link to it).

Both projects and packages are indexed for searchs. There is a rake task that updates the indexes periodically. However, sometimes the task doesn't run. To check this and re-run, please follow the steps:

• Go to production server via SSH.
• $ run_in_api rails ts:index
• You might have the following output

WARNING: The following indexing guard files exist:  /srv/www/obs/api/db/sphinx/production/ts---all.tmp
These files indicate indexing is already happening. If that is not the case, these files should be deleted to ensure all indices can be processed

• As we can read in the output, the task is not going to run again if the guard file is there. Maybe the process didn't finish the last time it run or there was a deployment at the same time and it got stuck.
• Then check if that guard file is too old. ls -lah /srv/www/obs/api/db/sphinx/production/ts---all.tmp
• If the file is new (less than a few hours), please wait, maybe the ts:index task is running now. Stop reading here and check again after one hour.
• If the file is old, remove it.
• Run $ run_in_api rails ts:index again. It will take a few minutes.
• Check in OBS WebUI if search works now.

RabbitMQ

When you see errors like SSL_connect SYSCALL returned=5 errno=0 state=unknown state in Errbit this usually means that there is some issue with the RabbitMQ server / connection. The maintenance window of the RabbitMQ server is Thursday, 8:00am to 10:00am CET. This can also cause this issue.

Airbrake/Errbit

To make sure that Airbrake is working, run the following command run_in_api rake airbrake:test and it will send an airbrake event to our errbit. It should show up in the OBS frontend App.

Disabling the rpm test suite

In case that the deployment caused some breakage and you might need to build some new packages quickly. In that case you might want to temporarily disable the test suite in our rpm spec. To do that add %disable_obs_test_suite 1 to the project config of OBS:Server:Unstable. Important: It needs to go into the Macros: ... :Macros section.

Downgrading packages

It might happen that a deployment breaks OBS badly and you need to get it to work again quickly. In that case you can check if zypper still has the old packages in it's cache. Just run zypper se --details and verify that the package version you want is still available. If that's the case, run zypper in --oldpackage obs-api-$VERSION.$ARCH to downgrade the package.

Disabling OBS integration in GitHub

OBS packages are build whenever a PR get's merged to master. This might delay publishing of built packages. To prevent this, disable the OBS integration in GitHub:

1. Go to settings of the OBS GitHub project
2. Select the 'Integration & services' tab and click on 'Edit' in the OBS column
3. Uncheck the 'Activate' checkbox and 'Update services'
4. Once the deployment is done activate the checkbox again ;-)

Deployment of the obs-cloud-uploader package

• ssh to our cloud uploader instance
• Run zypper up obs-cloud-uploader