DM-44635: mobu in CI

[fajpunk]

Mobu in CI

• Converts the existing refresh functionality into a GitHub app integration (from a simple webhook integration)
• Adds a new GitHub app integration to generate GitHub actions checks for commits pushed to notebook repo branches that are part of active PRs. These checks trigger and report on a solitary Mobu run of the changed notebooks in the commit
  • Special care has been taken to not leave these checks in a forever-in-progress state, even in the case of (graceful) mobu shutdown/restart
• All GitHub app functionality can be enabled and disabled per environment based on app config
• Adds a documenteer docs site and publishes to https://mobu.lsst.io
• See the here (or docs/ in this PR) for more detailed info on how this works from both user and admin perspectives
• The commits should be a rough guide to the different changes in this PR

There is a separate GitHub app for each environment for each of the refresh and CI functionalities. This lets you enable these integrations for different combinations of environments. For example, you can enable the auto-refresh integration in idfdev, idfint, usdfdev, usdfint, and usdfprod, but the CI integration only in idfint and usdfint.

Goes with the Phalanx changes in lsst-sqre/phalanx#3453

Testing

Test repo

This repo has a bunch of different notebooks in a bunch of different directories, some of which intentionally throw exceptions.
It contains a script to update certain notebooks, conditionally updating the poison ones.

I installed the data-dev Mobu CI app in the lsst-sqre GitHub org, and configured it to access "Only select repositories", and I added the dfuchs-test-mobu repo in the dropdown.

Local testing with smee.io

I installed the tool from smee.io and pointed it at my running local mobu:

❯ smee -p 8000 -P /mobu/github/ci/webhook
Forwarding https://smee.io/kTysHB50d4pgUmRN to http://127.0.0.1:8000/mobu/github/ci/webhook

I configured the data-dev GitHub Mobu CI app to send webooks to the smee URL.
I ran my local mobu against data-dev:

#!/usr/bin/env bash

set -euo pipefail

ci_config_dir="/tmp/mobu_test"
ci_config_file="mobu_config.yaml"
ci_config_path="$ci_config_dir/$ci_config_file"

mkdir -p "$ci_config_dir"
cat <<- 'END' > "$ci_config_path"
	users:
	- username: bot-mobu-ci-1
	- username: bot-mobu-ci-2
	accepted_github_orgs:
	- lsst-sqre
END

export MOBU_ENVIRONMENT_URL=https://data-dev.lsst.cloud
export MOBU_GAFAELFAWR_TOKEN=$(op read "op://Employee/data-dev.lsst.cloud personal token/credential")
#export MOBU_AUTOSTART_PATH=~/junk/autostart.yaml
export MOBU_LOG_LEVEL=debug
export MOBU_GITHUB_REFRESH_ENABLED=true
export MOBU_GITHUB_REFRESH_APP_WEBHOOK_SECRET=$(op read "op://RSP data-dev.lsst.cloud/mobu/github-refresh-app-webhook-secret")
export MOBU_GITHUB_CI_APP_ENABLED=true
export MOBU_GITHUB_CI_APP_WEBHOOK_SECRET=$(op read "op://RSP data-dev.lsst.cloud/mobu/github-ci-app-webhook-secret")
export MOBU_GITHUB_CI_APP_ID=$(op read "op://RSP data-dev.lsst.cloud/mobu/github-ci-app-id")
export MOBU_GITHUB_CI_APP_PRIVATE_KEY=$(op read "op://RSP data-dev.lsst.cloud/mobu/github-ci-app-private-key" | base64 -d)
export MOBU_GITHUB_CONFIG_PATH="$ci_config_path"

uvicorn mobu.main:app 2>&1

The requests sent by the smee proxy have :port suffixes in the X-Forwarded-For values. Safir doesn't handle this (and I don't think it's Safir's fault; I think the port should be in X-Forwarded-Port), so I ran a local safir:

index ce732db..ce2d693 100644
--- a/requirements/main.in
+++ b/requirements/main.in
@@ -22,6 +22,6 @@ pydantic-settings
 pyvo
 pyyaml
-safir>=5.0.0
+# safir>=5.0.0
+-e /home/danfuchs/src/safir
 shortuuid
 structlog

With this change:

index 2a3f40c..7211241 100644
--- a/src/safir/middleware/x_forwarded.py
+++ b/src/safir/middleware/x_forwarded.py
@@ -116,5 +116,5 @@ class XForwardedMiddleware:
         return [
             ip_address(addr)
-            for addr in (a.strip() for a in forwarded_for_str[0].split(","))
+            for addr, _ in (a.strip().split(':') for a in forwarded_for_str[0].split(","))
             if addr
         ]

data-dev testing

Queued

In Progress

Success

Failure

Failure due to mobu stop/restart

Success re-running check

Success re-running suite

[rra]

I just have a bunch of minor comments and only a few things that jumped out at me.

For the future (no need to break this up), this is a lot for one PR. Ideally this would be three or four PRs (the documentation build system and new manual as a separate PR, for instance), which would make it easier to review, if only because it would be easier to do one PR, take a break, come back and do another, etc. I know it's really hard to break up large PRs like this when you want to test the combination of them all.

Usually what I do is maintain a branch with a whole queue of commits, test them all together, and then cherry-pick them into separate branches to submit as PRs. That way the reviewer can work through the stack while I'm continuing development and neither blocks on the other.

[rra]

Nit: Once we start generating Python API docs, which I usually do even for applications when they have a doc site since it can be very useful for development, it will want all of this to be reformatted as rST, not Markdown.

[rra]

Future work here too: we should probably just shut down the notebook immediately rather than wait for the jobs to finish.