Merge pull request #354 from lsst-sqre/tickets/DM-44635/nicer-github-…

…integration-config

DM-44635: Nicer GitHub integration config

.pre-commit-config.yaml

@@ -7,14 +7,14 @@ repos:
       - id: check-toml
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.4.10
+    rev: v0.5.1
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
 
   - repo: https://github.com/adamchainz/blacken-docs
-    rev: 1.16.0
+    rev: 1.18.0
     hooks:
       - id: blacken-docs
         additional_dependencies: [black==24.3.0]

changelog.d/20240711_095604_danfuchs_nicer_github_integration_config.md

@@ -0,0 +1,6 @@
+<!-- Delete the sections that don't apply -->
+
+### Backwards-incompatible changes
+
+- GitHub CI and refresh app config are now each a separate, all-or-nothing set of config that comes from a mix of a yaml file and env vars. This requires some new and different Helm values in Phalanx (see https://mobu.lsst.io/operations/github_ci_app.html#add-phalanx-configuration)
+- The GitHub CI app now takes the scopes it assigns from config values, rather than hardcoding a list of scopes.

docs/operations/github_ci_app.rst

@@ -0,0 +1,72 @@
+######################################
+Adding a new GitHub CI app integration
+######################################
+
+
+Create a new GitHub app
+=======================
+
+#. Click the ``New GitHub App`` button in the `lsst-sqre org Developer Settings apps page <https://github.com/organizations/lsst-sqre/settings/apps>`__.
+#. Name it :samp:`mobu CI ({env URL or id if the URL is too long})`.
+#. Make sure the ``Active`` checkbox is checked in the ``Webhook`` section.
+#. Enter :samp:`https://{env URL}/mobu/github/ci/webhook` in the :guilabel:`Webhook URL` input.
+#. Generate a strong password to use as the webhook secret.
+#. Store this in the ``SQuaRE`` vault in the ``LSST IT`` 1Password account in an ``Server`` item named :samp:`mobu ({env URL})` in a ``password`` field named ``mobu-github-ci-app-webhook-secret``.
+#. Get this into the Phalanx secret store for that env under the key: ``github-ci-app-webhook-secret`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
+#. Enter this secret in the :guilabel:`Webhook secret (optional)` box in the GitHub App config.
+#. Select :menuselection:`Read and Write` in the dropdown of the :guilabel:`Checks` access category in the :guilabel:`Repository Permissions` section.
+#. Select :menuselection:`Read-only` in the dropdown of the :guilabel:`Contents` access category in the :guilabel:`Repository Permissions` section.
+#. Check the :guilabel:`Check suite` and :guilabel:`Check run` checkboxes in the :guilabel:`Subscribe to events` section.
+#. Select the :guilabel:`Any account` radio button in the :guilabel:`Where can this GitHub App be installed?` section.
+#. Click the :guilabel:`Create GitHub App` button.
+#. Find the :guilabel:`App ID` (an integer) in the :guilabel:`About` section. Get this into the Phalanx secret store for that env under the key: ``github-ci-app-id`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
+#. Click the :guilabel:`Generate a private key` button in the :guilabel:`Private keys` section.
+#. Store this private key in the same :samp:`mobu ({env URL})` item in a ``text`` key called ``github-mobu-ci-app-private-key``.
+#. Get this into the Phalanx secret store for that env under the key: ``github-ci-app-private-key`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
+
+Install the app for a repo
+==========================
+
+#. Go to new app’s homepage (something like https://github.com/apps/mobu-refresh-usdfdev).
+#. Click the :guilabel:`Install` button.
+#. Select the :guilabel:`Only select repositories` radio button.
+#. Select the repo in the dropdown.
+#. Click :guilabel:`Install`.
+
+Add Phalanx configuration
+=========================
+In :samp:`applications/mobu/values-{env}.yaml`, add a ``config.githubCiApp`` value:
+
+.. code:: yaml
+
+   config:
+     github:
+       acceptedGithubOrgs:
+         - lsst-sqre
+       users:
+         - username: "bot-mobu-ci-user-1"
+           uidnumber: 123
+           gidnumber: 456
+         - username: "bot-mobu-ci-user-2"
+           uidnumber: 789
+           gidnumber: 876
+       scopes:
+         - "exec:notebook"
+         - "exec:portal"
+         - "read:image"
+         - "read:tap"
+
+All items are required.
+
+``accepted_github_orgs``
+    A list of GitHub organizations from which this instance of Mobu will accept webhook requests.
+    Webhook requests from any orgs not in this list will get a ``403`` response.
+
+``users``
+    Follows the same rules as the ``users`` list in a flock autostart config.
+    The usernames must all start with ``bot-mobu``.
+    In envs with Firestore integration, you only need to specify ``username``.
+    In envs without it, you need to ensure that users are manually provisioned, and then you need all three of ``username``, ``uidnumber``, and ``gidnumber``.
+
+``scopes``
+    A list of `Gafaelfawr scopes <https://dmtn-235.lsst.io/#current-scopes>`__ to grant to the users running in the monkeys started from GitHub CI checks.

docs/operations/github_refresh_app.rst

@@ -0,0 +1,49 @@
+###########################################
+Adding a new GitHub Refresh app integration
+###########################################
+
+Adding the GitHub refresh app integration to a new environment requires configuring things in GitHub and Phalanx.
+
+Create a new GitHub app
+=======================
+
+
+#. Click the ``New GitHub App`` button in the `lsst-sqre org Developer Settings apps page <https://github.com/organizations/lsst-sqre/settings/apps>`__.
+#. Name it :samp:`mobu refresh ({env URL or id if the URL is too long})`.
+#. Make sure the :guilabel:`Active` checkbox is checked in the :guilabel:`Webhook` section.
+#. Enter :samp:`https://{env URL}/mobu/github/refresh/webhook` in the :guilabel:`Webhook URL` input.
+#. Generate a strong password to use as the webhook secret.
+#. Store this in the ``SQuaRE`` vault in the ``LSST IT`` 1Password account in an ``Server`` item named :samp:`mobu ({env URL})` in a ``password`` field called ``github-refresh-app-webhook-secret``.
+#. Get this into the Phalanx secret store for that env under the key: ``github-refresh-app-webhook-secret`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
+#. Enter this secret in the :guilabel:`Webhook secret (optional)` box in the GitHub App config.
+#. Select :menuselection:`Read and Write` in the dropdown of the :guilabel:`Checks` access category in the :guilabel:`Repository Permissions` section.
+#. Select :menuselection:`Read-only` in the dropdown of the :guilabel:`Contents` access category in the :guilabel:`Repository Permissions` section.
+#. Check the :guilabel:`Check suite` and :guilabel:`Check run` checkboxes in the :guilabel:`Subscribe to events` section.
+#. Select the :guilabel:`Any account` radio button in the :guilabel:`Where can this GitHub App be installed?` section.
+#. Click the :guilabel:`Create GitHub App` button.
+
+Install the app for a repo
+==========================
+
+#. Go to new app’s homepage (something like https://github.com/apps/mobu-refresh-usdfdev).
+#. Click the :guilabel:`Install` button.
+#. Select the :guilabel:`Only select repositories` radio button.
+#. Select the repo in the dropdown.
+#. Click :guilabel:`Install`.
+
+Add Phalanx configuration
+=========================
+In :samp:`applications/mobu/values-{env}.yaml`, add a ``config.githubRefreshApp`` value:
+
+.. code:: yaml
+
+   config:
+     githubRefreshApp:
+       acceptedGithubOrgs:
+         - lsst-sqre
+
+All of these items are required.
+
+``accepted_github_orgs``
+    A list of GitHub organizations from which this instance of Mobu will accept webhook requests.
+    Webhook requests from any orgs not in this list will get a ``403`` response.

docs/operations/index.rst

@@ -7,114 +7,13 @@ GitHub integration
 
 Each integration has as GitHub application created in the `lsst-sqre org <https://github.com/organizations/lsst-sqre/settings/apps>`__ for every environment in which it is enabled.
 
-All of the applications:
+All of the GitHub applications:
 
 * `mobu refresh (data-dev.lsst.cloud) <https://github.com/organizations/lsst-sqre/settings/apps/mobu-refresh-data-dev-lsst-cloud>`__
 * `mobu CI (data-dev.lsst.cloud) <https://github.com/organizations/lsst-sqre/settings/apps/mobu-ci-data-dev-lsst-cloud>`__
 
-GitHub application configuration
-================================
+.. toctree::
+   :maxdepth: 1
 
-To enable the GitHub integrations for another mobu env, you have to create a new GitHub application and sync Phalanx secrets.
-
-Refresh app
------------
-
-Create a new GitHub app
-~~~~~~~~~~~~~~~~~~~~~~~
-
-
-#. Click the ``New GitHub App`` button in the `lsst-sqre org Developer Settings apps page <https://github.com/organizations/lsst-sqre/settings/apps>`__.
-
-#. Name it ``mobu refresh (<env URL or id if the URL is too long>)``.
-
-#. Make sure the ``Active`` checkbox is checked in the ``Webhook`` section.
-
-#. Enter ``https://<env URL>/mobu/github/refresh/webhook`` in the ``Webhook URL`` input.
-#. Generate a strong password to use as the webhook secret.
-#. Store this in the ``SQuaRE`` vault in the ``LSST IT`` 1Password account in an item named ``mobu GitHub refresh app webhook secret (<env URL>)``.
-#. Get this into the Phalanx secret store for that env under the key: ``github-refresh-app-webhook-secret`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
-#. Enter this secret in the ``Webhook secret (optional)`` box in the GitHub App config.
-#. Select ``Read-only`` in the dropdown of the ``Contents`` access category in the ``Repository Permissions`` section.
-#. Check the ``Push`` checkbox in the ``Subscribe to events`` section.
-#. Select the ``Any account`` radio button in the ``Where can this GitHub App be installed?`` section.
-#. Click the ``Create GitHub App`` button.
-#. Do the `Phalanx configuration <#phalanx-configuration>`__.
-
-Install the app for a repo
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-#. Go to new app’s homepage (something like https://github.com/apps/mobu-refresh-usdfdev).
-#. Click the ``Install`` button.
-#. Select the ``Only select repositories`` radio button.
-#. Select the repo in the dropdown.
-#. Click ``Install``.
-
-CI app
-------
-
-Create a new GitHub app
-~~~~~~~~~~~~~~~~~~~~~~~
-
-#. Click the ``New GitHub App`` button in the `lsst-sqre org Developer Settings apps page <https://github.com/organizations/lsst-sqre/settings/apps>`__.
-#. Name it ``mobu CI (<env URL or id if the URL is too long>)``.
-#. Make sure the ``Active`` checkbox is checked in the ``Webhook`` section.
-#. Enter ``https://<env URL>/mobu/github/ci/webhook`` in the ``Webhook URL`` input.
-#. Generate a strong password to use as the webhook secret.
-#. Store this in the ``SQuaRE`` vault in the ``LSST IT`` 1Password account in an item named ``mobu GitHub CI app webhook secret (<env URL>)``.
-#. Get this into the Phalanx secret store for that env under the key: ``github-ci-app-webhook-secret`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
-#. Enter this secret in the ``Webhook secret (optional)`` box in the GitHub App config.
-#. Select ``Read and Write`` in the dropdown of the ``Checks`` access category in the ``Repository Permissions`` section.
-#. Select ``Read-only`` in the dropdown of the ``Contents`` access category in the ``Repository Permissions`` section.
-#. Check the ``Check suite`` and ``Check run`` checkboxes in the ``Subscribe to events`` section.
-#. Select the ``Any account`` radio button in the ``Where can this GitHub App be installed?`` section.
-#. Click the ``Create GitHub App`` button.
-#. Find the ``App ID`` (an integer) in the ``About`` section. Get this into the Phalanx secret store for that env under the key: ``github-ci-app-id`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
-#. Click the ``Generate a private key`` button in the ``Private keys`` section.
-#. Store this private key in the ``SQuaRE`` vault in the ``LSST IT`` 1Password account in an item named ``mobu GitHub CI app private key (<env URL>)``.
-#. Get this into the Phalanx secret store for that env under the key: ``github-ci-app-private-key`` (`this process <https://phalanx.lsst.io/admin/add-new-secret.html>`__ is different for different envs).
-#. Do the `Phalanx configuration <#phalanx-configuration>`__.
-
-Install the app for a repo
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-#. Go to new app’s homepage (something like https://github.com/apps/mobu-refresh-usdfdev).
-#. Click the ``Install`` button.
-#. Select the ``Only select repositories`` radio button.
-#. Select the repo in the dropdown.
-#. Click ``Install``.
-
-Phalanx configuration
-=====================
-
-The GitHub integrations each need to be explicitly enabled in Phalanx for a given environment.
-If an integration is not enabled, then the webhook route for that integration will not be mounted, GitHub webhook requests will get ``404`` responses.
-To enable these integrations for an environment, set these values to ``true``:
-
-* ``config.githubRefreshAppEnabled``
-* ``config.githubCiAppEnabled``
-
-If you want to enable either GitHub integration in a given environment, you also need to add a ``config.github`` section to that env’s values in Mobu.
-That needs to be a dict with at ``users`` and ``accepted_github_orgs`` entries.
-It should look something like this:
-
-.. code:: yaml
-
-   config:
-     github:
-       accepted_github_orgs:
-         - lsst-sqre
-       users:
-         - username: "bot-mobu-ci-user-1"
-           uidnumber: 123
-           gidnumber: 456
-         - username: "bot-mobu-ci-user-2"
-           uidnumber: 789
-           gidnumber: 876
-
-The organization of any repo that uses any of the GitHub integrations in an env must be added to the ``accepted_github_orgs`` list, otherwise Github webhook requests will get ``403`` responses.
-
-The ``users`` list follows the same rules as the ``users`` list in a flock autostart config.
-The usernames must all start with ``bot-mobu``.
-In envs with Firestore integration, you only need to specify ``username``.
-In envs without it, you need to ensure that users are manually provisioned, and then you need all three of ``username``, ``uidnumber``, and ``gidnumber``.
+   github_ci_app
+   github_refresh_app

pyproject.toml

@@ -165,6 +165,7 @@ ignore = [
     "TRY003",  # good general advice but lint is way too aggressive
     "TRY301",  # sometimes raising exceptions inside try is the best flow
     "UP040",   # Python 3.12 supports `type` alias kw, but mypy doesn't yet
+    "S113",    # httpx enforces timeouts everywhere, 5s by default
 
     # The following settings should be disabled when using ruff format
     # per https://docs.astral.sh/ruff/formatter/#conflicting-lint-rules