DM-48101: Sentry for notifications and timings

* Replace Safir `SlackException` functionality with Sentry error
reporting.
* Replace `timings` functionality with Sentry tracing.

The biggest conceptual change is that instead of putting error details
on the exception objects themselves, they get put into the
global-per-business-execution Sentry scope, and whatever is in the scope
is sent to Sentry when an error occurs (except for a very few cases
where we still attach info directly to exceptions).

This instruments custom scopes and transactions and fingerprints, which
are described in the new `sentry.rst` file in the docs.

Finally, this uses some more general Sentry helpers proposed for Safir
in lsst-sqre/safir#366.

docs/development/index.rst

@@ -8,3 +8,4 @@ Here are some guides on developing and testing mobu.
 
    idfdev
    github
+   sentry

docs/development/sentry.rst

@@ -0,0 +1,35 @@
+##################
+Sentry Integration
+##################
+
+Mobu integrates with sentry differently than most other apps, because the unit of work isn't a request like in a web app, it's an iteration (or part of an iteration) of a business.
+As a result, some things are different about this Sentry integration.
+
+Scopes and Transactions
+=======================
+
+`Isolation scopes <https://docs.sentry.io/platforms/python/enriching-events/scopes/>`_ and `transactions <https://docs.sentry.io/platforms/python/tracing/instrumentation/custom-instrumentation/#add-a-transaction>`_ are created manually.
+All tags and contexts are set on the isolation scope, and any exceptions are manually captured in the monkey runner loop.
+
+* There is one isolation scope for every execution of a business's ``run`` method.
+* There is one transaction for every execution of a business's ``run`` method.
+* For ``Nublado`` businesses, there are transactions created for startup and shutdown.
+* For ``NotebookRunner`` businesses, a transaction is created for every notebook execution.
+
+The ``Nublado`` and ``NotebookRunner`` business transactions will appear in Sentry as separate transactions.
+The details of these transactions will not appear in the ``run`` transaction.
+We have these separate inner transactions because Sentry waits until all spans are closed in a transaction before sending it, and we don't want to wait for ``max_executions`` notebook executions before tracing information for a single notebook executions is sent.
+
+Fingerprints
+============
+
+We add tags to the fingerprint of every event sent to Sentry to force the creation of separate issues where only one issue would be created by default.
+For example, we add notebook and cell ids to the fingerprint to force different issues (and thus different notifications) for errors from different notebooks and cells.
+
+Traces Sampling
+===============
+
+The ``sentry_traces_sample_config``/``sentryTracesSampleConfig`` option can be set to:
+
+* A float, in which case it will be passed directly to `traces_sample_rate <https://docs.sentry.io/platforms/python/configuration/options/#traces-sample-rate>`_ when initializing Sentry, and only send that percentage of transactions to sentry
+* The string "errors", in which case transactions will be sent to Sentry only if errors occurred during them

docs/index.rst

@@ -17,7 +17,7 @@ Mobu
 ####
 
 mobu (short for "monkey business") is a continous integration testing framework for the `Rubin Science Platform <https://phalanx.lsst.io/>`__ .
-It attempts to simulate user interactions with Science Platform services continuously, recording successes and failures and optionally reporting failures to a Slack channel via a Slack incoming webhook.
+It attempts to simulate user interactions with Science Platform services continuously, recording successes and failures and reporting failures to `Sentry <https://sentry.io>`_.
 It runs some number of "monkeys" that simulate a random user of the Science Platform.
 Those monkeys are organized into "flocks" that share a single configuration across all of the monkeys.
 It can be used for both monitoring and synthetic load testing.

src/mobu/config.py

@@ -4,7 +4,7 @@
 
 from pathlib import Path
 from textwrap import dedent
-from typing import Self
+from typing import Literal, Self
 
 import yaml
 from pydantic import AliasChoices, Field, HttpUrl
@@ -182,6 +182,42 @@ class Configuration(BaseSettings):
         ),
     )
 
+    sentry_dsn: str | None = Field(
+        None,
+        title="Sentry DSN",
+        description="The Sentry DSN: https://docs.sentry.io/platforms/python/#configure",
+        examples=[
+            "https://[email protected]/123456",
+        ],
+        validation_alias=AliasChoices("MOBU_SENTRY_DSN", "mobuSentryDsn"),
+    )
+
+    sentry_traces_sample_config: float | Literal["errors"] = Field(
+        0,
+        title="Sentry traces sample config",
+        description=(
+            "Set the Sentry sampling strategy for traces. If this is a float,"
+            " it will be passed as the traces_sample_rate: https://docs.sentry.io/platforms/python/configuration/sampling/#configuring-the-transaction-sample-rate"
+            ' If this is set to "errors", then all transactions during which'
+            " an error occurred will be sent."
+        ),
+        examples=[0, 0.5, "errors"],
+        validation_alias=AliasChoices(
+            "MOBU_SENTRY_TRACES_SAMPLE_CONFIG", "sentryTracesSampleConfig"
+        ),
+    )
+
+    sentry_environment: str = Field(
+        ...,
+        title="Sentry environment",
+        description=(
+            "The Sentry environment: https://docs.sentry.io/concepts/key-terms/environments/"
+        ),
+        validation_alias=AliasChoices(
+            "MOBU_SENTRY_ENVIRONMENT", "sentryEnvironment"
+        ),
+    )
+
     gafaelfawr_token: str | None = Field(
         None,
         title="Gafaelfawr admin token",

src/mobu/constants.py

@@ -49,3 +49,6 @@
     "^[a-z0-9](?:[a-z0-9]|-[a-z0-9])*[a-z](?:[a-z0-9]|-[a-z0-9])*$"
 )
 """Regex matching all valid usernames."""
+
+SENTRY_ERRORED_KEY = "errored"
+"""Tag name to set on transactions that had exceptions."""