Switch to towncrier (#3231)

Co-authored-by: Ilan Gold <[email protected]>

.gitignore

@@ -18,7 +18,6 @@
 /tests/**/*failed-diff.png
 
 # Environment management
-/hatch.toml
 /Pipfile
 /Pipfile.lock
 /requirements*.lock

.readthedocs.yml

@@ -2,9 +2,16 @@ version: 2
 submodules:
   include: all
 build:
-  os: ubuntu-20.04
+  os: ubuntu-24.04
   tools:
     python: '3.12'
+  jobs:
+    post_checkout:
+      # unshallow so version can be derived from tag
+      - git fetch --unshallow || true
+    pre_build:
+      # run towncrier to preview the next version’s release notes
+      - ( find docs/release-notes -regex '[^.]+[.][^.]+.md' | grep -q . ) && towncrier build --keep || true
 sphinx:
   fail_on_warning: true # do not change or you will be fired
   configuration: docs/conf.py

ci/scripts/min-deps.py

@@ -1,4 +1,4 @@
-#!python3
+#!/usr/bin/env python3
 from __future__ import annotations
 
 import argparse

ci/scripts/towncrier_automation.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import subprocess
+from typing import TYPE_CHECKING
+
+from packaging.version import Version
+
+if TYPE_CHECKING:
+    from collections.abc import Sequence
+
+
+class Args(argparse.Namespace):
+    version: str
+    dry_run: bool
+
+
+def parse_args(argv: Sequence[str] | None = None) -> Args:
+    parser = argparse.ArgumentParser(
+        prog="towncrier-automation",
+        description=(
+            "This script runs towncrier for a given version, "
+            "creates a branch off of the current one, "
+            "and then creates a PR into the original branch with the changes. "
+            "The PR will be backported to main if the current branch is not main."
+        ),
+    )
+    parser.add_argument(
+        "version",
+        type=str,
+        help=(
+            "The new version for the release must have at least three parts, like `major.minor.patch` and no `major.minor`. "
+            "It can have a suffix like `major.minor.patch.dev0` or `major.minor.0rc1`."
+        ),
+    )
+    parser.add_argument(
+        "--dry-run",
+        help="Whether or not to dry-run the actual creation of the pull request",
+        action="store_true",
+    )
+    args = parser.parse_args(argv, Args())
+    # validate the version
+    if len(Version(args.version).release) != 3:
+        msg = f"Version argument {args.version} must contain major, minor, and patch version."
+        raise ValueError(msg)
+    return args
+
+
+def main(argv: Sequence[str] | None = None) -> None:
+    args = parse_args(argv)
+
+    # Run towncrier
+    subprocess.run(
+        ["towncrier", "build", f"--version={args.version}", "--yes"], check=True
+    )
+
+    # Check if we are on the main branch to know if we need to backport
+    base_branch = subprocess.run(
+        ["git", "rev-parse", "--abbrev-ref", "HEAD"],
+        capture_output=True,
+        text=True,
+        check=True,
+    ).stdout.strip()
+    pr_description = (
+        "" if base_branch == "main" else "@meeseeksmachine backport to main"
+    )
+    branch_name = f"release_notes_{args.version}"
+
+    # Create a new branch + commit
+    subprocess.run(["git", "switch", "-c", branch_name], check=True)
+    subprocess.run(["git", "add", "docs/release-notes"], check=True)
+    pr_title = f"(chore): generate {args.version} release notes"
+    subprocess.run(["git", "commit", "-m", pr_title], check=True)
+
+    # push
+    if not args.dry_run:
+        subprocess.run(
+            ["git", "push", "--set-upstream=origin", branch_name], check=True
+        )
+    else:
+        print("Dry run, not pushing")
+
+    # Create a PR
+    subprocess.run(
+        [
+            "gh",
+            "pr",
+            "create",
+            f"--base={base_branch}",
+            f"--title={pr_title}",
+            f"--body={pr_description}",
+            *(["--label=no milestone"] if base_branch == "main" else []),
+            *(["--dry-run"] if args.dry_run else []),
+        ],
+        check=True,
+    )
+
+    # Enable auto-merge
+    if not args.dry_run:
+        subprocess.run(
+            ["gh", "pr", "merge", branch_name, "--auto", "--squash"], check=True
+        )
+    else:
+        print("Dry run, not merging")
+
+
+if __name__ == "__main__":
+    main()

docs/conf.py

@@ -79,6 +79,7 @@
     "scanpydoc",  # needs to be before sphinx.ext.linkcode
     "sphinx.ext.linkcode",
     "sphinx_design",
+    "sphinx_tabs.tabs",
     "sphinx_search.extension",
     "sphinxext.opengraph",
     *[p.stem for p in (HERE / "extensions").glob("*.py") if p.stem not in {"git_ref"}],

docs/dev/code.md

@@ -12,13 +12,12 @@
 
 ## Code style
 
-New code should follow
-[Black](https://black.readthedocs.io/en/stable/the_black_code_style.html)
-and
-[flake8](https://flake8.pycqa.org).
-We ignore a couple of flake8 checks which are documented in the .flake8 file in the root of this repository.
-To learn how to ignore checks per line please read
-[flake8 violations](https://flake8.pycqa.org/en/latest/user/violations.html).
-Additionally, we use Scanpy’s
-[EditorConfig](https://github.com/scverse/scanpy/blob/main/.editorconfig),
+Code contributions will be formatted and style checked using [Ruff][].
+Ignored checks are configured in the `tool.ruff.lint` section of {file}`pyproject.toml`.
+To learn how to ignore checks per line please read about [ignoring errors][].
+Additionally, we use Scanpy’s [EditorConfig][],
 so using an editor/IDE with support for both is helpful.
+
+[Ruff]: https://docs.astral.sh/ruff/
+[ignoring errors]: https://docs.astral.sh/ruff/tutorial/#ignoring-errors
+[EditorConfig]: https://github.com/scverse/scanpy/blob/main/.editorconfig

docs/dev/documentation.md

@@ -4,38 +4,37 @@
 
 ## Building the docs
 
-Dependencies for building the documentation for scanpy can be installed with `pip install -e "scanpy[doc]"`
-
-To build the docs, enter the `docs` directory and run `make html`. After this process completes you can take a look at the docs by opening `scanpy/docs/_build/html/index.html`.
+To build the docs, run `hatch run docs:build`.
+Afterwards, you can run `hatch run docs:open` to open {file}`docs/_build/html/index.html`.
 
 Your browser and Sphinx cache docs which have been built previously.
 Sometimes these caches are not invalidated when you've updated the docs.
 If docs are not updating the way you expect, first try "force reloading" your browser page – e.g. reload the page without using the cache.
-Next, if problems persist, clear the sphinx cache and try building them again (`make clean` from `docs` directory).
-
-```{note}
-If you've cloned the repository pre 1.8.0, you may need to be more thorough in cleaning.
-If you run into warnings try removing all untracked files in the docs directory.
-```
+Next, if problems persist, clear the sphinx cache (`hatch run docs:clean`) and try building them again.
 
 ## Adding to the docs
 
-For any user-visible changes, please make sure a note has been added to the release notes for the relevant version so we can credit you!
-These files are found in the `docs/release-notes/` directory.
+For any user-visible changes, please make sure a note has been added to the release notes using [`hatch run towncrier:create`][towncrier create].
 We recommend waiting on this until your PR is close to done since this can often causes merge conflicts.
 
 Once you've added a new function to the documentation, you'll need to make sure there is a link somewhere in the documentation site pointing to it.
 This should be added to `docs/api.md` under a relevant heading.
 
-For tutorials and more in depth examples, consider adding a notebook to [scanpy-tutorials](https://github.com/scverse/scanpy-tutorials/).
+For tutorials and more in depth examples, consider adding a notebook to the [scanpy-tutorials][] repository.
 
-The tutorials are tied to this repository via a submodule. To update the submodule, run `git submodule update --remote` from the root of the repository. Subsequently, commit and push the changes in a PR. This should be done before each release to ensure the tutorials are up to date.
+The tutorials are tied to this repository via a submodule.
+To update the submodule, run `git submodule update --remote` from the root of the repository.
+Subsequently, commit and push the changes in a PR.
+This should be done before each release to ensure the tutorials are up to date.
+
+[towncrier create]: https://towncrier.readthedocs.io/en/stable/tutorial.html#creating-news-fragments
+[scanpy-tutorials]: https://github.com/scverse/scanpy-tutorials/
 
 ## docstrings format
 
 We use the numpydoc style for writing docstrings.
-We'd primarily suggest looking at existing docstrings for examples, but the [napolean guide to numpy style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy) is also a great source.
-If you're unfamiliar with the reStructuredText (`rst`) markup format, [Sphinx has a useful primer](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html).
+We'd primarily suggest looking at existing docstrings for examples, but the [napolean guide to numpy style docstrings][] is also a great source.
+If you're unfamiliar with the reStructuredText (rST) markup format, check out the [Sphinx rST primer][].
 
 Some key points:
 
@@ -46,26 +45,34 @@ Some key points:
 
 Look at [sc.tl.louvain](https://github.com/scverse/scanpy/blob/a811fee0ef44fcaecbde0cad6336336bce649484/scanpy/tools/_louvain.py#L22-L90) as an example for everything mentioned here.
 
+[napolean guide to numpy style docstrings]: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy
+[sphinx rst primer]: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+
 ### Plots in docstrings
 
 One of the most useful things you can include in a docstring is examples of how the function should be used.
 These are a great way to demonstrate intended usage and give users a template they can copy and modify.
-We're able to include the plots produced by these snippets in the rendered docs using [matplotlib's plot directive](https://matplotlib.org/devel/plot_directive.html).
+We're able to include the plots produced by these snippets in the rendered docs using [matplotlib's plot directive][].
 For examples of this, see the `Examples` sections of {func}`~scanpy.pl.dotplot` or {func}`~scanpy.pp.calculate_qc_metrics`.
 
 Note that anything in these sections will need to be run when the docs are built, so please keep them computationally light.
 
 - If you need computed features (e.g. an embedding, differential expression results) load data that has this precomputed.
 - Try to re-use datasets, this reduces the amount of data that needs to be downloaded to the CI server.
 
+[matplotlib's plot directive]: https://matplotlib.org/devel/plot_directive.html
+
 ### `Params` section
 
 The `Params` abbreviation is a legit replacement for `Parameters`.
 
 To document parameter types use type annotations on function parameters.
 These will automatically populate the docstrings on import, and when the documentation is built.
 
-Use the python standard library types (defined in [collections.abc](https://docs.python.org/3/library/collections.abc.html) and [typing](https://docs.python.org/3/library/typing.html) modules) for containers, e.g. `Sequence`s (like `list`), `Iterable`s (like `set`), and `Mapping`s (like `dict`).
+Use the python standard library types (defined in {mod}`collections.abc` and {mod}`typing` modules) for containers, e.g.
+{class}`~collections.abc.Sequence`s (like `list`),
+{class}`~collections.abc.Iterable`s (like `set`), and
+{class}`~collections.abc.Mapping`s (like `dict`).
 Always specify what these contain, e.g. `{'a': (1, 2)}` → `Mapping[str, Tuple[int, int]]`.
 If you can’t use one of those, use a concrete class like `AnnData`.
 If your parameter only accepts an enumeration of strings, specify them like so: `Literal['elem-1', 'elem-2']`.
@@ -80,8 +87,7 @@ There are three types of return sections – prose, tuple, and a mix of both.
 
 #### Examples
 
-For simple cases, use prose as in
-{func}`~scanpy.pp.normalize_total`
+For simple cases, use prose as in {func}`~scanpy.pp.normalize_total`:
 
 ```rst
 Returns
@@ -110,7 +116,7 @@ def myfunc(...) -> tuple[int, str]:
 ```
 
 Many functions also just modify parts of the passed AnnData object, like e.g. {func}`~scanpy.tl.dpt`.
-You can then combine prose and lists to best describe what happens.
+You can then combine prose and lists to best describe what happens:
 
 ```rst
 Returns