Merge pull request #70 from flyingcircusio/PL-132755-refactor-module-…

…and-subcommand-structure

Pl 132755 refactor module and subcommand structure

ARCHITECTURE.md

@@ -0,0 +1,48 @@
+
+# Daemon
+
+Backy has a daemon that is responsible to:
+
+1. Schedule jobs in a timely manner according to their configuration.
+
+2. Provide an HTTP API to allow multiple backy servers to interact with each
+   other.
+
+There is a (PostgreSQL) database to store metadata about backups that both
+the daemon, the CLI (including the sources) interact with.
+
+# CLI
+
+The are two levels of CLI interactions:
+
+1. The main `backy` command provides administrators interaction capabilities
+   with the backy environment on a server to retrieve status information,
+   run backups, restore data and some maintenance tasks.
+
+2. Backy itself interacts with sources through a second layer of CLI commands,
+   specific to each source. They are called by the higher level CLI as well as
+   from the daemon. We use this layering to allow implementing sources in
+   different languages.
+
+The CLI ideally does not interact with the daemon directly, but by inspecting
+or updating the database.
+
+# Nomenclature
+
+Words within the context of backup software are a bit muddy, specifically
+the meaning of "a backup". We decided to take inspiration from the git dictionary
+and use it the following way:
+
+1. A **repository** is - similar to git - the logical container for the user
+   data relevant to one thing that we are backing up.
+
+2. A **source** provides the data that should be backed up. Different kinds
+   of sources can model arbitrary data models: backy does not care whether
+   you are backing up virtual disk images or S3 buckets.
+
+3. A **revision** specifies the state of the source at a certain point in time
+   and corresponds to what would be colloquially called "a backup".
+
+4. The daemon uses a **job** for every repository to execute the steps necessary
+   to perform regular backups with all surrounding management tasks like
+   garbage collection, verification, etc.

changelog.d/20240812_133802_ct_PL_132755_refactor_module_and_subcommand_structure.rst

@@ -0,0 +1,5 @@
+.. A new scriv changelog fragment.
+
+- Refactor the overall structure to prepare for more diverse sources:
+  use a sub-CLI pattern to talk to source implementations and clean up
+  the Ceph source, removing unused Code. (PL-132755)

lib.nix

@@ -127,7 +127,8 @@ in
 
   devShells = {
     default = mkShellNoCC {
-      BACKY_CMD = "${poetryEnv}/bin/backy";
+      BACKY_CLI_CMD = "${poetryEnv}/bin/backy";
+      BACKY_RBD_CMD = "${poetryEnv}/bin/backy-rbd";
       packages = [
         poetryEnv
         poetry

pyproject.toml

@@ -75,5 +75,24 @@ zest-releaser = "^9.1.1"
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
 
+[tool.poetry.plugins.'backy.sources']
+file = 'backy.file:FileSource'
+rbd = 'backy.rbd:RBDSource'
+s3 = 'backy.s3:S3Source'
+
 [tool.poetry.scripts]
-backy = "backy.main:main"
+backy = "backy.cli:main"
+backyd = "backy.daemon:main"
+backy-rbd = "backy.rbd:main"
+backy-s3 = "backy.s3:main"
+backy-file = "backy.file:main"
+
+[[tool.mypy.overrides]]
+module = "backy.*"
+check_untyped_defs = true
+
+[tool.zest-releaser]
+prereleaser.before = [ "release_helper.ignore_history_file" ]
+prereleaser.middle = [ "release_helper.update_poetry_version release_helper.scriv_collect" ]
+postreleaser.before = [ "release_helper.ignore_history_file" ]
+postreleaser.middle = [ "release_helper.update_poetry_version" ]