0.0.3 release

.gitignore

@@ -6,5 +6,6 @@ pip-wheel-metadata/
 .ipynb_checkpoints/
 *.ipynb
 notes.md
-*.yaml
+config.yaml
 dist/
+build/

CHANGES.md

@@ -1,5 +1,12 @@
 # Changes
 
+## 0.0.3 (2020-07-19)
+
+- FEATURE: Added wizard GUI for backup tasks (`snap`, `backup`, `cleanup`)
+- FEATURE: Added new configuration options (`always_changed`, `written_threshold`, `check_diff`) for detecting snapshot tasks
+- FEATURE: CLI and GUI translations (i18n)
+- FIX: Added missing type checks
+
 ## 0.0.2 (2020-07-14)
 
 - FEATURE: New, fully object oriented base library

MANIFEST.in

@@ -0,0 +1 @@
+recursive-include src/abgleich/share/ *.*

README.md

@@ -2,23 +2,33 @@
 
 ## SYNOPSIS
 
-`abgleich` is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output.
+`abgleich` is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output. It also includes a GUI.
+
+## CLI EXAMPLE
 
 ![demo](https://github.com/pleiszenburg/abgleich/blob/master/docs/demo.png?raw=true "demo")
 
+## GUI EXAMPLE
+
+| snap | backup | cleanup |
+|:----:|:------:|:-------:|
+| ![snap](https://github.com/pleiszenburg/abgleich/blob/master/docs/demo_gui01.png?raw=true "snap") | ![backup](https://github.com/pleiszenburg/abgleich/blob/master/docs/demo_gui02.png?raw=true "backup") | ![cleanup](https://github.com/pleiszenburg/abgleich/blob/master/docs/demo_gui03.png?raw=true "cleanup") |
+
 ## INSTALLATION
 
+The base CLI tool can be installed as follows:
+
 ```bash
 pip install -vU abgleich
 ```
 
-or
+An installation also including a GUI can be triggered by running:
 
 ```bash
-pip install -vU git+https://github.com/pleiszenburg/abgleich.git@master
+pip install -vU abgleich[gui]
 ```
 
-Requires [CPython](https://en.wikipedia.org/wiki/CPython) 3.6 or later, a [Unix shell](https://en.wikipedia.org/wiki/Unix_shell) and [ssh](https://en.wikipedia.org/wiki/Secure_Shell). Tested with [OpenZFS](https://en.wikipedia.org/wiki/OpenZFS) 0.8.x on Linux.
+Requires [CPython](https://en.wikipedia.org/wiki/CPython) 3.6 or later, a [Unix shell](https://en.wikipedia.org/wiki/Unix_shell) and [ssh](https://en.wikipedia.org/wiki/Secure_Shell). GUI support requires [Qt5](https://en.wikipedia.org/wiki/Qt_(software)) in addition. Tested with [OpenZFS](https://en.wikipedia.org/wiki/OpenZFS) 0.8.x on Linux.
 
 `abgleich`, CPython and the Unix shell must only be installed on one of the involved systems. Any remote system will be contacted via ssh and provided with direct ZFS commands.
 
@@ -55,6 +65,9 @@ target:
     host: bigdata
     user: zfsadmin
 keep_snapshots: 2
+always_changed: no
+written_threshold: 1048576
+check_diff: yes
 suffix: _backup
 digits: 2
 ignore:
@@ -65,7 +78,7 @@ ssh:
     cipher: [email protected]
 ```
 
-The prefix can be empty on either side. If a `host` is set to `localhost`, the `user` field can be left empty. Both source and target can be remote hosts or localhost at the same time. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `suffix` contains the name suffix for new snapshots. `digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups. `ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends.
+The prefix can be empty on either side. If a `host` is set to `localhost`, the `user` field can be left empty. Both source and target can be remote hosts or localhost at the same time. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `suffix` contains the name suffix for new snapshots. Setting `always_changed` to `yes` causes `abgleich` to beliefe that all datasets have always changed since the last snapshot, completely ignoring what ZFS actually reports. No diff will be checked produced for values of `written` lower than `written_threshold`. Checking diffs can be completely deactivated by setting `check_diff` to `no`. `digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups. `ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends.
 
 ## USAGE
 
@@ -91,6 +104,10 @@ Send (new) datasets and new snapshots from source to target.
 
 Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at least `keep_snapshots` number of snapshots on source side.
 
+### `abgleich wizard config.yaml`
+
+Runs a sequence of `snap`, `backup` and `cleanup` in a wizard GUI. This command is only available if `abgleich` was installed with GUI support.
+
 ## SPEED
 
 `abgleich` uses Python's [type hints](https://docs.python.org/3/library/typing.html) and enforces them with [typeguard](https://github.com/agronholm/typeguard) at runtime. It furthermore makes countless assertions.

makefile

@@ -1,4 +1,7 @@
 
+black:
+	black .
+
 clean:
 	-rm -r build/*
 	find src/ -name '*.pyc' -exec sudo rm -f {} +
@@ -14,14 +17,14 @@ clean:
 
 release:
 	make clean
-	# python setup.py sdist bdist_wheel
+	python setup.py sdist bdist_wheel
 	python setup.py sdist
-	# gpg --detach-sign -a dist/abgleich*.whl
+	gpg --detach-sign -a dist/abgleich*.whl
 	gpg --detach-sign -a dist/abgleich*.tar.gz
 
 install:
 	pip install -vU pip setuptools
-	pip install -v -e .[dev]
+	pip install -v -e .[all]
 
 upload:
 	for filename in $$(ls dist/*.tar.gz dist/*.whl) ; do \

pyproject.toml

@@ -1,2 +1,22 @@
 [build-system]
 requires = ["setuptools", "wheel"]
+
+[tool.black]
+target-version = ['py36']
+include = '\.pyi?$'
+exclude = '''
+/(
+    \.eggs
+  | \.git
+  | \.hg
+  | \.mypy_cache
+  | \.tox
+  | \.venv
+  | _build
+  | buck-out
+  | build
+  | dist
+  | env??
+  | env
+)/
+'''

setup.py

@@ -6,9 +6,9 @@
 zfs sync tool
 https://github.com/pleiszenburg/abgleich
 
-	setup.py: Used for package distribution
+    setup.py: Used for package distribution
 
-	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -40,7 +40,7 @@
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 # Package version
-__version__ = "0.0.2"
+__version__ = "0.0.3"
 
 # List all versions of Python which are supported
 python_minor_min = 6
@@ -57,6 +57,15 @@
 # Define source directory (path)
 SRC_DIR = "src"
 
+# Requirements
+extras_require = {
+    "dev": ["black", "python-language-server[all]", "setuptools", "twine", "wheel",],
+    "gui": ["pyqt5",],
+}
+extras_require["all"] = list(
+    {rq for target in extras_require.keys() for rq in extras_require[target]}
+)
+
 # Install package
 setup(
     name="abgleich",
@@ -78,20 +87,13 @@
     python_requires=">=3.{MINOR:d}".format(MINOR=python_minor_min),
     setup_requires=[],
     install_requires=["click", "tabulate", "pyyaml", "typeguard",],
-    extras_require={
-        "dev": [
-            "black",
-            "python-language-server[all]",
-            "setuptools",
-            "twine",
-            "wheel",
-        ]
-    },
+    extras_require=extras_require,
     zip_safe=False,
     entry_points={"console_scripts": ["abgleich = abgleich.cli:cli",],},
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Environment :: Console",
+        "Environment :: X11 Applications",
         "Intended Audience :: Developers",
         "Intended Audience :: Education",
         "Intended Audience :: Information Technology",

src/abgleich/cli/_main_.py

@@ -6,9 +6,9 @@
 zfs sync tool
 https://github.com/pleiszenburg/abgleich
 
-	src/abgleich/cli/_main_.py: CLI auto-detection
+    src/abgleich/cli/_main_.py: CLI auto-detection
 
-	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -46,7 +46,12 @@ def _add_commands(ctx):
         for item in os.listdir(os.path.dirname(__file__))
         if not item.startswith("_")
     ):
-        ctx.add_command(getattr(importlib.import_module("abgleich.cli.%s" % cmd), cmd))
+        try:
+            ctx.add_command(
+                getattr(importlib.import_module("abgleich.cli.%s" % cmd), cmd)
+            )
+        except ModuleNotFoundError:  # likely no gui support
+            continue
 
 
 @click.group()

src/abgleich/cli/backup.py

@@ -32,6 +32,7 @@
 import click
 
 from ..core.config import Config
+from ..core.i18n import t
 from ..core.zpool import Zpool
 
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -51,10 +52,10 @@ def backup(configfile):
     transactions = source_zpool.get_backup_transactions(target_zpool)
 
     if len(transactions) == 0:
-        print("nothing to do")
+        print(t("nothing to do"))
         return
     transactions.print_table()
 
-    click.confirm("Do you want to continue?", abort=True)
+    click.confirm(t("Do you want to continue?"), abort=True)
 
     transactions.run()

src/abgleich/cli/cleanup.py

@@ -34,6 +34,7 @@
 import click
 
 from ..core.config import Config
+from ..core.i18n import t
 from ..core.io import humanize_size
 from ..core.zpool import Zpool
 
@@ -55,11 +56,11 @@ def cleanup(configfile):
     transactions = source_zpool.get_cleanup_transactions(target_zpool)
 
     if len(transactions) == 0:
-        print("nothing to do")
+        print(t("nothing to do"))
         return
     transactions.print_table()
 
-    click.confirm("Do you want to continue?", abort=True)
+    click.confirm(t("Do you want to continue?"), abort=True)
 
     transactions.run()
 

src/abgleich/cli/snap.py

@@ -32,6 +32,7 @@
 import click
 
 from ..core.config import Config
+from ..core.i18n import t
 from ..core.zpool import Zpool
 
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -47,10 +48,10 @@ def snap(configfile):
     transactions = zpool.get_snapshot_transactions()
 
     if len(transactions) == 0:
-        print("nothing to do")
+        print(t("nothing to do"))
         return
     transactions.print_table()
 
-    click.confirm("Do you want to continue?", abort=True)
+    click.confirm(t("Do you want to continue?"), abort=True)
 
     transactions.run()

src/abgleich/cli/wizard.py

@@ -0,0 +1,47 @@
+# -*- coding: utf-8 -*-
+
+"""
+
+ABGLEICH
+zfs sync tool
+https://github.com/pleiszenburg/abgleich
+
+    src/abgleich/cli/wizard.py: wizard command entry point
+
+    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+
+<LICENSE_BLOCK>
+The contents of this file are subject to the GNU Lesser General Public License
+Version 2.1 ("LGPL" or "License"). You may not use this file except in
+compliance with the License. You may obtain a copy of the License at
+https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt
+https://github.com/pleiszenburg/abgleich/blob/master/LICENSE
+
+Software distributed under the License is distributed on an "AS IS" basis,
+WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
+specific language governing rights and limitations under the License.
+</LICENSE_BLOCK>
+
+"""
+
+
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+# IMPORT
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+import click
+
+from ..core.config import Config
+from ..gui.lib import run_app
+from ..gui.wizard import WizardUi
+
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+# ROUTINES
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+
+@click.command(short_help="run wizard gui")
+@click.argument("configfile", type=click.File("r", encoding="utf-8"))
+def wizard(configfile):
+
+    run_app(WizardUi, config=Config.from_fd(configfile))

src/abgleich/core/abc.py

@@ -51,6 +51,10 @@ class ComparisonItemABC(abc.ABC):
     pass
 
 
+class ConfigABC(abc.ABC):
+    pass
+
+
 class DatasetABC(abc.ABC):
     pass
 

src/abgleich/core/command.py

@@ -48,7 +48,7 @@ def __init__(self, cmd: typing.List[str]):
 
     def __str__(self) -> str:
 
-        return " ".join(self._cmd)
+        return " ".join([item.replace(" ", "\\ ") for item in self._cmd])
 
     def run(self):
 
@@ -60,7 +60,7 @@ def run(self):
         output, errors = output.decode("utf-8"), errors.decode("utf-8")
 
         if not status or len(errors.strip()) > 0:
-            raise SystemError("command failed", self.cmd, output, errors)
+            raise SystemError("command failed", str(self), output, errors)
 
         return output, errors
 
@@ -93,7 +93,11 @@ def run_pipe(self, other: CommandABC):
             )
         ):
             raise SystemError(
-                "command pipe failed", self.cmd, other.cmd, errors_1, output_2, errors_2
+                "command pipe failed",
+                f"{str(self):s} | {str(other):s}",
+                errors_1,
+                output_2,
+                errors_2,
             )
 
         return errors_1, output_2, errors_2

src/abgleich/core/comparison.py

@@ -72,6 +72,10 @@ def __init__(
 
         self._a, self._b, self._merged = a, b, merged
 
+    def __len__(self) -> int:
+
+        return len(self._merged)
+
     @property
     def a(self) -> ComparisonParentTypes: