Version 1.4.0

.github/workflows/test.yml

@@ -19,11 +19,17 @@ jobs:
 
     runs-on: ${{ matrix.platform }}
     steps:
-      - name: Checkout project
+
+      - name: Checkout merge commit
         uses: actions/checkout@v2
+        if: github.event_name == 'pull_request_target'
         with:
           ref: 'refs/pull/${{ github.event.number }}/merge'
 
+      - name: Checkout head commit
+        uses: actions/checkout@v2
+        if: github.event_name != 'pull_request_target'
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
@@ -34,19 +40,20 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install --upgrade pytest
           python -m pip install --upgrade pytest-cov
+          python -m pip install --upgrade pytest-rerunfailures
           python -m pip install .
 
       - name: Test with pytest
         run: |
-          pytest --cov=maestral --cov-report=xml tests/offline
+          pytest  --reruns 5 --cov=maestral --cov-report=xml tests/offline
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v1
         with:
           file: ./coverage.xml
           flags: pytest
           env_vars: OS,PYTHON,TYPE
-          name: pytests
+          name: pytests -v
         env:
           OS: ${{ matrix.platform }}
           PYTHON: ${{ matrix.python-version }}
@@ -71,11 +78,16 @@ jobs:
 
     runs-on: ${{ matrix.platform }}
     steps:
-      - name: Checkout project
+      - name: Checkout merge commit
         uses: actions/checkout@v2
+        if: github.event_name == 'pull_request_target'
         with:
           ref: 'refs/pull/${{ github.event.number }}/merge'
 
+      - name: Checkout head commit
+        uses: actions/checkout@v2
+        if: github.event_name != 'pull_request_target'
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
@@ -86,6 +98,7 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install --upgrade pytest
           python -m pip install --upgrade pytest-cov
+          python -m pip install --upgrade pytest-rerunfailures
           python -m pip install .
 
       - name: Get short-lived Dropbox token
@@ -99,11 +112,11 @@ jobs:
               -d client_id=2jmbq42w7vof78h)
           token=$(echo $auth_result | python -c "import sys, json; print(json.load(sys.stdin)['access_token'])")
           echo "::add-mask::$token"
-          echo "DROPBOX_TOKEN=$token" >> $GITHUB_ENV
+          echo "DROPBOX_ACCESS_TOKEN=$token" >> $GITHUB_ENV
 
       - name: Test with pytest
         run: |
-          pytest --cov=maestral --cov-report=xml tests/linked
+          pytest -v --reruns 5 --cov=maestral --cov-report=xml tests/linked
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v1

CHANGELOG.md

@@ -1,6 +1,124 @@
+## v1.4.0
+
+This release brings significant extensions to the command line interface: It introduces
+commands to create and manage shared links, to compare older version of a file and print
+the diff output to the terminal, and commands for direct access to config values (note
+the warning below). It also adds optional one-way syncing, for instance to keep a mirror
+of a remote Dropbox folder while ignoring local changes.
+
+Several bugs have been fixed which could occur when resuming the sync activity after the
+connection had been lost while indexing a remote folder.
+
+Finally, this release removes automatic error reporting via Bugsnag. Please file any bug
+reports as issues on GitHub where it is possible to follow up.
+
+#### Added:
+
+* Added a command `maestral diff` to compare different versions of a text file. The
+  resulting diff is printed to the console. Credit goes to @OrangeFran.
+* Resurrected the command `maestral revs` to list previous versions (revisions) of a file.
+* Added a command group `maestral sharelink` to create and manage shared links.
+  Subcommands are:
+
+  * `create`: Create a shared link for a file or folder, optionally with password
+    protection and an expiry date on supported accounts (business and professional).
+  * `list`: List shared links, either for a specific file or folder or for all items
+    in your Dropbox.
+  * `revoke`: Revoke a shared link.
+
+* Added a command group `maestral config` to provide direct access to config values.
+  Subcommands are:
+
+  * `get`: Gets the config value for a key.
+  * `set`: Sets the config value for a key.
+
+  This provides access to previously inaccessible config values such as
+  `reindex_interval` or `max_cpu_percent`. Please refer to a Wiki for an overview of all
+  config values. Use the `set` command with caution: setting some config values may
+  leave the daemon in an inconsistent state (e.g., changing the location of the Dropbox
+  folder). Always use the equivalent command from the Settings group (e.g., `maestral
+  move-dir`).
+* Added the ability to disable a single sync direction, for instance to enable download
+  syncs only. This can be useful when you want to mirror a remote folder while ignoring
+  local changes or when syncing to a file system which does not support inotify. To use
+  this, set the respective config values for `upload` or `download` to False. Note that
+  conflict resolution remains unaffected. For instance, when an unsynced local change
+  would be overwritten by a remote change, the local file will be moved to a
+  "conflicting copy" first. However, the conflicting copy will not be uploaded.
+
+#### Changed:
+
+* Changes to indexing:
+
+  * Avoid scanning of objects matching an  `.mignore` pattern (file watches will still be
+    added however). This results in performance improvements during startup and resume.
+    A resulting behavioral change is that **maestral will remove files matching an
+    ignore pattern from Dropbox**. After this change it will be immaterial if an
+    `.mignore` pattern is added before or after having matching files in Dropbox.
+  * If Maestral is quit or interrupted during indexing, for instance due to connection
+    problems, it will later resume from the same position instead of restarting from the
+    beginning.
+  * Indexing will no longer skip excluded folders. This is necessary for the above
+    change.
+  * Defer periodic reindexing, typically carried out weekly, if the device is not
+    connected to an AC power supply. This prevents draining the battery when hashing
+    file contents.
+
+* Changes to CLI:
+
+  * Moved linking and unlinking to a new command group `maestral auth` with subcommands
+    `link`, `unlink` and `status`.
+  * Renamed the command `file-status` to `filestatus`.
+  * Added a `--yes, -Y` flag to the `unlink` to command to skip the confirmation prompt.
+  * Renamed the `configs` command to list config files to `config-files`.
+  * Added an option `--clean` to `config-files` to remove all stale config files (those
+    without a linked Dropbox account).
+
+* Improved the error message when the user is running out of inotify watches: Recommend
+  default values of `max_user_watches = 524288` and `max_user_instances = 1024` or
+  double the current values, whichever is higher. Advise to apply the changes with
+  `sysctl -p`.
+
+#### Fixed:
+
+* Fixes an issue with the CLI on Python 3.6 where commands that print dates to the console
+  would raise an exception.
+* Properly handle a rare OSError "[Errno 41] Protocol wrong type for socket" on macOS,
+  see https://bugs.python.org/issue33450.
+* Allow creating local files even if we cannot set their permissions, for instances on
+  some mounted NTFS drives.
+* Fixes an issue with the selective sync dialog in the Qt / Linux GUI where the "Update"
+  button could be incorrectly enabled or disabled.
+* Fixes an issue where a lost internet connection while starting the sync could lead to
+  a stuck sync thread or an endless indexing cycle.
+* Fixes an issue where a lost internet connection during the download of a folder newly
+  included in selective sync could result in the download never being completed.
+* Fixes an issue where pausing the sync during the download of a folder newly included
+  in selective sync could result in the download never being completed.
+
+#### Removed:
+
+* Removed automatic error reporting via bugsnag.
+* Removed from CLI:
+
+  * The `maestral restart` command. Use `stop` and `start` instead.
+  * The `maestral account-info` command. Use `maestral auth status` instead.
+å
+* Removed the public API methods `Maestral.resume_sync` and `Maestral.pause_sync`. Use
+  `Maestral.start_sync` and `Maestral.stop_sync` instead.
+
+#### Dependencies:
+
+* Bumped survey to version >=3.2.2,<4.0.
+* Bumped keyring to version >=22.
+* Bumped watchdog to version >= 2.0.
+* Added `desktop-notifier` dependency. This is spin-off project from Maestral, built on
+  the code previously in the `notify` module.
+* Removed the bugsnag dependency.
+
 ## v1.3.1
 
-#### Fixes:
+#### Fixed:
 
 * Fixes an incorrect entry point for the Qt GUI.
 
@@ -25,7 +143,7 @@ series of bug fixes for GUI and daemon.
 #### Changed:
 
 * Significant improvements to the command line interface:
-    * Overhauled all CLI dialogs with nicer formatting and more interactive prompts 
+    * Overhauled all CLI dialogs with nicer formatting and more interactive prompts
       using the `survey` package.
     * Improved output of many CLI commands, including `ls`, `activity`, and `restore`.
     * Increased speed of many CLI commands by importing only necessary modules.
@@ -42,7 +160,7 @@ series of bug fixes for GUI and daemon.
 * The `Maestral.excluded_items` property is no longer read-only.
 * Some refactoring of the `cli` module to prepare for shell completion support.
 
-#### Fixes:
+#### Fixed:
 
 * Fixes an issue where all newly downloaded files would be created with 755 permissions.
   They are now created with the user's default permissions for new files instead.
@@ -55,10 +173,10 @@ series of bug fixes for GUI and daemon.
 * Fixes possible loss of data when excluding an item from syncing while it is
   downloaded. This is no longer possible and will raise a `BusyError` instead.
 * Fixes an issue where `maestral ls` would fail when run with the `-l, --long` flag.
-* Fixes an occasional `IndexError` during a download sync when trying to query past 
+* Fixes an occasional `IndexError` during a download sync when trying to query past
   versions of a deleted item.
 * Fixes an issue which could cause a segfault of the selective sync dialog on macOS.
-* Fixes an issue where the selective sync dialog on Linux would not load the contents of 
+* Fixes an issue where the selective sync dialog on Linux would not load the contents of
   more than 10 folders.
 * Fixes a regression with the autostart functionality of the Linux GUI. Autostart
   entries created with v1.2.2 will need be reset by toggling the checkbox "start on
@@ -94,7 +212,7 @@ series of bug fixes for GUI and daemon.
 This release focuses on bug fixes and performance improvements. In particular, memory
 usage has been improved when syncing a Dropbox folder with a large number of items.
 
-#### Changes:
+#### Changed:
 
 - `maestral file-status` now accepts relative paths.
 - Runs the daemon in a Python interpreter with -OO flags. This strips docstrings and saves
@@ -112,7 +230,7 @@ usage has been improved when syncing a Dropbox folder with a large number of ite
 - Switch from PyInstaller to [briefcase](https://github.com/beeware/briefcase) for
   packaging on macOS.
 
-#### Fixes:
+#### Fixed:
 
 - Fixes an issue which would prevent the daemon from starting on macOS when running with
   Python 3.6.
@@ -155,7 +273,7 @@ full compatibility from macOS 10.13 High Sierra to macOS 11.0 Big Sur.
 - Improves log messages when the connection to Dropbox is lost.
 - Performance improvements to `maestral activity` in case of very large sync queues.
 
-#### Fixes:
+#### Fixed:
 
 - Fixes a database integrity error due to an unfulfilled unique constraint.
 - Fixes an issue when the daemon is launched with systemd where systemd would unexpectedly

CONTRIBUTING.md

@@ -1,5 +1,5 @@
 
-## Guidelines
+### Code
 
 To start, install maestral with the `dev` extra to get all dependencies required for
 development:
@@ -8,7 +8,8 @@ development:
 pip3 install maestral[dev]
 ```
 
-### Checking the Format, Coding Style, and Type Hints
+This will install packages to check and enforce the code style, use pre-commit hooks and
+bump the current version.
 
 Code is formatted with [black](https://github.com/psf/black).
 Coding style is checked with [flake8](http://flake8.pycqa.org).
@@ -23,7 +24,8 @@ pre-commit run -a
 ```
 
 You can also install the provided pre-commit hook to run checks on every commit. This
-will however significantly slow down commits.
+will however significantly slow down commits. An introduction to pre-commit commit hooks
+is given at [https://pre-commit.com](https://pre-commit.com).
 
 ### Documentation
 
@@ -40,7 +42,7 @@ pip3 install maestral[docs]
 The API documentation is mostly based on doc strings. Inline comments should be used 
 whenever code may be difficult to understand for others.
 
-## Tests
+### Tests
 
 The test suite uses a mixture of [unittest](https://docs.python.org/3.8/library/unittest.html)
 and [pytest](https://pytest-cov.readthedocs.io/en/latest/), depending on what is most
@@ -56,10 +58,30 @@ indexing and cleaning up sync events, and for particularly complex functions tha
 prone to regressions.
 
 The current test suite uses a Dropbox access token provided by the environment variable
-`DROPBOX_TOKEN` to connect to a real account. The GitHub action which is running the
-tests will set this environment variable for you with a temporary access token that
+`DROPBOX_ACCESS_TOKEN` or a refresh token provided by `DROPBOX_REFRESH_TOKEN` to connect
+to a real account. The GitHub action which is running the tests will set the
+`DROPBOX_ACCESS_TOKEN` environment variable for you with a temporary access token that
 expires after 4 hours. Tests are run on `ubuntu-latest` and `macos-latest` in parallel
-on different accounts and you should acquire a "lock" on the account before running
-tests. Fixtures to create and clean up a test config and to acquire a lock are provided
-in the `tests/linked/conftest.py`. If you run the tests locally, you will need to
-provide an access token for your own Dropbox account.
+on different accounts.
+
+When using the GitHub test runner, you should acquire a "lock" on the account before
+running tests to prevent them from interfering which each other by creating a folder
+`test.lock` in the root of the Dropbox folder. This folder should have a 
+`client_modified` time set in the future, to the expiry time of the lock. Fixtures to
+create and clean up a test config and to acquire a lock are provided in the
+`tests/linked/conftest.py`.
+
+If you run the tests locally, you will need to provide a refresh or access token for 
+your own Dropbox account. If your account is already linked with Maestral, it will have 
+saved a long-lived "refresh token" in your system keyring. You can access it manually or 
+through the Python API:
+
+```Python
+from maestral.main import Maestral
+
+m = Maestral()
+print(m.client.auth.refresh_token)
+```
+
+You can then store the retrieved refresh token in the environment variable
+`DROPBOX_REFRESH_TOKEN` to be automatically picked up by the tests.

README.md

@@ -31,7 +31,8 @@ or downloading a file if it already exists with the same content locally or in t
 
 ## Warning
 
-- Never sync a folder with both the official Dropbox client and Maestral at the same time.
+- Never sync a local folder with both the official Dropbox client and Maestral at the same
+  time.
 - Network drives and some external hard drives are not supported as locations for the
   Dropbox folder.
 
@@ -167,6 +168,22 @@ month to offset the cost of an Apple Developer account to sign and notarize the
 
 # Acknowledgements
 
+Maestral directly uses code from the following projects:
+
 - The config module uses code from the [Spyder IDE](https://github.com/spyder-ide)
-- The DropboxClient is inspired by work from [Orphilia](https://github.com/ksiazkowicz/orphilia-dropbox)
-- Error reporting is powered by bugsnag.
+- The DropboxClient module is inspired by work from [Orphilia](https://github.com/ksiazkowicz/orphilia-dropbox)
+
+It also would not be possible without the following excellent Python packages:
+
+- Communication between sync daemon and frontends uses [Pyro5](https://github.com/irmen/Pyro5).
+- The command line interface is built with [click](https://github.com/pallets/click) and
+  uses beautiful interactive prompts by [survey](https://github.com/Exahilosys/survey).
+- The Cocoa GUI is built using [toga](https://github.com/beeware/toga) and the macOS app
+  bundle is built using [briefcase](https://github.com/beeware/briefcase), both part of
+  the [beeware](https://beeware.org) project for writing cross-platform Python applications.
+- Credential storage uses system keychains via [keyring](https://github.com/jaraco/keyring).
+- [watchdog](https://github.com/gorakhargosh/watchdog) allows us to receive local file
+  system events.  
+- Error reporting is generously provided by [bugsnag](https://www.bugsnag.com).
+- Many more well known libraries that have become the backbone of Python projects
+  such as requests, sqlalchemy, etc.