Merge branch 'main' of github.com:snakemake/snakemake-storage-plugin-fs

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## [1.0.3](https://github.com/snakemake/snakemake-storage-plugin-fs/compare/v1.0.2...v1.0.3) (2024-04-17)
+
+
+### Bug Fixes
+
+* fix query validation (used deprecated API) ([9bbeaaa](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/9bbeaaa302683fc08b15c27f1de5b4ab4492f2aa))
+
+## [1.0.2](https://github.com/snakemake/snakemake-storage-plugin-fs/compare/v1.0.1...v1.0.2) (2024-04-16)
+
+
+### Miscellaneous Chores
+
+* release 1.0.2 ([173ce62](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/173ce62626d49f17fb99fc94519ec981339ffc1e))
+
+## [1.0.1](https://github.com/snakemake/snakemake-storage-plugin-fs/compare/v1.0.0...v1.0.1) (2024-04-16)
+
+
+### Bug Fixes
+
+* do not consider URL-like queries as valid ([bb0cd39](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/bb0cd390ac288600f61ed4fb7236b8dc2bfdbc03))
+
+## [1.0.0](https://github.com/snakemake/snakemake-storage-plugin-fs/compare/v0.2.0...v1.0.0) (2024-02-24)
+
+
+### ⚠ BREAKING CHANGES
+
+* remove latency wait support as this should rather be handled in main Snakemake (as it is already done). The reason is that the plugin cannot distinguish between cases where latency has to be taken into accound and where not, leading to overall much slower processing when latency wait is applied regardless of the context. This introduces a breaking change because this plugin now does not offer any settings anymore. ([#13](https://github.com/snakemake/snakemake-storage-plugin-fs/issues/13))
+
+### Bug Fixes
+
+* remove latency wait support as this should rather be handled in main Snakemake (as it is already done). The reason is that the plugin cannot distinguish between cases where latency has to be taken into accound and where not, leading to overall much slower processing when latency wait is applied regardless of the context. This introduces a breaking change because this plugin now does not offer any settings anymore. ([#13](https://github.com/snakemake/snakemake-storage-plugin-fs/issues/13)) ([1c78d88](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/1c78d880925ad1af08195d4035028a9117755551))
+* respect permissions of the target dir, especially setgid ([#11](https://github.com/snakemake/snakemake-storage-plugin-fs/issues/11)) ([2132a5a](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/2132a5a845f865fd076235620f8e2a91a2300206))
+
+## [0.2.0](https://github.com/snakemake/snakemake-storage-plugin-fs/compare/v0.1.5...v0.2.0) (2024-02-19)
+
+
+### Features
+
+* add touch support ([#9](https://github.com/snakemake/snakemake-storage-plugin-fs/issues/9)) ([7ed5a6a](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/7ed5a6a5a6d4208124f946ae9269e72ab0d9e509))
+* provide setting for latency wait ([#8](https://github.com/snakemake/snakemake-storage-plugin-fs/issues/8)) ([9ac5c8d](https://github.com/snakemake/snakemake-storage-plugin-fs/commit/9ac5c8ddc89d6a35c36dc991bdc24f1965f42a16))
+
 ## [0.1.5](https://github.com/snakemake/snakemake-storage-plugin-fs/compare/v0.1.4...v0.1.5) (2023-12-20)
 
 

docs/further.md

@@ -16,4 +16,20 @@ shared-fs-usage:
   - software-deployment
   - sources
   - source-cache
-```
+```
+
+If the shared scratch is e.g. specific for each job (e.g. controlled by a ``$JOBID``), one can define a job-specific local storage prefix like this
+
+```yaml
+default-storage-provider: fs
+local-storage-prefix: /local/work/$USER
+remote-job-local-storage-prefix: /local/work/$USER/$JOBID
+shared-fs-usage:
+  - persistence
+  - software-deployment
+  - sources
+  - source-cache
+```
+
+Note that the non-remote job local storage prefix is still always needed, because Snakemake can also decide to run certain jobs without submission to the cluster or cloud.
+This can happen either on dev request because a certain rule is very lightweight, or by Snakemake's own decision, e.g. in case of rules that just format a template (see `docs <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#template-rendering-integration>`_).

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "snakemake-storage-plugin-fs"
-version = "0.1.5"
+version = "1.0.3"
 description = " A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync"
 authors = ["Johannes Koester <[email protected]>"]
 readme = "README.md"
@@ -11,9 +11,10 @@ keywords = ["snakemake", "plugin", "storage", "filesystem", "rsync"]
 
 [tool.poetry.dependencies]
 python = "^3.11"
-snakemake-interface-common = "^1.14.2"
-snakemake-interface-storage-plugins = "^3.0.0"
+snakemake-interface-common = "^1.17.0"
+snakemake-interface-storage-plugins = "^3.2.2"
 sysrsync = "^1.1.1"
+reretry = "^0.11.8"
 
 
 [tool.poetry.group.dev.dependencies]

snakemake_storage_plugin_fs/__init__.py

@@ -3,6 +3,7 @@
 import shutil
 import subprocess
 from typing import Any, Iterable, List, Optional
+from urllib.parse import urlparse
 
 import sysrsync
 
@@ -18,12 +19,14 @@
     StorageObjectRead,
     StorageObjectWrite,
     StorageObjectGlob,
+    StorageObjectTouch,
 )
 from snakemake_interface_storage_plugins.io import (
     IOCacheStorageInterface,
     get_constant_prefix,
     Mtime,
 )
+from snakemake_interface_common.utils import lutime
 
 
 # Required:
@@ -77,13 +80,23 @@ def is_valid_query(cls, query: str) -> StorageQueryValidationResult:
         # Ensure that also queries containing wildcards (e.g. {sample}) are accepted
         # and considered valid. The wildcards will be resolved before the storage
         # object is actually used.
+
+        # disallow queries that are URL like
+        parsed = urlparse(query)
+        if parsed.scheme:
+            return StorageQueryValidationResult(
+                query=query,
+                valid=False,
+                reason="Query is URL-like, but should be a system path instead.",
+            )
+
         try:
             Path(query)
         except Exception:
             return StorageQueryValidationResult(
                 query=query,
                 valid=False,
-                message="Query is not a valid path.",
+                reason="Query is not a valid path.",
             )
         return StorageQueryValidationResult(
             query=query,
@@ -109,7 +122,9 @@ def list_objects(self, query: Any) -> Iterable[str]:
 # storage (e.g. because it is read-only see
 # snakemake-storage-http for comparison), remove the corresponding base classes
 # from the list of inherited items.
-class StorageObject(StorageObjectRead, StorageObjectWrite, StorageObjectGlob):
+class StorageObject(
+    StorageObjectRead, StorageObjectWrite, StorageObjectGlob, StorageObjectTouch
+):
     # For compatibility with future changes, you should not overwrite the __init__
     # method. Instead, use __post_init__ to set additional attributes and initialize
     # futher stuff.
@@ -135,11 +150,11 @@ async def inventory(self, cache: IOCacheStorageInterface):
             # already inventorized, stop here
             return
 
-        try:
-            stat = self._stat()
-        except FileNotFoundError:
+        if not self.exists():
             cache.exists_in_storage[key] = False
             return
+
+        stat = self._stat()
         if self.query_path.is_symlink():
             # get symlink stat
             lstat = self._stat(follow_symlinks=False)
@@ -170,9 +185,6 @@ def cleanup(self):
         # Nothing to be done here.
         pass
 
-    # Fallible methods should implement some retry logic.
-    # The easiest way to do this (but not the only one) is to use the retry_decorator
-    # provided by snakemake-interface-storage-plugins.
     def exists(self) -> bool:
         # return True if the object exists
         return self.query_path.exists()
@@ -196,6 +208,9 @@ def store_object(self):
         # Ensure that the object is stored at the location specified by
         # self.local_path().
         self.query_path.parent.mkdir(exist_ok=True, parents=True)
+        # We want to respect the permissions in the target folder, in particular the
+        # setgid bit. Hence, we use --no-p to avoid preserving of permissions from the
+        # source to the target.
         cmd = sysrsync.get_rsync_command(
             str(self.local_path()),
             str(self.query_path),
@@ -234,3 +249,15 @@ def _stat(self, follow_symlinks: bool = True):
         # We don't want the cached variant (Path.stat), as we cache ourselves in
         # inventory and afterwards the information may change.
         return os.stat(self.query_path, follow_symlinks=follow_symlinks)
+
+    def touch(self):
+        if self.query_path.exists():
+            if self.query_path.is_dir():
+                flag = self.query_path / ".snakemake_timestamp"
+                # Create the flag file if it doesn't exist
+                if not flag.exists():
+                    with open(flag, "w"):
+                        pass
+                lutime(flag, None)
+            else:
+                lutime(str(self.query_path), None)

tests/tests.py

@@ -10,6 +10,7 @@
 class TestStorageNoSettings(TestStorageBase):
     __test__ = True
     retrieve_only = False
+    touch = True
 
     def get_query(self, tmp_path) -> str:
         parent = f"{tmp_path}/storage/test/"