Merge tag '0.19.3'

Update version to 0.19.3

* tag '0.19.3': (39 commits)
  [skip ci] Update docs/source/changelog.rst
  [skip ci] Update CHANGELOG
  Fix RRID badge URL
  Display RRID in README
  RF(TST): use straight "assert" so may be pytest would provide more context
  [release-action] Autogenerate changelog snippet for PR 7472
  BF: dereference relative path while checking for bare from the original repo path not curdir
  RF(TST): move common checks into a helper
  [release-action] Autogenerate changelog snippet for PR 7467
  Disable some S3 tests of their VCR taping where they fail for known issues
  Test the fix by cloning from a URL and not specifying destination
  [release-action] Autogenerate changelog snippet for PR 7463
  BF: pass branch option into recursive call within Install
  Add changelog snippet
  docs: update candidate priorities (costs) in get docstring
  docs: fix placeholder name (get's subdataset source candidate)
  Update funding acknowledgements
  improve changelog
  add testcase for datalad#7429
  [release-action] Autogenerate changelog snippet for PR 7430
  ...

CHANGELOG.md

@@ -1,4 +1,47 @@
 
+<a id='changelog-0.19.3'></a>
+# 0.19.3 (2023-08-10)
+
+## 🐛 Bug Fixes
+
+- Type annotate get_status_dict and note that we can pass Exception or CapturedException which is not subclass.  [PR #7403](https://github.com/datalad/datalad/pull/7403) (by [@yarikoptic](https://github.com/yarikoptic))
+
+- BF: create-sibling-gitlab used to raise a TypeError when attempting a recursive operation in a dataset with uninstalled subdatasets. It now raises an impossible result instead.  [PR #7430](https://github.com/datalad/datalad/pull/7430) (by [@adswa](https://github.com/adswa))
+
+- Pass branch option into recursive call within Install - for the cases whenever install is invoked with URL(s).  Fixes [#7461](https://github.com/datalad/datalad/issues/7461) via [PR #7463](https://github.com/datalad/datalad/pull/7463) (by [@yarikoptic](https://github.com/yarikoptic))
+
+- Allow for reckless=ephemeral clone using relative path for the original location.  Fixes [#7469](https://github.com/datalad/datalad/issues/7469) via [PR #7472](https://github.com/datalad/datalad/pull/7472) (by [@yarikoptic](https://github.com/yarikoptic))
+
+## 📝 Documentation
+
+- Fix a property name and default costs described in "getting subdatasets" section of `get` documentation.
+  Fixes [#7458](https://github.com/datalad/datalad/issues/7458) via
+  [PR #7460](https://github.com/datalad/datalad/pull/7460)
+  (by [@mslw](https://github.com/mslw))
+
+## 🏠 Internal
+
+- Copy an adjusted environment only if requested to do so.
+  [PR #7399](https://github.com/datalad/datalad/pull/7399)
+  (by [@christian-monch](https://github.com/christian-monch))
+
+- Eliminate uses of `pkg_resources`.  Fixes [#7435](https://github.com/datalad/datalad/issues/7435) via [PR #7439](https://github.com/datalad/datalad/pull/7439) (by [@jwodder](https://github.com/jwodder))
+
+## 🧪 Tests
+
+- Disable some S3 tests of their VCR taping where they fail for known issues.  [PR #7467](https://github.com/datalad/datalad/pull/7467) (by [@yarikoptic](https://github.com/yarikoptic))
+
+<a id='changelog-0.19.2'></a>
+# 0.19.2 (2023-07-03)
+
+## 🐛 Bug Fixes
+
+- Remove surrounding quotes in output filenames even for newer version of annex.  Fixes [#7440](https://github.com/datalad/datalad/issues/7440) via [PR #7443](https://github.com/datalad/datalad/pull/7443) (by [@yarikoptic](https://github.com/yarikoptic))
+
+## 📝 Documentation
+
+- DOC: clarify description of the "install" interface to reflect its convoluted behavior.  [PR #7445](https://github.com/datalad/datalad/pull/7445) (by [@yarikoptic](https://github.com/yarikoptic))
+
 <a id='changelog-0.19.1'></a>
 # 0.19.1 (2023-06-26)
 

README.md

@@ -19,6 +19,7 @@
 [![https://www.singularity-hub.org/static/img/hosted-singularity--hub-%23e32929.svg](https://www.singularity-hub.org/static/img/hosted-singularity--hub-%23e32929.svg)](https://singularity-hub.org/collections/667)
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](https://github.com/datalad/datalad/blob/master/CODE_OF_CONDUCT.md)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.808846.svg)](https://doi.org/10.5281/zenodo.808846)
+[![RRID](https://img.shields.io/badge/RRID-SCR__003931-blue)](https://identifiers.org/RRID:SCR_003931)
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
 [![All Contributors](https://img.shields.io/badge/all_contributors-49-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
@@ -139,16 +140,39 @@ contributing to the project.
 
 ## Acknowledgements
 
-DataLad development is supported by a US-German collaboration in computational
-neuroscience (CRCNS) project "DataGit: converging catalogues, warehouses, and
-deployment logistics into a federated 'data distribution'" (Halchenko/Hanke),
-co-funded by the US National Science Foundation (NSF 1429999) and the German
-Federal Ministry of Education and Research (BMBF 01GQ1411). Additional support
-is provided by the German federal state of Saxony-Anhalt and the European
-Regional Development Fund (ERDF), Project: Center for Behavioral Brain
-Sciences, Imaging Platform.  This work is further facilitated by the ReproNim
-project (NIH 1P41EB019936-01A1). Mac mini instance for development is provided
-by [MacStadium](https://www.macstadium.com/).
+The DataLad project received support through the following grants:
+
+- US-German collaboration in computational neuroscience (CRCNS) project
+  "DataGit: converging catalogues, warehouses, and deployment logistics into a
+  federated 'data distribution'" (Halchenko/Hanke), co-funded by the US National
+  Science Foundation (NSF 1429999) and the German Federal Ministry of
+  Education and Research (BMBF 01GQ1411).
+
+- CRCNS US-German Data Sharing "DataLad - a decentralized system for integrated
+  discovery, management, and publication of digital objects of science"
+  (Halchenko/Pestilli/Hanke), co-funded by the US National Science Foundation
+  (NSF 1912266) and the German Federal Ministry of Education and Research
+  (BMBF 01GQ1905).
+
+- Helmholtz Research Center Jülich, FDM challenge 2022
+
+- German federal state of Saxony-Anhalt and the European Regional Development
+  Fund (ERDF), Project: Center for Behavioral Brain Sciences, Imaging Platform
+
+- ReproNim project (NIH 1P41EB019936-01A1).
+
+- Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant
+  SFB 1451 ([431549029](https://gepris.dfg.de/gepris/projekt/431549029),
+  INF project)
+
+- European Union’s Horizon 2020 research and innovation programme under grant
+  agreements:
+  - [Human Brain Project SGA3 (H2020-EU.3.1.5.3, grant no. 945539)](https://cordis.europa.eu/project/id/945539)
+  - [VirtualBrainCloud (H2020-EU.3.1.5.3, grant no. 826421)](https://cordis.europa.eu/project/id/826421)
+
+Mac mini instance for development is provided by
+[MacStadium](https://www.macstadium.com/).
+
 
 ### Contributors ✨
 

datalad/cli/parser.py

@@ -11,29 +11,29 @@
 # like error handling, must be done conditionally in-line.
 
 import argparse
+import logging
+import sys
 from collections import defaultdict
 from functools import partial
-import sys
-
 
 from datalad import __version__
-
-from .common_args import common_args
 from datalad.interface.base import (
-    is_api_arg,
     get_cmd_doc,
     get_interface_groups,
+    is_api_arg,
     load_interface,
 )
+from datalad.support.constraints import EnsureChoice
 from datalad.utils import getargspec
+
+from .common_args import common_args
+from .exec import call_from_parser
+from .helpers import get_commands_from_groups
 from .interface import (
     alter_interface_docs_for_cmdline,
     get_cmd_ex,
     get_cmdline_command_name,
 )
-from datalad.support.constraints import EnsureChoice
-from .helpers import get_commands_from_groups
-from .exec import call_from_parser
 
 # special case imports
 #  .helpers import add_entrypoints_to_interface_groups
@@ -43,7 +43,6 @@
 #  .interface._known_extension_commands
 #  .interface._deprecated_commands
 
-import logging
 lgr = logging.getLogger('datalad.cli.parser')
 
 
@@ -138,6 +137,7 @@ def setup_parser(
         # we need the full help, or we have a potential command that
         # lives in an extension, must load all extension, expensive
         from .helpers import add_entrypoints_to_interface_groups
+
         # need to load all the extensions and try again
         # TODO load extensions one-by-one and stop when a command was found
         add_entrypoints_to_interface_groups(interface_groups)
@@ -404,8 +404,8 @@ def try_suggest_extension_with_command(parser, cmd, completing, known_cmds):
     """If completing=False, this function will trigger sys.exit()"""
     # check if might be coming from known extensions
     from .interface import (
-        _known_extension_commands,
         _deprecated_commands,
+        _known_extension_commands,
     )
     extension_commands = {
         c: e
@@ -541,8 +541,13 @@ def print_version():
             # Let's use the standard Python mechanism if underlying module
             # did not provide __version__
             try:
-                import pkg_resources
-                version = pkg_resources.get_distribution(mod_name).version
+                if sys.version_info < (3, 10):
+                    import importlib_metadata as im
+                else:
+                    import importlib.metadata as im
+
+                pkg = im.packages_distributions()[mod_name][0]
+                version = im.version(pkg)
             except Exception:
                 version = "unknown"
         if include_name:

datalad/core/distributed/clone_ephemeral.py

@@ -92,6 +92,10 @@ def _setup_ephemeral_annex(ds: Dataset, remote: str):
                 # If origin isn't local, we have nothing to do.
                 origin_git_path = Path(RI(origin_annex_url).localpath)
 
+                if not origin_git_path.is_absolute():
+                    # relative path would be relative to the ds, not pwd!
+                    origin_git_path = ds.pathobj / origin_git_path
+
                 # we are local; check for a bare repo first to not mess w/
                 # the path
                 if GitRepo(origin_git_path, create=False).bare:

datalad/core/distributed/tests/test_clone.py

@@ -1365,42 +1365,38 @@ def test_ria_http_storedataladorg(path=None):
 @with_tempfile
 @with_tempfile
 @with_tempfile
+@with_tempfile
 def test_ephemeral(origin_path=None, bare_path=None,
-                   clone1_path=None, clone2_path=None, clone3_path=None):
+                   clone1_path=None, clone2_path=None,
+                   clone3_path=None, clone4_path=None):
+    can_symlink = has_symlink_capability()
 
     file_test = Path('ds') / 'test.txt'
     file_testsub = Path('ds') / 'subdir' / 'testsub.txt'
 
     origin = Dataset(origin_path).create(force=True)
     origin.save()
-    # 1. clone via path
-    clone1 = clone(origin_path, clone1_path, reckless='ephemeral')
-    eq_(clone1.config.get("annex.private"), "true")
 
-    can_symlink = has_symlink_capability()
+    def check_clone(clone_):
+        # common checks to do on a clone
+        eq_(clone_.config.get("annex.private"), "true")
+        if can_symlink:
+            clone_annex = (clone_.repo.dot_git / 'annex')
+            ok_(clone_annex.is_symlink())
+            ok_(clone_annex.resolve().samefile(origin.repo.dot_git / 'annex'))
+            if not clone_.repo.is_managed_branch():
+                # TODO: We can't properly handle adjusted branch yet
+                eq_((clone_.pathobj / file_test).read_text(), 'some')
+                eq_((clone_.pathobj / file_testsub).read_text(), 'somemore')
 
-    if can_symlink:
-        clone1_annex = (clone1.repo.dot_git / 'annex')
-        ok_(clone1_annex.is_symlink())
-        ok_(clone1_annex.resolve().samefile(origin.repo.dot_git / 'annex'))
-        if not clone1.repo.is_managed_branch():
-            # TODO: We can't properly handle adjusted branch yet
-            eq_((clone1.pathobj / file_test).read_text(), 'some')
-            eq_((clone1.pathobj / file_testsub).read_text(), 'somemore')
+    # 1. clone via path
+    clone1 = clone(origin_path, clone1_path, reckless='ephemeral')
+    check_clone(clone1)
 
     # 2. clone via file-scheme URL
     clone2 = clone('file://' + Path(origin_path).as_posix(), clone2_path,
                    reckless='ephemeral')
-    eq_(clone2.config.get("annex.private"), "true")
-
-    if can_symlink:
-        clone2_annex = (clone2.repo.dot_git / 'annex')
-        ok_(clone2_annex.is_symlink())
-        ok_(clone2_annex.resolve().samefile(origin.repo.dot_git / 'annex'))
-        if not clone2.repo.is_managed_branch():
-            # TODO: We can't properly handle adjusted branch yet
-            eq_((clone2.pathobj / file_test).read_text(), 'some')
-            eq_((clone2.pathobj / file_testsub).read_text(), 'somemore')
+    check_clone(clone2)
 
     # 3. add something to clone1 and push back to origin availability from
     # clone1 should not be propagated (we declared 'here' dead to that end)
@@ -1456,6 +1452,12 @@ def test_ephemeral(origin_path=None, bare_path=None,
         ok_(eph_annex.is_symlink())
         ok_(eph_annex.resolve().samefile(Path(bare_path) / 'annex'))
 
+    # 5. ephemeral clone using relative path
+    # https://github.com/datalad/datalad/issues/7469
+    with chpwd(op.dirname(origin_path)):
+        clone4 = clone(op.basename(origin_path), op.basename(clone4_path), reckless='ephemeral')
+    check_clone(clone4)
+
 
 @with_tempfile(mkdir=True)
 def test_clone_unborn_head(path=None):

datalad/core/local/save.py

@@ -13,48 +13,44 @@
 __docformat__ = 'restructuredtext'
 
 import logging
-
 from functools import partial
+from pathlib import Path
 
+import datalad.utils as ut
+from datalad.distribution.dataset import (
+    Dataset,
+    EnsureDataset,
+    datasetmethod,
+    require_dataset,
+)
 from datalad.interface.base import (
     Interface,
     build_doc,
     eval_results,
 )
 from datalad.interface.common_opts import (
     jobs_opt,
-    recursion_limit,
     recursion_flag,
+    recursion_limit,
     save_message_opt,
 )
 from datalad.interface.utils import (
-    get_tree_roots,
     discover_dataset_trace_to_targets,
+    get_tree_roots,
 )
-from datalad.support.param import Parameter
 from datalad.support.constraints import (
-    EnsureStr,
     EnsureNone,
+    EnsureStr,
 )
 from datalad.support.exceptions import CommandError
 from datalad.support.parallel import (
-    no_subds_in_futures,
     ProducerConsumerProgressLog,
+    no_subds_in_futures,
 )
-from datalad.utils import (
-    ensure_list,
-)
-import datalad.utils as ut
+from datalad.support.param import Parameter
+from datalad.utils import ensure_list
 
-from datalad.distribution.dataset import (
-    Dataset,
-    EnsureDataset,
-    datasetmethod,
-    require_dataset,
-)
-from .status import (
-    Status,
-)
+from .status import Status
 
 lgr = logging.getLogger('datalad.core.local.save')
 
@@ -328,7 +324,7 @@ def save_ds(args, version_tag=None):
                         if k in res:
                             res[k] = str(
                                 # recode path back to dataset path anchor
-                                pds.pathobj / res[k].relative_to(
+                                pds.pathobj / Path(res[k]).relative_to(
                                     pds_repo.pathobj)
                             )
                     yield res

datalad/distributed/create_sibling_gitlab.py

@@ -298,16 +298,21 @@ def __call__(
                     return_type='list')
             if not subds:
                 # we didn't find anything to operate on, let the user know
-                for p in path:
+                res_kwargs = {'status': 'impossible', 'refds': ds.path,
+                              'type':'dataset', 'logger': lgr,
+                              'action': 'create_sibling_gitlab'}
+                if path is not None:
+                    for p in path:
+                        yield dict(
+                              path=p,
+                              message=('No installed dataset found under %s, forgot to "get" it?' % p),
+                              **res_kwargs
+                        )
+                else:
                     yield dict(
-                          action='create_sibling_gitlab',
-                          status='impossible',
-                          refds=ds.path,
-                          path=p,
-                          message=('No dataset found under %s' % p),
-                          type='dataset',
-                          logger=lgr,
-                    )
+                        path=ds.path,
+                        message=('No installed subdatasets found underneath %s, forgot to "get" any?' % ds.path),
+                        **res_kwargs)
             else:
                 for sub in subds:
                     for r in _proc_dataset(