feat(drs-pull-objects): initial support for supplying a list of DRS I… (

#195)

* feat(drs-pull-objects): initial support for supplying a list of DRS IDs to download

* Apply automatic documentation changes

* feat(logging): add arg for silencing logging (that way we can ensure click.echos are the only stdout)

* feat(drs-pull): correctly allow list of object IDs as an arg, support JSON output

* Apply automatic documentation changes

* chore(spelling): fix typo

* Apply automatic documentation changes

* chore(drs-pull): refactor to reduce function calls, support passing a list of objects to underlying download functions

* Apply automatic documentation changes

* chore(tests): fix test and version

* Apply automatic documentation changes

docs/_build/html/_modules/gen3/tools/download/drs_download.html

@@ -1211,7 +1211,7 @@ <h1>Source code for gen3.tools.download.drs_download</h1><div class="highlight">
 <span class="k">def</span> <span class="nf">_download_obj</span><span class="p">(</span>
     <span class="n">hostname</span><span class="p">,</span>
     <span class="n">auth</span><span class="p">,</span>
-    <span class="n">object_id</span><span class="p">,</span>
+    <span class="n">object_ids</span><span class="p">,</span>
     <span class="n">output_dir</span><span class="o">=</span><span class="s2">&quot;.&quot;</span><span class="p">,</span>
     <span class="n">show_progress</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
     <span class="n">unpack_packages</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
@@ -1222,7 +1222,7 @@ <h1>Source code for gen3.tools.download.drs_download</h1><div class="highlight">
 <span class="sd">    Args:</span>
 <span class="sd">        hostname (str): hostname of Gen3 commons to use for access and WTS</span>
 <span class="sd">        auth: Gen3 Auth instance</span>
-<span class="sd">        object_id (str): DRS object id</span>
+<span class="sd">        object_ids (List[str]): DRS object id</span>
 <span class="sd">        output_dir: directory to save downloaded files to</span>
 <span class="sd">        show_progress: show progress bar</span>
 <span class="sd">        unpack_packages (bool): set to False to disable the unpacking of downloaded packages</span>
@@ -1237,7 +1237,7 @@ <h1>Source code for gen3.tools.download.drs_download</h1><div class="highlight">
         <span class="n">logger</span><span class="o">.</span><span class="n">critical</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unable to authenticate your credentials with </span><span class="si">{</span><span class="n">hostname</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
         <span class="k">return</span> <span class="kc">None</span>
 
-    <span class="n">object_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">Downloadable</span><span class="p">(</span><span class="n">object_id</span><span class="o">=</span><span class="n">object_id</span><span class="p">)]</span>
+    <span class="n">object_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">Downloadable</span><span class="p">(</span><span class="n">object_id</span><span class="o">=</span><span class="n">object_id</span><span class="p">)</span> <span class="k">for</span> <span class="n">object_id</span> <span class="ow">in</span> <span class="n">object_ids</span><span class="p">]</span>
     <span class="n">downloader</span> <span class="o">=</span> <span class="n">DownloadManager</span><span class="p">(</span>
         <span class="n">hostname</span><span class="o">=</span><span class="n">hostname</span><span class="p">,</span>
         <span class="n">auth</span><span class="o">=</span><span class="n">auth</span><span class="p">,</span>
@@ -1430,12 +1430,10 @@ <h1>Source code for gen3.tools.download.drs_download</h1><div class="highlight">
 
 
 
-<div class="viewcode-block" id="download_drs_object">
-<a class="viewcode-back" href="../../../../tools/drs_pull.html#gen3.tools.download.drs_download.download_drs_object">[docs]</a>
-<span class="k">def</span> <span class="nf">download_drs_object</span><span class="p">(</span>
+<span class="k">def</span> <span class="nf">download_drs_objects</span><span class="p">(</span>
     <span class="n">hostname</span><span class="p">,</span>
     <span class="n">auth</span><span class="p">,</span>
-    <span class="n">object_id</span><span class="p">,</span>
+    <span class="n">object_ids</span><span class="p">,</span>
     <span class="n">output_dir</span><span class="p">,</span>
     <span class="n">show_progress</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
     <span class="n">unpack_packages</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
@@ -1446,7 +1444,7 @@ <h1>Source code for gen3.tools.download.drs_download</h1><div class="highlight">
 <span class="sd">    Args:</span>
 <span class="sd">        hostname (str): hostname of Gen3 commons to use for access and WTS</span>
 <span class="sd">        auth: Gen3 Auth instance</span>
-<span class="sd">        object_id (str): DRS object id</span>
+<span class="sd">        object_ids (List[str]): DRS object ids</span>
 <span class="sd">        output_dir: directory to save downloaded files to</span>
 <span class="sd">        unpack_packages (bool): set to False to disable the unpacking of downloaded packages</span>
 <span class="sd">        delete_unpacked_packages (bool): set to True to delete package files after unpacking them</span>
@@ -1457,13 +1455,12 @@ <h1>Source code for gen3.tools.download.drs_download</h1><div class="highlight">
     <span class="k">return</span> <span class="n">_download_obj</span><span class="p">(</span>
         <span class="n">hostname</span><span class="p">,</span>
         <span class="n">auth</span><span class="p">,</span>
-        <span class="n">object_id</span><span class="p">,</span>
+        <span class="n">object_ids</span><span class="p">,</span>
         <span class="n">output_dir</span><span class="p">,</span>
         <span class="n">show_progress</span><span class="p">,</span>
         <span class="n">unpack_packages</span><span class="p">,</span>
         <span class="n">delete_unpacked_packages</span><span class="p">,</span>
-    <span class="p">)</span></div>
-
+    <span class="p">)</span>
 
 
 <div class="viewcode-block" id="list_access_in_drs_manifest">

docs/_build/html/genindex.html

@@ -220,8 +220,6 @@ <h2 id="D">D</h2>
         <li><a href="wss.html#gen3.wss.Gen3WsStorage.download">(gen3.wss.Gen3WsStorage method)</a>
 </li>
       </ul></li>
-      <li><a href="tools/drs_pull.html#gen3.tools.download.drs_download.download_drs_object">download_drs_object() (in module gen3.tools.download.drs_download)</a>
-</li>
       <li><a href="tools/drs_pull.html#gen3.tools.download.drs_download.download_files_in_drs_manifest">download_files_in_drs_manifest() (in module gen3.tools.download.drs_download)</a>
 </li>
       <li><a href="file.html#gen3.file.Gen3File.download_single">download_single() (gen3.file.Gen3File method)</a>

docs/_build/html/index.html

@@ -223,7 +223,6 @@ <h1>Welcome to Gen3 SDK’s documentation!<a class="headerlink" href="#welcome-t
 <li class="toctree-l4"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.Manifest.load_manifest"><code class="docutils literal notranslate"><span class="pre">Manifest.load_manifest()</span></code></a></li>
 </ul>
 </li>
-<li class="toctree-l3"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.download_drs_object"><code class="docutils literal notranslate"><span class="pre">download_drs_object()</span></code></a></li>
 <li class="toctree-l3"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.download_files_in_drs_manifest"><code class="docutils literal notranslate"><span class="pre">download_files_in_drs_manifest()</span></code></a></li>
 <li class="toctree-l3"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.list_access_in_drs_manifest"><code class="docutils literal notranslate"><span class="pre">list_access_in_drs_manifest()</span></code></a></li>
 <li class="toctree-l3"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.list_drs_object"><code class="docutils literal notranslate"><span class="pre">list_drs_object()</span></code></a></li>

docs/_build/html/tools.html

@@ -80,7 +80,6 @@ <h1>Gen3 Tools<a class="headerlink" href="#gen3-tools" title="Link to this headi
 <li class="toctree-l3"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.Manifest.load_manifest"><code class="docutils literal notranslate"><span class="pre">Manifest.load_manifest()</span></code></a></li>
 </ul>
 </li>
-<li class="toctree-l2"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.download_drs_object"><code class="docutils literal notranslate"><span class="pre">download_drs_object()</span></code></a></li>
 <li class="toctree-l2"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.download_files_in_drs_manifest"><code class="docutils literal notranslate"><span class="pre">download_files_in_drs_manifest()</span></code></a></li>
 <li class="toctree-l2"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.list_access_in_drs_manifest"><code class="docutils literal notranslate"><span class="pre">list_access_in_drs_manifest()</span></code></a></li>
 <li class="toctree-l2"><a class="reference internal" href="tools/drs_pull.html#gen3.tools.download.drs_download.list_drs_object"><code class="docutils literal notranslate"><span class="pre">list_drs_object()</span></code></a></li>

docs/_build/html/tools/drs_pull.html

@@ -405,27 +405,6 @@ <h1>DRS Download Tools<a class="headerlink" href="#drs-download-tools" title="Li
 
 </dd></dl>
 
-<dl class="py function">
-<dt class="sig sig-object py" id="gen3.tools.download.drs_download.download_drs_object">
-<span class="sig-prename descclassname"><span class="pre">gen3.tools.download.drs_download.</span></span><span class="sig-name descname"><span class="pre">download_drs_object</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">hostname</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">auth</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">object_id</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_dir</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">show_progress</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unpack_packages</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">delete_unpacked_packages</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="reference internal" href="../_modules/gen3/tools/download/drs_download.html#download_drs_object"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#gen3.tools.download.drs_download.download_drs_object" title="Link to this definition">¶</a></dt>
-<dd><p>A convenience function used to download a single DRS object.
-:param hostname: hostname of Gen3 commons to use for access and WTS
-:type hostname: str
-:param auth: Gen3 Auth instance
-:param object_id: DRS object id
-:type object_id: str
-:param output_dir: directory to save downloaded files to
-:param unpack_packages: set to False to disable the unpacking of downloaded packages
-:type unpack_packages: bool
-:param delete_unpacked_packages: set to True to delete package files after unpacking them
-:type delete_unpacked_packages: bool</p>
-<dl class="field-list simple">
-<dt class="field-odd">Returns<span class="colon">:</span></dt>
-<dd class="field-odd"><p>List of DownloadStatus objects for the DRS object</p>
-</dd>
-</dl>
-</dd></dl>
-
 <dl class="py function">
 <dt class="sig sig-object py" id="gen3.tools.download.drs_download.download_files_in_drs_manifest">
 <span class="sig-prename descclassname"><span class="pre">gen3.tools.download.drs_download.</span></span><span class="sig-name descname"><span class="pre">download_files_in_drs_manifest</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">hostname</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">auth</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">infile</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_dir</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">show_progress</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unpack_packages</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">delete_unpacked_packages</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="reference internal" href="../_modules/gen3/tools/download/drs_download.html#download_files_in_drs_manifest"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#gen3.tools.download.drs_download.download_files_in_drs_manifest" title="Link to this definition">¶</a></dt>
@@ -569,7 +548,6 @@ <h3>Navigation</h3>
 <li class="toctree-l4"><a class="reference internal" href="#gen3.tools.download.drs_download.Manifest.load_manifest"><code class="docutils literal notranslate"><span class="pre">Manifest.load_manifest()</span></code></a></li>
 </ul>
 </li>
-<li class="toctree-l3"><a class="reference internal" href="#gen3.tools.download.drs_download.download_drs_object"><code class="docutils literal notranslate"><span class="pre">download_drs_object()</span></code></a></li>
 <li class="toctree-l3"><a class="reference internal" href="#gen3.tools.download.drs_download.download_files_in_drs_manifest"><code class="docutils literal notranslate"><span class="pre">download_files_in_drs_manifest()</span></code></a></li>
 <li class="toctree-l3"><a class="reference internal" href="#gen3.tools.download.drs_download.list_access_in_drs_manifest"><code class="docutils literal notranslate"><span class="pre">list_access_in_drs_manifest()</span></code></a></li>
 <li class="toctree-l3"><a class="reference internal" href="#gen3.tools.download.drs_download.list_drs_object"><code class="docutils literal notranslate"><span class="pre">list_drs_object()</span></code></a></li>

docs/_build/html/tools/indexing.html

@@ -380,7 +380,7 @@ <h2>Verify<a class="headerlink" href="#verify" title="Link to this heading">¶</
 
 <dl class="py function">
 <dt class="sig sig-object py" id="gen3.tools.indexing.verify_manifest.async_verify_object_manifest">
-<em class="property"><span class="k"><span class="pre">async</span></span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">gen3.tools.indexing.verify_manifest.</span></span><span class="sig-name descname"><span class="pre">async_verify_object_manifest</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">commons_url</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">manifest_file</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_concurrent_requests=24</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">manifest_row_parsers={'acl':</span> <span class="pre">&lt;function</span> <span class="pre">_get_acl_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'authz':</span> <span class="pre">&lt;function</span> <span class="pre">_get_authz_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'file_name':</span> <span class="pre">&lt;function</span> <span class="pre">_get_file_name_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'file_size':</span> <span class="pre">&lt;function</span> <span class="pre">_get_file_size_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'guid':</span> <span class="pre">&lt;function</span> <span class="pre">_get_guid_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'md5':</span> <span class="pre">&lt;function</span> <span class="pre">_get_md5_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'urls':</span> <span class="pre">&lt;function</span> <span class="pre">_get_urls_from_row&gt;}</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">manifest_file_delimiter=None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_filename='verify-manifest-errors-1692717039.3338752.log'</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/gen3/tools/indexing/verify_manifest.html#async_verify_object_manifest"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#gen3.tools.indexing.verify_manifest.async_verify_object_manifest" title="Link to this definition">¶</a></dt>
+<em class="property"><span class="k"><span class="pre">async</span></span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">gen3.tools.indexing.verify_manifest.</span></span><span class="sig-name descname"><span class="pre">async_verify_object_manifest</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">commons_url</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">manifest_file</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_concurrent_requests=24</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">manifest_row_parsers={'acl':</span> <span class="pre">&lt;function</span> <span class="pre">_get_acl_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'authz':</span> <span class="pre">&lt;function</span> <span class="pre">_get_authz_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'file_name':</span> <span class="pre">&lt;function</span> <span class="pre">_get_file_name_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'file_size':</span> <span class="pre">&lt;function</span> <span class="pre">_get_file_size_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'guid':</span> <span class="pre">&lt;function</span> <span class="pre">_get_guid_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'md5':</span> <span class="pre">&lt;function</span> <span class="pre">_get_md5_from_row&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">'urls':</span> <span class="pre">&lt;function</span> <span class="pre">_get_urls_from_row&gt;}</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">manifest_file_delimiter=None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_filename='verify-manifest-errors-1692886743.381985.log'</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/gen3/tools/indexing/verify_manifest.html#async_verify_object_manifest"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#gen3.tools.indexing.verify_manifest.async_verify_object_manifest" title="Link to this definition">¶</a></dt>
 <dd><p>Verify all file object records into a manifest csv</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>