Skip to content

Commit

Permalink
feat(indexing): new optional prev_guid column support to create new v… (
Browse files Browse the repository at this point in the history
#75)

* feat(indexing): new optional prev_guid column support to create new version

* Apply automatic documentation changes

* feat(tests): modify unit tests to check prev_guid, only check format when value provided

* Apply automatic documentation changes

* chore(pyproject): update version for new release

* Apply automatic documentation changes

* Apply automatic documentation changes

Co-authored-by: Alexander VT <[email protected]>
  • Loading branch information
Avantol13 and Avantol13-machine-user authored Feb 11, 2021
1 parent 05c89ac commit 7195c2f
Show file tree
Hide file tree
Showing 14 changed files with 116 additions and 17 deletions.
Binary file modified docs/_build/doctrees/environment.pickle
Binary file not shown.
Binary file modified docs/_build/doctrees/tools/indexing.doctree
Binary file not shown.
Binary file modified docs/_build/doctrees/tools/metadata.doctree
Binary file not shown.
45 changes: 42 additions & 3 deletions docs/_build/html/_modules/gen3/tools/indexing/index_manifest.html
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ <h1>Source code for gen3.tools.indexing.index_manifest</h1><div class="highlight
<span class="sd"> ACLS (list(string)): supported acl column names</span>
<span class="sd"> URLS (list(string)): supported url column names</span>
<span class="sd"> AUTHZ (list(string)): supported authz column names</span>

<span class="sd"> PREV_GUID (list(string)): supported previous guid column names</span>

<span class="sd">Usages:</span>
<span class="sd"> python index_manifest.py --commons_url https://giangb.planx-pla.net --manifest_file path_to_manifest --auth &quot;admin,admin&quot; --replace_urls False --thread_num 10</span>
Expand Down Expand Up @@ -97,6 +97,8 @@ <h1>Source code for gen3.tools.indexing.index_manifest</h1><div class="highlight
<span class="n">URLS_STANDARD_KEY</span><span class="p">,</span>
<span class="n">AUTHZ_COLUMN_NAMES</span><span class="p">,</span>
<span class="n">AUTHZ_STANDARD_KEY</span><span class="p">,</span>
<span class="n">PREV_GUID_COLUMN_NAMES</span><span class="p">,</span>
<span class="n">PREV_GUID_STANDARD_KEY</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span> <span class="nn">gen3.utils</span> <span class="k">import</span> <span class="p">(</span>
<span class="n">UUID_FORMAT</span><span class="p">,</span>
Expand All @@ -109,6 +111,7 @@ <h1>Source code for gen3.tools.indexing.index_manifest</h1><div class="highlight
<span class="n">_standardize_str</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">import</span> <span class="nn">indexclient.client</span> <span class="k">as</span> <span class="nn">client</span>
<span class="kn">from</span> <span class="nn">indexclient.client</span> <span class="k">import</span> <span class="n">Document</span>


<span class="n">CURRENT_DIR</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">realpath</span><span class="p">(</span><span class="vm">__file__</span><span class="p">))</span>
Expand Down Expand Up @@ -217,6 +220,19 @@ <h1>Source code for gen3.tools.indexing.index_manifest</h1><div class="highlight
<span class="n">f</span><span class="s2">&quot;ERROR: </span><span class="si">{row[current_column_name]}</span><span class="s2"> is not in int format&quot;</span>
<span class="p">)</span>
<span class="n">is_row_valid</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">elif</span> <span class="n">current_column_name</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="n">PREV_GUID_COLUMN_NAMES</span><span class="p">:</span>
<span class="n">fieldnames</span><span class="p">[</span>
<span class="n">fieldnames</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">current_column_name</span><span class="p">)</span>
<span class="p">]</span> <span class="o">=</span> <span class="n">PREV_GUID_STANDARD_KEY</span>
<span class="n">output_column_name</span> <span class="o">=</span> <span class="n">PREV_GUID_STANDARD_KEY</span>
<span class="c1"># only validate format if value is provided (since this is optional)</span>
<span class="k">if</span> <span class="n">row</span><span class="p">[</span><span class="n">current_column_name</span><span class="p">]</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">_verify_format</span><span class="p">(</span>
<span class="n">row</span><span class="p">[</span><span class="n">current_column_name</span><span class="p">],</span> <span class="n">UUID_FORMAT</span>
<span class="p">):</span>
<span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
<span class="n">f</span><span class="s2">&quot;ERROR: </span><span class="si">{row[current_column_name]}</span><span class="s2"> is not in UUID_FORMAT format&quot;</span>
<span class="p">)</span>
<span class="n">is_row_valid</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">elif</span> <span class="n">include_additional_columns</span><span class="p">:</span>
<span class="n">output_column_name</span> <span class="o">=</span> <span class="n">current_column_name</span>

Expand Down Expand Up @@ -381,6 +397,11 @@ <h1>Source code for gen3.tools.indexing.index_manifest</h1><div class="highlight
<span class="k">else</span><span class="p">:</span>
<span class="n">file_name</span> <span class="o">=</span> <span class="s2">&quot;&quot;</span>

<span class="k">if</span> <span class="n">fi</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">PREV_GUID_STANDARD_KEY</span><span class="p">):</span>
<span class="n">prev_guid</span> <span class="o">=</span> <span class="n">fi</span><span class="p">[</span><span class="n">PREV_GUID_STANDARD_KEY</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">prev_guid</span> <span class="o">=</span> <span class="kc">None</span>

<span class="n">doc</span> <span class="o">=</span> <span class="kc">None</span>

<span class="k">if</span> <span class="n">fi</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">GUID_STANDARD_KEY</span><span class="p">):</span>
Expand Down Expand Up @@ -446,8 +467,26 @@ <h1>Source code for gen3.tools.indexing.index_manifest</h1><div class="highlight
<span class="n">URLS_STANDARD_KEY</span><span class="p">:</span> <span class="n">urls</span><span class="p">,</span>
<span class="n">FILENAME_STANDARD_KEY</span><span class="p">:</span> <span class="n">file_name</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">f</span><span class="s2">&quot;creating: </span><span class="si">{record}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">doc</span> <span class="o">=</span> <span class="n">indexclient</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="o">**</span><span class="n">record</span><span class="p">)</span>

<span class="k">if</span> <span class="n">prev_guid</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">f</span><span class="s2">&quot;creating new version of </span><span class="si">{prev_guid}</span><span class="s2">: </span><span class="si">{record}</span><span class="s2">&quot;</span><span class="p">)</span>

<span class="c1"># indexd exports a &quot;form&quot; field that gets populated in indexclient.create,</span>
<span class="c1"># but not indexclient.add_version, need to add manually here</span>
<span class="n">record</span><span class="o">.</span><span class="n">update</span><span class="p">({</span><span class="s2">&quot;form&quot;</span><span class="p">:</span> <span class="s2">&quot;object&quot;</span><span class="p">})</span>

<span class="c1"># to generate new GUID, new version indexd API expects body to not</span>
<span class="c1"># contain &quot;did&quot;, rather than have it be None or &quot;&quot;</span>
<span class="n">new_guid</span> <span class="o">=</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;did&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;did&quot;</span><span class="p">]:</span>
<span class="k">del</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;did&quot;</span><span class="p">]</span>
<span class="n">new_guid</span> <span class="o">=</span> <span class="kc">None</span>

<span class="n">new_doc</span> <span class="o">=</span> <span class="n">Document</span><span class="p">(</span><span class="n">client</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">did</span><span class="o">=</span><span class="n">new_guid</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">record</span><span class="p">)</span>
<span class="n">doc</span> <span class="o">=</span> <span class="n">indexclient</span><span class="o">.</span><span class="n">add_version</span><span class="p">(</span><span class="n">current_did</span><span class="o">=</span><span class="n">prev_guid</span><span class="p">,</span> <span class="n">new_doc</span><span class="o">=</span><span class="n">new_doc</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">f</span><span class="s2">&quot;creating: </span><span class="si">{record}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">doc</span> <span class="o">=</span> <span class="n">indexclient</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="o">**</span><span class="n">record</span><span class="p">)</span>

<span class="n">fi</span><span class="p">[</span><span class="n">GUID_STANDARD_KEY</span><span class="p">]</span> <span class="o">=</span> <span class="n">doc</span><span class="o">.</span><span class="n">did</span>

Expand Down
9 changes: 9 additions & 0 deletions docs/_build/html/genindex.html
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ <h1 id="index">Index</h1>
| <a href="#L"><strong>L</strong></a>
| <a href="#M"><strong>M</strong></a>
| <a href="#O"><strong>O</strong></a>
| <a href="#P"><strong>P</strong></a>
| <a href="#Q"><strong>Q</strong></a>
| <a href="#R"><strong>R</strong></a>
| <a href="#S"><strong>S</strong></a>
Expand Down Expand Up @@ -357,6 +358,14 @@ <h2 id="O">O</h2>
</ul></td>
</tr></table>

<h2 id="P">P</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="tools/indexing.html#gen3.tools.indexing.index_manifest.PREV_GUID">PREV_GUID (in module gen3.tools.indexing.index_manifest)</a>
</li>
</ul></td>
</tr></table>

<h2 id="Q">Q</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
Expand Down
Binary file modified docs/_build/html/objects.inv
Binary file not shown.
Loading

0 comments on commit 7195c2f

Please sign in to comment.