column diffing

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.6.0

Notes to tag/commit: 4.6.1

Generated: 2022-02-03

New Features

• Takes the differential between the matching columns of two manifests of the
  same ext type (#108)
• Important note: rows present in prev manifest but not in succeeding
  manifest will not be in the differential output, those rows are treated as
  dep records (#108)
• Differential is output to a file (#108)

Bug Fixes

• Package contents: the size should be an integer, not any number (#110)

Improvements

• run tests on multiple python versions (#112)
• remove use of := to ensure backwards compatibility to earlier python
  versions (#111)

Package ingestion

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.5.0

Notes to tag/commit: 4.6.0

Generated: 2022-01-21

New Features

• Additional metadata that do not fit in indexd can be submitted to the
  metadata service using the
  gen3.tools.indexing.index_manifest.index_object_manifest function's
  submit_additional_metadata_columns parameter, or the gen3 objects manifest publish CLI command's --submit-additional-metadata-columns flag
  (#107)
• Package metadata can be submitted to the metadata service during indexing,
  by enabling the submission of additional metadata and using record_type = package in the indexing manifest. The package_contents column can be
  used to describe the package (#107)

DRS Manifest/Object Download, Upload File and Delete File

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.4.0

Notes to tag/commit: 4.5.0

Generated: 2021-12-17

New Features

• Adds support for DRS prefix resolution which reduces the number of calls to
  dataguids.org. (#95)
• Provides a means to list access right to commons in a manifest. (#95)
• Add upload_file and delete_files to Gen3File (#94)

Bug Fixes

• remove JSON decode when handling request error (#105)

Improvements

• Refines the listing of manifest and now supports listing bundles much like
  Linux tree (#95)
• Updated documentation and tutorials (#95)
• Update test suite for Gen3File coverage (#94)

Dependency Updates

• Include pytest-cov = "^2.12.1" (#94)

Object Manifest CLI functionality

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.3.0

Notes to tag/commit: 4.4.0

Generated: 2021-12-02

New Features

• objects CLI with support for calling functions to manage file object
  manifests (functions already available in SDK) (#97)
• Add Aggregate metadata service support for discovery read: --agg flag to
  enable (#99)
• Adding a configure command which lets the user add a profile name and
  attach a credentials file (#93)

Improvements

• Print the API call output when unable to parse it to JSON (#102)
• The "Gen3File.get_presigned_url" function does not have a default
  "protocol" anymore. If no protocol is specified, rely on Fence and Indexd's
  default behavior, which is to use the first URL they find (#91)

Discovery Metadata Functions and Improved Manfiest Merging

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.2.0

Notes to tag/commit: d82eeea

Generated: 2021-05-10

New Features

• ability to specify columns that should be treated like arrays (so values
  are combined correctly from various sources) during bucket manifest merging
  (#85)
• add discovery read, discovery publish, delete sub commands to CLI (#83)

Breaking Changes

• Commas are no longer allowed as additional delimiters in a space-delimited
  array for indexing manifests (this was causing a bug for valid file names
  containing commas) (#85)

Bug Fixes

• Include retry logic for caching refreshed access tokens. (#88)
• Fix merging edge cases where multiple rows of same hash, diff guids,
  weren't merging data as expected (#85)
• Fixed issue where an indexing manifest was considered invalid if it had
  commas in filenames in the URL column (#85)
• manifest and ingestion tools break with commas in file names, not anymore
  (#84)
• fix bug where certain fields are not dicts, but we're generically trying to
  do a comparison. AttributeError: 'str' object has no attribute 'keys'
  before, now downstream function handles this case by doing a generic ==
  comparison (#82)
• requests.HTTPError not available in utils b/c missing import (#81)

Improvements

• Added notes for Python set up and manual testing in
  devTest.md
  (#88)
• Additional unit tests for Gen3Auth / Gen3Submission (#88)
• refactor merge code to be more readable and handle edge cases (#85)
• Unit tests for indexing manifest functionality better reflect edge cases
  around spaces & commas in URLs, duplicates, etc. (#85)
• breakout utility function for getting urls from manifest string (#84)
• Functions print error details instead of just exiting when a request fails
  (#79)

Dependency Updates

• Include using backoff lib (#88)

support prev_guid in indexing tooling

4.2.0

feat(indexing): new optional prev_guid column support to create new v…

Gen3Query module

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.0.0

Notes to tag/commit: 4.1.0

Generated: 2021-01-13

New Features

• Add Gen3Query module with "query", "graphql_query" and "raw_data_download"
  functions (#70)

Gen3 CLI + Gen3WsStorage module

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 3.1.0

Notes to tag/commit: 4.0.0

Generated: 2021-01-13

Improvements

• use GEN3_API_KEY environment variable - default to
  ~/.gen3/credentials.json (#72)
• derive endpoint from api key (#72)
• add access-token cache (#72)
• add access-token expiration check (#72)
• add workspace storage class and cli integration (#72)
• add auth cli integration (#72)
• arrange markdown docs (#72)
• add Jenkins smoke test (#72)

"delete_records", "delete_node" and "delete_nodes" functions

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 3.0.0

Notes to tag/commit: 3.1.0

Generated: 2020-12-14

New Features

• Add Gen3Submission "delete_records", "delete_node" and "delete_nodes"
  functions (#68)

Improvements

• fix auto generation of docs (#64)

New build tool, auto PyPI deploys, skeleton for CLI

Release Notes

For: uc-cdis/gen3sdk-python
Notes since tag: 2.7.0
Notes to tag/commit: 3.0.0
Generated: 2020-12-03

New Features

• initial CLI skeleton (#63)
• CLI automatically supports pypfb and subcommands with "gen3 pfb" (#63)
• Added ingestion tool for bundles given a manifest. (#60)

Improvements

• use poetry (#63)
• automatic PyPI deploys on master, automatic TestPyPI deploys on branches
  (#63)
• "Gen3Submission.submit_file" improvements: do not fail silently in case of
  unexpected status code, add logs to troubleshoot failures (#59)

Dependency Updates

• updated to latest versions of many deps (#63)