Bug fixes and improved logs

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 3.1.0

Notes to tag/commit: 3.1.1

Generated: 2021-05-04

Bug Fixes

• Fix bug when submitting array items as TSV (#346)
• Fix bug when exporting array items to TSV (#346)
• Fix bug when reporting array items errors to users (#346)

Improvements

• Add logging to facilitate debugging validation issues (#346)
• Improve program/project creation error messages (#345)

Dependency Updates

• gdcdatamodel to 3.0.3 (#346)
• Bumps pyyaml from 5.1 to 5.4. (#343)
• psqlgraph: up to 3.0.1 (#344)

TSV/JSON export updates

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 3.0.0

Notes to tag/commit: 3.1.0

Generated: 2020-01-31

Improvements

• The formats of exported and imported files (JSON and TSV) now match (breaking change) (#318)

Fix mapping files for any node category ending in "_file"

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 4.1.1

Notes to tag/commit: 6289f43

Generated: 2020-11-06

Bug Fixes

• Fix mapping files for any node category ending in "_file" (#334)
• Fix duplication of application logging messages by not propagating (#334)
• Accept TSV submissions with empty link columns, for links that are not
  required (#331)
• Only write acl and authz fields of corresponding Indexd record during data
  submission iff both fields are empty to begin with. Prevents authz field
  from being unintentionally populated in commons not using authz. (#330)

Improvements

• Set application log level based on
  GEN3_DEBUG
  environment variable (#334)
• Existing links can be removed by explicitly setting them to None in
  submissions (use null in JSON, empty values in TSV/CSV) (#331)

Dependency Updates

• dictionaryutils to 3.4.0 (#331)

Breaking Changes

• Note that this PR will cause Sheepdog behavior to change slightly during a
  data submission flow in which the acl and/or authz field are previously
  populated in the corresponding Indexd record and the Indexd uploader field
  is not null and the Sheepdog submitter does not match the Indexd uploader.
  While the existing behavior will cause data submission to fail and report
  an error message to the user, this PR’s changes will mean that data
  submission can still succeed in this case of populated acl/authz and
  uploader mismatch. I believe this change in behavior is not problematic in
  that it would be unexpected in any commons for uploader and (acl and/or
  authz) to both be populated. (#330)

New export with streaming

New export with streaming for large number of items

Bug Fixes

• Using streaming to export large amount of data

Ability to null values

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 4.0.1

Notes to tag/commit: 4.1.0

Generated: 2020-04-29

New Features

• Implemented the ability to remove values from fields that are not required
  in existing entities (use null in JSON, empty values in TSV/CSV). (#320)

Dependency Updates

• dictionaryutils to 3.1.0 (#320)

Fix None export

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 4.0.0

Notes to tag/commit: 4.0.1

Generated: 2020-04-22

Bug Fixes

• When exporting entities as a tsv file Null values would have 'None' instead
  of an empty string. This would cause an error when resubmitting. Fixed by
  checking file format and making Null values an empty string when the format
  is tsv. (#323)
• Fix sheepdog X.0 integer handling (#322)

Dependency Updates

• pin cryptography 2.8 b/c of known bug with 2.9 (#324)

Use Arborist admin policies

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 3.1.0

Notes to tag/commit: 4.0.0

Generated: 2020-03-24

New Features

• Switch checks guarding program/project CRUD to centralized auth (instead of
  checking for "admin" in JWT, check for Sheepdog admin policy in Arborist)
  (#319)

Breaking Changes

• Having admin: true in user.yaml no longer allows a user to do
  program/project CRUD in Sheepdog. Users now need to have the Sheepdog admin
  policy in Arborist. (#319)

Deployment Changes

• The environment's user.yaml needs to be updated to add the new Sheepdog
  admin policy/resources/roles; for each user that has admin: true, grant
  them the new policy. See uc-cdis/commons-users#810
  for an example. (#319)

Python 3 Release

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 2.1.2

Notes to tag/commit: 3.0.0

Generated: 2020-01-06

New Features

• Updated Dockerfile to use Python3 (#314)
• ran 2to3 for syntax changes (#314)
• updated travis to ignore py2.7 failures (#314)
• Use special .secinclude file to list source code for Veracode security
  scanning (#313)
• Use special .secexclude to forcefully exclude some files (empty) (#313)

Breaking Changes

• Requires Python 3 (#314)
• Updates Sheepdog to use new Indexd aliases endpoint rather than deprecated
  aliases endpoint (uc-cdis/indexd#248) (#314)

Dependency Updates

• bumped indexd to 2.3.0 (#314)
• updated indexclient to 2.0.0 (#314)
• psqlgraph to
  release/py3
  branch to be compatible to python3 (#314)
• Updated lxml from 3.8.0 -> 4.4.0 (versions <4.0.0 are not compatible with
  python3) (#314)

Deployment Changes

• Fix Travis coverage reports (combine the results of all 3 tests into one
  coverage report). (#314)

Improvements

• When trying to submit a metadata record to a node that's not in the
  dictionary, user will receive an explanatory error message (#316)
• Dependabot creates public PRs for security updates. Disabling it. (#315)

2.1.2

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 2.1.1

Notes to tag/commit: 2.1.2

Generated: 2019-10-10

Bug Fixes

• Fix validate_request bug when AUTH_SUBMISSION_LIST config is not set or
  set to True (#312)

Improvements

• Make DependaBot only update to "security" updates (#311)
• Setup Dependabot automatic security scanning (#305)

Get ARBORIST_URL from manifest

Release Notes

For: uc-cdis/sheepdog

Notes since tag: 2.1.0

Notes to tag/commit: 2.1.1

Generated: 2019-10-04

Deployment Changes

• ARBORIST_URL environment variable replaces broken default ARBORIST config
  variable (#304)