Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest, backend): do not approve curated revisions (by users other than insdc_ingest_user), send notification if ingest wants to revise a curated sequence #3124

Merged
merged 7 commits into from
Oct 31, 2024

Conversation

anna-parker
Copy link
Contributor

@anna-parker anna-parker commented Oct 30, 2024

resolves #3087, #3084, #2613

partially: #3085,

preview URL: https://insdc-curation-ingest-qui.loculus.org/

Summary

  1. Ingest no longer approves all revisions in the insdc_ingest_group but only revisions it submitted (e.g. user= insdc_ingest_user) - this allows curators to revise a sequence and not have it be approved directly by ingest. In order to do this the backend now allows filtering by submitter not just submissionGroup
  2. Send slack notification if ingest wants to revise a curated sequence - see if revision is in a sequence where any version was submitted by a user other than insdc_ingest_user - in this case send a notification and put the sequence into state blocked. Again the backend must be modified to also return the submitter of the original-metadata.

Screenshot

PR Checklist

micromamba activate pp-integrity
cd tests/regression-testing 
snakemake results/ebola-zaire.meta.main.insdc-curation-ingest-qui.diff 
snakemake results/cchf.meta.main.insdc-curation-ingest-qui.diff 
snakemake results/west-nile.meta.main.insdc-curation-ingest-qui.diff 

There is no diff in the ebola-zaire, cchf and west nile metadata.

  • Curate three sequences via the revise page as super-user: curate metadata, curate sequence, curate metadata and sequence -> check these are not overwritten by ingest
  • Make sure ingest doesn't approve curated sequences
  • Connect to preview db - change original data hash of a curated sequence (three cases as above), make sure that we get a notification that ingest doesn't know what to do and will not revise these sequences.
    I see this in logs:
15:24:26  WARNING (   compare_hashes.py:  63) - Sequence MZ424862 has been curated before - do not know how to proceed

Somehow the slack hook was removed from the config, I had to add it back. Now we are notified:
image

  • Double check slack hook is still in pathoplexus ingest config

@anna-parker anna-parker changed the base branch from insdc_approve_ingest_quick_fix to main October 30, 2024 08:45
@anna-parker anna-parker force-pushed the insdc_curation_ingest_quick_fix branch from c2a6788 to 93d9864 Compare October 30, 2024 08:51
@anna-parker anna-parker added the preview Triggers a deployment to argocd label Oct 30, 2024
@anna-parker
Copy link
Contributor Author

anna-parker commented Oct 30, 2024

Testing:

  1. LOC_000373F.1: curated metadata -> stable
  2. LOC_000372H.1: curated sequence -> stable
  3. LOC_00036WV.1: both -> stable
  4. LOC_000RSPC.2: both also multi-segmented -> stable
  5. Curated LOC_000371K.1 but did not approve - should not show up
  6. Then edited hash of LOC_000373F and LOC_000RSPC for v1 and v2 in db to test notification system for single and multi-segmented
  7. Modify hash of non-curated LOC_000RSNE and LOC_00036ZP to trigger v2

…ed revisions (by users other than insdc_ingest_user), send notification if ingest wants to revise a curated sequence.
@anna-parker anna-parker force-pushed the insdc_curation_ingest_quick_fix branch from fd6f342 to 2c93dea Compare October 30, 2024 15:52
@anna-parker anna-parker marked this pull request as ready for review October 30, 2024 16:31
@anna-parker anna-parker changed the title Quick curation fix: allow revisions via webpage, do not approve curated revisions (by users other than insdc_ingest_user), send notification if ingest wants to revise a curated sequence. feat(ingest, backend): do not approve curated revisions (by users other than insdc_ingest_user), send notification if ingest wants to revise a curated sequence Oct 30, 2024
@corneliusroemer
Copy link
Contributor

Very cool stuff, this was on my backlog and I'm very grateful you've done already so much more! Still reviewing, but wanted to ask what you think about adding/modifying one or two backend tests to make the new behaviour tested?

Copy link
Contributor

@corneliusroemer corneliusroemer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, backend changes are very minimal, would just be good to keep things tested in backend.

Python is good, I didn't verify things super closely, but I trust your testing, the changes seem to be fairly small, mostly a refactor - the new logic is very localized I think, simply bailing when a sequence was ever touched by non-insdc_ingest_user.

@anna-parker anna-parker merged commit ecc3ee9 into main Oct 31, 2024
18 checks passed
@anna-parker anna-parker deleted the insdc_curation_ingest_quick_fix branch October 31, 2024 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Triggers a deployment to argocd
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ingest should not auto-approve submissions that weren't submitted by "insdc_ingest_user"
2 participants