Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update cleanup job to scrub ensemble data #833

Closed
WalterKolczynski-NOAA opened this issue Jun 1, 2022 · 1 comment · Fixed by #2919
Closed

Update cleanup job to scrub ensemble data #833

WalterKolczynski-NOAA opened this issue Jun 1, 2022 · 1 comment · Fixed by #2919
Assignees
@AntonMFernando-NOAA
Labels
feature New feature or request

Comments

@WalterKolczynski-NOAA
Copy link
Contributor

Description
As forecast ensemble jobs are added to the global workflow, we need to ensure the output is being cleaned up properly once it is no longer needed.

Requirements
The cleanup job (currently part of archive) needs to properly scrub ensemble data once it is has been archived and is no longer needed. Until ensemble verification is added as part of the workflow instead of a stand-alone system, this may involve a smaller set of data than current deterministic, which does verification within the workflow.

This will be an ongoing issue completed incrementally as pieces of the forecast ensemble are added.

Acceptance Criteria (Definition of Done)
A cleanup job that properly removes old ensemble data once it is no longer needed.

Dependencies
Add atm-only ensemble free-forecast (#822)
Add atm ensemble product generation (#823)
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the feature New feature or request label Jun 1, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA added this to the Incorporate GEFS milestone Jun 1, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA removed this from the Incorporate GEFS milestone Sep 3, 2024
@WalterKolczynski-NOAA
Copy link
Contributor Author

I think after the changes in #2753, the only thing that is needed for this is to add the cleanup job to the rocoto mesh.

@AntonMFernando-NOAA AntonMFernando-NOAA mentioned this issue Sep 13, 2024
Merged
7 tasks
aerorahul pushed a commit that referenced this issue Sep 14, 2024
@AntonMFernando-NOAA
Cleanup job for GEFS (#2919)
b7f71e0 
As forecast ensemble jobs are added to the global workflow, this PR
ensures the output is being cleaned up properly once it is no longer
needed.
  Resolves #833
bbakernoaa pushed a commit to bbakernoaa/global-workflow that referenced this issue Sep 19, 2024
@AntonMFernando-NOAA @bbakernoaa
Cleanup job for GEFS (NOAA-EMC#2919)
b22d7bc 
As forecast ensemble jobs are added to the global workflow, this PR
ensures the output is being cleaned up properly once it is no longer
needed.
  Resolves NOAA-EMC#833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AntonMFernando-NOAA AntonMFernando-NOAA

Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Cleanup job for GEFS AntonMFernando-NOAA/global-workflow
2 participants
@WalterKolczynski-NOAA @AntonMFernando-NOAA