Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't rerun unchanged recipes #3

Open
cisaacstern opened this issue Apr 25, 2022 · 2 comments
Open

Don't rerun unchanged recipes #3

cisaacstern opened this issue Apr 25, 2022 · 2 comments

Comments

@cisaacstern
Copy link
Member

cisaacstern commented Apr 25, 2022

User Profile

As a recipe maintainer

User Action

I want to push commits to the default branch of my feedstock repository, and have the resulting production deployment only rerun new recipes or those that have changed, and not rerun unchanged recipes

User Goal

So that I can add or update certain recipes in my feedstock without rerunning all of them.

Acceptance Criteria

A mechanism to check the hash of all recipes at deployment time, and skip re-running if the hash matches the hash for the same recipe in the last production deployment

Linked Issues

In the order in which they should be merged:

  1. Recipe hashes pangeo-forge-recipes#349
  2. Add recipe and pattern hashes to RecipeRun table pangeo-forge-orchestrator#63
  3. https://github.com/pangeo-forge/registrar/pull/36
@cisaacstern
Copy link
Member Author

Just noting that the main blocker to implementing this feature a new of release pangeo-forge-recipes (which will include pangeo-forge/pangeo-forge-recipes#349).

@sharkinsspatial, let's sync next week to determine if it's we want to just manually build bakery images for this upcoming release, or if we'd like to use this opportunity to start working on some of the items in #9.

@cisaacstern
Copy link
Member Author

With pangeo-forge/pangeo-forge-recipes#359, we arguably no longer need pangeo-forge/pangeo-forge-orchestrator#63, which would duplicate this information into the database/API. This would slightly reduce the number of lines required to retrieve this information, but at the cost of additional complexity in the database (a PR/migration would be required) and, perhaps a greater cost, the risk of the two storage locations (dataset and database) falling out of sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

2 participants