Allow fetching previously built artifacts from S3 #43
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Closing this as the main issue was mitigated by #57. If we decide to revisit this, we should open a new PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION
This gives us the ability to fetch artifacts for some packages from a previous build.
When we do so is dictated by the REUSE_CACHED_ARTIFACTS policy, which can have 3 possible values:
Note that if we pass custom parameters for a package (e.g. provide a custom git url or branch), then we will never re-use artifacts from a previous build.
The "previous build" URL will be passed by Jenkins (devops-gate change to be posted soon) and is the latest post-push build.
Note that linux-pkg will not attempt to compare the hash of previous packages to determine whether or not it should be rebuilt, nor will it care if linux-pkg itself has changed. This simplifies the logic a lot but puts the burden or determining when it is safe to use each setting on the developer that tests the changes.
MOTIVATION
We've had intermittent external dependency errors such as #20 and #7 that prevent developers from testing their changes in other packages. This helps developers to avoid being affected by those errors unless they are making changes to the affected packages.
This will also give the ability to manually launch a post-push job with the "on-failure" setting, to avoid blocking teams that are waiting on some changes to be rolled-out.
By default, git-ab-pre-push will set the "on-failure" setting. As a bonus, we also have the "always" setting, which significantly speeds-up builds. We may decide to use this as the default setting if we deem it safe enough. In any case, we will add an option to pass this setting to git-ab-pre-push.
TODO
FOLLOW-UP WORK
TESTING
"always": http://collins.d.delphix.com:21968/job/devops-gate/job/master/job/linux-pkg-build/job/master/job/userland/job/pre-push/11/consoleFull
"always", 1 custom package: http://collins.d.delphix.com:21968/job/devops-gate/job/master/job/linux-pkg-build/job/master/job/userland/job/pre-push/13
"on-failure": http://collins.d.delphix.com:21968/job/devops-gate/job/master/job/linux-pkg-build/job/master/job/userland/job/pre-push/14/parameters/
"never": http://collins.d.delphix.com:21968/job/devops-gate/job/master/job/linux-pkg-build/job/master/job/userland/job/pre-push/16/parameters/
fixes #27