Add failed SIPs/PIPs buckets #930
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
c8295ff to
2326b10
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #930 +/- ##
==========================================
- Coverage 55.52% 54.62% -0.90%
==========================================
Files 103 103
Lines 6039 7506 +1467
==========================================
+ Hits 3353 4100 +747
- Misses 2433 3158 +725
+ Partials 253 248 -5 ☔ View full report in Codecov by Sentry. |
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
fd19cff to
d7e12ce
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
3 times, most recently
from
2374d29 to
d6dc096
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
07e2db8 to
0ea4885
Compare
jraddaoui
changed the title
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
0ea4885 to
5aa84f6
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
74ef2d2 to
afeef67
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
f427963 to
cf5fc14
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
5d1cd0a to
5ef0ba9
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
6294306 to
e53ab9b
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
3 times, most recently
from
41e8e38 to
b4ffc28
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
b4ffc28 to
fda1bee
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
628474b to
713caa2
Compare
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
2 times, most recently
from
a86f330 to
6627229
Compare
jraddaoui
changed the title
jraddaoui
commented
Oct 17, 2024
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
6627229 to
43b48df
Compare
jraddaoui
requested review from
djjuhasz and
sevein
and removed request for
djjuhasz
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
43b48df to
dd678d1
Compare
djjuhasz
approved these changes
Oct 17, 2024
There was a problem hiding this comment.
Thanks @jraddaoui, this looks good to me and I don't see any blockers. I did make a few comments about how we might want to manage tinfo going forward. I also had a couple of questions about how we manage SIPs in the "sips" bucket.
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
040ab12 to
668d564
Compare
djjuhasz
approved these changes
Oct 18, 2024
If there is an error before the SIP is sent to preservation, copy the downloaded SIP file to a failed SIPs bucket. If there is an error on preservation, send a copy of the PIP to a failed PIPs bucket, with all the transformations made before it was sent to preservation. - Add failed buckets configuration. - Use temporal-activities/bucketupload registered with two different names for each failed bucket. - Modify Kubernetes jobs to set up and re-create buckets. [skip-codecov]
jraddaoui
force-pushed
the
dev/issue-929-failed-buckets
branch
from
bb302e0 to
f995a8f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If there is an error before the SIP is sent to preservation, copy the
downloaded SIP file to a failed SIPs bucket. If there is an error on
preservation, send a copy of the PIP to a failed PIPs bucket, with all
the transformations made before it was sent to preservation.
names for each failed bucket.
Refs #929.