Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 1357 - Bulk import with S3 state store #1362

Merged
merged 18 commits into from
Sep 29, 2023
Merged

Issue 1357 - Bulk import with S3 state store #1362

merged 18 commits into from
Sep 29, 2023

Conversation

patchwork01
Copy link
Collaborator

@patchwork01 patchwork01 commented Sep 25, 2023 •
edited
Loading

Still need to add a Maven property to set state store type for system tests.

Make sure you have checked all steps below.

Issue

Tests

  • My PR adds the following tests OR does not need testing for this extremely good reason:
    • Covered by existing tests
    • Ran system tests in AWS

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
    separate issue for that below.
  • If I have added, removed, or updated any external dependencies used in the project, I have updated the
    NOTICES file to reflect this.

patchwork01 and others added 14 commits September 25, 2023 12:16
@patchwork01
Run system tests against S3 state store
06fe51f
@patchwork01
Use HadoopConfigurationProvider in BulkImportStarterLambda
623f2f9
@patchwork01
Use HadoopConfigurationProvider in BulkImportJobDriver
4460917
@patchwork01
Set system test state store at deploy time
c0eb6c5
@patchwork01
Remove StateStoreProvider constructor without Hadoop conf
2673199
@patchwork01
Merge remote-tracking branch 'origin/main' into 1357-bulk-import-s3-s…
d9127c7 
…tate-store

# Conflicts:
#	java/query/src/test/java/sleeper/query/lambda/SqsQueryProcessorLambdaIT.java
@patchwork01
Allow reinitialising state store with no files
2779574
@patchwork01
Grant access to data bucket for SleeperTableLambda
8580b4c
@patchwork01
Run system tests against S3 state store nightly
5a4b6b4
@kr565370
Handle null keys in KeySerDe
6d1ff04
@patchwork01
Fix SleeperTableLambdaIT
e495cc2
@patchwork01
Fix QueryExecutorIT
ec375b6
@kr565370
Merge remote-tracking branch 'origin/main' into 1357-bulk-import-s3-s…
f3d15d1 
…tate-store
@patchwork01
Set lastStateStoreUpdateTime for files in bulk import
cef78bc
@patchwork01 patchwork01 marked this pull request as ready for review September 28, 2023 11:18
@kr565370 kr565370 removed their assignment Sep 28, 2023
@patchwork01 patchwork01 added the pr-base-for-stacking Base for stacked pull requests (a dependency for others) label Sep 28, 2023
This was referenced Sep 29, 2023
Merged
Merged
@gaffer01 gaffer01 removed their assignment Sep 29, 2023
@patchwork01 patchwork01 merged commit ac9934a into main Sep 29, 2023
15 checks passed
@patchwork01 patchwork01 deleted the 1357-bulk-import-s3-state-store branch September 29, 2023 12:17
@patchwork01 patchwork01 mentioned this pull request Sep 29, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kr565370 kr565370 kr565370 approved these changes

@gaffer01 gaffer01 gaffer01 approved these changes

Assignees
No one assigned
Labels
pr-base-for-stacking Base for stacked pull requests (a dependency for others)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

NPE when using EMR serverless bulk import into a table using the S3 state store
3 participants
@patchwork01 @gaffer01 @kr565370