Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Storage] Remove file path cache tech debt from S3SingleDriverLogStore #3685

Conversation

scottsand-db
Copy link
Collaborator

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (Storage)

Description

S3 has been strongly consistent (for GET, PUT, LIST) for years (announcement), yet the S3SingleDriverLogStore has had code that assumes S3 is read-after-write inconsistent.

This PR removes such redundant code.

How was this patch tested?

Existing unit tests.

Does this PR introduce any user-facing changes?

No.

@scottsand-db scottsand-db self-assigned this Sep 17, 2024
@scottsand-db scottsand-db force-pushed the s3_logstore_remove_written_path_cache branch from 94b239a to 5e12b9f Compare September 17, 2024 22:07
@scottsand-db scottsand-db added this to the 3.3.0 milestone Sep 17, 2024
@scottsand-db scottsand-db changed the title [Storage] Remove tech debt from S3SingleDriverLogStore, as S3 has been strongly consistent for years [Storage] Remove file cache tech debt from S3SingleDriverLogStore, as S3 has been strongly consistent for years Sep 17, 2024
@scottsand-db scottsand-db changed the title [Storage] Remove file cache tech debt from S3SingleDriverLogStore, as S3 has been strongly consistent for years [Storage] Remove file path cache tech debt from S3SingleDriverLogStore Sep 17, 2024
Copy link
Collaborator

@vkorukanti vkorukanti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tests are failing. Merging after test fixes, SGTM

@scottsand-db scottsand-db force-pushed the s3_logstore_remove_written_path_cache branch from 4065a9e to d06282e Compare September 18, 2024 20:38
@scottsand-db scottsand-db merged commit 93eef11 into delta-io:master Sep 18, 2024
14 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants