Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v24.3.x] iceberg: fix spec inconsistency in manifest list files_count #24650

Open
wants to merge 2 commits into
base: v24.3.x
Choose a base branch
from
Open

[v24.3.x] iceberg: fix spec inconsistency in manifest list files_count #24650

wants to merge 2 commits into from
+68 −37

Conversation

vbotbuildovich
Copy link
Collaborator

@vbotbuildovich vbotbuildovich commented Dec 23, 2024 •
edited by andrwng
Loading

Backport of PR #24602

  • Added 7ec7c6d
  • Removed the upgrade test since 24.3 is the first version with Iceberg integration

@vbotbuildovich vbotbuildovich added this to the v24.3.x-next milestone Dec 23, 2024
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label Dec 23, 2024
@vbotbuildovich
Copy link
Collaborator Author

vbotbuildovich commented Dec 24, 2024 •
edited
Loading

Retry command for Build#60094

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_upload_after_external_update@{"cloud_storage_type":1}

@vbotbuildovich
Copy link
Collaborator Author

vbotbuildovich commented Dec 24, 2024 •
edited
Loading

CI test results

test results on build#60094
test_id test_kind job_url test_status passed
gtest_raft_rpunit.gtest_raft_rpunit unit https://buildkite.com/redpanda/redpanda/builds/60094#0193f598-ae63-4c65-8cb0-41cc29ff82dc FLAKY 1/2
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_upload_after_external_update.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60094#0193f5de-b7e9-4e27-a0e9-fb13bbb12a81 FAIL 0/1
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_upload_after_external_update.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60094#0193f5f1-6ec6-4417-93a7-6d4b32580e2b FAIL 0/1
test results on build#60133
test_id test_kind job_url test_status passed
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a19-49d2-8d6c-3202b4822d74 FAIL 0/1
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a18-4539-9613-5fae27a32dda FAIL 0/1
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a19-49d2-8d6c-3202b4822d74 FAIL 0/1
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a18-4539-9613-5fae27a32dda FAIL 0/6

mmaslankaprv and others added 2 commits December 24, 2024 10:52
@mmaslankaprv @andrwng
tests: added escaping table names in queries
7ec7c6d 
Some characters that may be used in topic names must be treated specialy
in SQL. Table names must be quoted when they contains the special
character. Quotation mark is specific to query engine. Added quotation
for table names for query engines used in tests.

Signed-off-by: Michał Maślanka <[email protected]>
(cherry picked from commit 8fe08cd)
@andrwng
iceberg: fix inconsistency in manifest list files_count
d9c34cb 
The schema we are using was pulled some time ago and appears to be
outdated. The Apache Iceberg Java implementation has since renamed
added_data_files_count and friends to added_files_count, to match the
documented spec.

This meant that after updating the table with an external non-Redpanda
writer, Redpanda wouldn't be able to download the current manifest list
when appending and get stuck, complaining about an EOF (presumably the
Avro C++ library throws this when there is an unknown field).

I suspect that this may have also been the cause of an EOF seen when
trying to read a manifest list with BigQuery:

Error while reading data, error message: The Apache Avro failed to read data with the following error: EOF reached File: [...]/metadata/snap-[...]-0.avro

The old names are added as an alias to ensure Redpanda can still
download Iceberg manifest lists from 24.3.

(cherry picked from commit a3e6880)
@andrwng andrwng force-pushed the backport-pr-24602-v24.3.x-82 branch from 038e933 to 2cfb307 Compare December 24, 2024 18:52
@vbotbuildovich
Copy link
Collaborator Author

Retry command for Build#60129

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}

@andrwng andrwng force-pushed the backport-pr-24602-v24.3.x-82 branch from 2cfb307 to d9c34cb Compare December 24, 2024 22:35
@vbotbuildovich
Copy link
Collaborator Author

vbotbuildovich commented Dec 25, 2024 •
edited
Loading

Retry command for Build#60133

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrwng andrwng Awaiting requested review from andrwng

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
area/redpanda kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v24.3.x-next
Development

Successfully merging this pull request may close these issues.
3 participants
@vbotbuildovich @mmaslankaprv @andrwng