Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Read iceberg v2 table which has equality deletes with nested fields failed #18625

Open
Heltman opened this issue Aug 10, 2023 · 7 comments · May be fixed by #24512
Open

Read iceberg v2 table which has equality deletes with nested fields failed #18625

Heltman opened this issue Aug 10, 2023 · 7 comments · May be fixed by #24512
Assignees
@ebyhr
Labels
bug Something isn't working iceberg Iceberg connector

Comments

@Heltman
Copy link
Contributor

Heltman commented Aug 10, 2023

Iceberg v2 table with equality deletes should support nested fields as delete column, but when read, it will throw error like below:

2023-08-09T15:16:47.338-0600	ERROR	stage-scheduler	io.trino.execution.StageStateMachine	Stage 20230809_211647_00008_s6wq4.0 failed
io.trino.spi.TrinoException: Multiple entries with same key: 3=f1 and 3=root.f1
	at io.trino.plugin.iceberg.IcebergPageSource.getNextPage(IcebergPageSource.java:142)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:299)
	at io.trino.operator.Driver.processInternal(Driver.java:395)
	at io.trino.operator.Driver.lambda$process$8(Driver.java:298)

For example:

-- create schema below
CREATE TABLE iceberg.default.test (id BIGINT, root ROW(nested BIGINT, nested_other BIGINT);
-- delete by `nested` to generate equality deletes
...
-- query table throw exception
SELECT * FROM iceberg.default.test;
@Heltman
Copy link
Contributor Author

Heltman commented Aug 10, 2023

See: #17115 (comment)
find by @findinpath

@Heltman Heltman mentioned this issue Aug 10, 2023
Merged
@findinpath findinpath added bug Something isn't working iceberg Iceberg connector labels Aug 10, 2023
@ebyhr
Copy link
Member

ebyhr commented Aug 10, 2023

-- delete by nested to generate equality deletes

Please share the exact steps to reproduce.

@findinpath
Copy link
Contributor

findinpath commented Aug 10, 2023 •
edited
Loading

@Heltman can you pls add a test PR to showcase this limitation?

BTW, it may be related to

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

Lines 253 to 255 in cd56ce6

deleteFilterRequiredColumns.stream()
.filter(not(icebergColumns::contains))
.forEach(requiredColumns::add);

@Heltman
Copy link
Contributor Author

Heltman commented Aug 10, 2023

Can we use sql to write a delete file to iceberg v2 table? Or I just add a unit test for this problem? @findinpath

@findinpath
Copy link
Contributor

Or I just add a unit test for this problem?

unit test will be just fine - similar to #17115 (comment)

so that we can showcase that the current Trino code has an issue.

@findinpath
Copy link
Contributor

@Heltman is this still relevant?

@ebyhr
Copy link
Member

ebyhr commented Dec 12, 2024

This issue still remains.

@ebyhr ebyhr self-assigned this Dec 16, 2024
@ebyhr ebyhr marked this as a duplicate of #24488 Dec 16, 2024
This was referenced Dec 16, 2024
Closed
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ebyhr ebyhr

Labels
bug Something isn't working iceberg Iceberg connector
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix failure when equality delete updated nested fields in Iceberg ebyhr/trino
3 participants
@ebyhr @Heltman @findinpath