Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Introduce withReadType in ReadBuilder #4214

Merged
merged 10 commits into from
Sep 25, 2024

Conversation

Zouxxyy
Copy link
Contributor

@Zouxxyy Zouxxyy commented Sep 19, 2024

Purpose

To #4209

public API

 /**
   * Push row type to the reader, support nested row pruning.
   *
   * @param readType read row type, can be a subset of {@link Table#rowType()}
   * @since 1.0.0
   */
  ReadBuilder withReadType(RowType readType);

mark ReadBuilder withProjection(int[][] projection) and ReadBuilder withProjection(int[] projection) as Deprecated

how to use

RowType writeType =
        DataTypes.ROW(
                DataTypes.FIELD(0, "pt", DataTypes.INT()),
                DataTypes.FIELD(1, "a", DataTypes.INT()),
                DataTypes.FIELD(2, "f0", DataTypes.INT()),
                DataTypes.FIELD(
                        3,
                        "f1",
                        DataTypes.ROW(
                                DataTypes.FIELD(4, "f0", DataTypes.INT()),
                                DataTypes.FIELD(5, "f1", DataTypes.INT()),
                                DataTypes.FIELD(6, "f2", DataTypes.INT()))));
// write
// GenericRow.of(0, 0, 0, GenericRow.of(10, 11, 12))


RowType readType =
        DataTypes.ROW(
                DataTypes.FIELD(
                        3,
                        "f1",
                        DataTypes.ROW(
                                DataTypes.FIELD(4, "f0", DataTypes.INT()),
                                DataTypes.FIELD(6, "f2", DataTypes.INT()))));

ReadBuilder readBuilder = table.newReadBuilder().withReadType(readType);

// read
// GenericRow.of(GenericRow.of(10, 12))

@Zouxxyy Zouxxyy force-pushed the variant/withRowTypeProjection branch from d587353 to 3259d18 Compare September 19, 2024 05:11
@JingsongLi
Copy link
Contributor

We just need a pruneColumns(RowType requiredSchema).

@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Sep 20, 2024

RowType contains all the information (field name, field id, nested structure ... ), it can replace projection

The final API will be modified to like this

    @Deprecated
    default ReadBuilder withProjection(int[] projection) {
        // projection -> requiredSchema
        return pruneColumns(RowType requiredSchema);
    }

    ReadBuilder pruneColumns(RowType requiredSchema);

@Zouxxyy Zouxxyy force-pushed the variant/withRowTypeProjection branch from e2339eb to 2102046 Compare September 22, 2024 10:48
@Zouxxyy Zouxxyy changed the title [WIP] Introduce RowTypeProjection for ReadBuilder [core] Introduce withReadType in ReadBuilder Sep 22, 2024
@Zouxxyy Zouxxyy force-pushed the variant/withRowTypeProjection branch from 2102046 to f14be50 Compare September 23, 2024 09:20
@Zouxxyy Zouxxyy force-pushed the variant/withRowTypeProjection branch from 8dd3634 to ddb6b42 Compare September 24, 2024 03:53
@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Sep 24, 2024

@JingsongLi Thanks for review, updated

Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit cba8447 into apache:master Sep 25, 2024
10 checks passed
@Zouxxyy Zouxxyy deleted the variant/withRowTypeProjection branch October 8, 2024 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants