Skip to content

[WIP] Attempt piping through field metadata in as many places as possible #15036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

paleolimbot
Copy link
Member

As discussed in #12644 and on a recent DataFusion sync, a lightweight mechanism to patch in user-defined type support is to use the Field rather than the DataType in as many places as possible.

This is a very in-progress experiment...I know there is also work on LogicalType that may also be able to solve this (as would adding an Extension member to the arrow-rs DataType enum). I figured experimenting in public might be productive but am happy to go back to experimenting in private if that is less confusing!

Which issue does this PR close?

Rationale for this change

Most database systems have the concept of a "user defined type" that can act as first-class types (or close to them). The exact things that can be customized about this vary by system but usually include the ability to define new functions whose signature matches that type, add overloads to existing functions (notably: cast), or sort/display values in a particular way.

Arrow has the concept of an "extension type", which is implemented in the C data interface and IPC formats as two Field metadata fields: ARROW:extension:name and ARROW:extension:metadata. In arrow-rs there is the ExtensionType which provides a means by which to (at least) centralize the serialization and deserialization of ARROW:extension:metadata.

What changes are included in this PR?

Experiments on the way towards replacing DataType with Field where possible.

Are these changes tested?

If this is ever more than just an experiment, they will be!

Are there any user-facing changes?

If this is ever more than just an experiment, there will be!

@github-actions github-actions bot added sql SQL Planner common Related to common crate proto Related to proto crate labels Mar 5, 2025
Copy link

github-actions bot commented May 6, 2025

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the Stale PR has not had any activity for some time label May 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
common Related to common crate logical-expr Logical plan and expressions proto Related to proto crate sql SQL Planner Stale PR has not had any activity for some time
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extension Types / User Defined Types
1 participant