Skip to content

[datafusion-spark] Test integrating datafusion-spark code into comet #1704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tracked by #15914
alamb opened this issue May 2, 2025 · 1 comment
Open
Tracked by #15914

[datafusion-spark] Test integrating datafusion-spark code into comet #1704

alamb opened this issue May 2, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented May 2, 2025

What is the problem the feature request solves?

@shehabgamin added the datafusion-spark crate in apache/datafusion#15168

The goal is to help centralize the development of this function library in the core repository rather than duplicated effort

To verify this is feasible I would like to verify that the setup created in datafusion-spark / apache/datafusion#15168 can be used by downstream crates like comet before we go too far

Describe the potential solution

I would like someone to make a PR that shows we can remove one or more of the spark function in datafusion-spark can be used in comet (and reduce the code here)

Additional context

cc @comphead @andygrove @mbutrovich

@andygrove
Copy link
Member

I created apache/datafusion#15947 to add Comet's hex function to datafusion-spark. This involved converting it from a PhysicalExpr to a ScalarUDFImpl.

Also, I created #1711 to use the expm1 function from datafusion-spark in Comet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants