Skip to content

[datafusion-spark] Implement ceil function #15916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tracked by #15914
alamb opened this issue May 2, 2025 · 4 comments · May be fixed by #15958
Open
Tracked by #15914

[datafusion-spark] Implement ceil function #15916

alamb opened this issue May 2, 2025 · 4 comments · May be fixed by #15958
Assignees
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented May 2, 2025

Is your feature request related to a problem or challenge?

Given the importantance of spark functions in general, we are consolidating / adding spark compatible functions in the datafusion-spark crate in DataFusion: https://github.com/apache/datafusion/tree/main/datafusion/spark

This ticket is part of a series of ticket to fill out the set of spark compatible functions in DataFusion

Describe the solution you'd like

Implement the function listed above in the datafusion-spark crate

Describe alternatives you've considered

  1. Add the relevant function here: https://github.com/apache/datafusion/tree/main/datafusion/spark/src/function/math
  2. Add tests. Instructions are here: https://github.com/apache/datafusion/blob/main/datafusion/spark/README.md

Examples to follow:

You can also find the corresponding datafusion implementations in the corresponding file here: https://github.com/apache/datafusion/tree/main/datafusion/functions/src (it would be ok to call into these implementations if they are the same as the spark implementation)

Additional context

No response

@alamb
Copy link
Contributor Author

alamb commented May 2, 2025

@shehabgamin and @andygrove -- here is a ticket for another spark function. I am hoping we can do one or two of these functions to set a pattern, and then we will be able to basically copy the pattern for the rest of the library

Questions for you:

  1. Is there any other function(s) that you would recommend as one of the first to try and consolidate?
  2. Are you willing to try and make a PR to add this function?

@irenjj
Copy link
Contributor

irenjj commented May 5, 2025

take

@shehabgamin
Copy link
Contributor

@shehabgamin and @andygrove -- here is a ticket for another spark function. I am hoping we can do one or two of these functions to set a pattern, and then we will be able to basically copy the pattern for the rest of the library

Questions for you:

1. Is there any other function(s) that you would recommend as one of the first to try and consolidate?

2. Are you willing to try and make a PR to add this function?
  1. Ceil and Floor are essentially the same function (in terms of code). Sail already has both these functions implemented, see here: https://github.com/lakehq/sail/blob/main/crates/sail-plan/src/extension/function/math/spark_ceil_floor.rs
  2. Not sure when I would have time, if someone else is eager to take it on that would be great! Otherwise, I can add it to my list. Should be an easy copy paste from the Sail code base hopefully.

@alamb
Copy link
Contributor Author

alamb commented May 5, 2025

Sounds good -- thanks @shehabgamin

Looks to me like @irenjj has offered to take a look at this one, so let's see what they come up with

Note that @andygrove created a PR recently for to_hex that might also serve as an example: #15947

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants