Skip to content

Add PartitionEvaluatorArgs to WindowUDFImpl::partition_evaluator #12803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jcsherin opened this issue Oct 8, 2024 · 0 comments · Fixed by #12804
Closed

Add PartitionEvaluatorArgs to WindowUDFImpl::partition_evaluator #12803

jcsherin opened this issue Oct 8, 2024 · 0 comments · Fixed by #12804
Labels
enhancement New feature or request

Comments

@jcsherin
Copy link
Contributor

jcsherin commented Oct 8, 2024

Is your feature request related to a problem or challenge?

In BuiltInWindowFunction::{Lead, Lag} to create an instance of PartitionEvaluator directly/indirectly depends on the following being available:

  1. arguments to the window function expression,
  2. the data type of the arguments,
  3. whether IGNORE NULLS is specified,
  4. whether the execution order is reversed.

Currently none of this is available when implementing user-defined window functions.

Describe the solution you'd like

  1. Add PartitionEvaluatorArgs to WindowUDFImpl::partition_evaulator.
pub trait WindowUDFImpl: Debug + Send + Sync {

    /// Invoke the function, returning the [`PartitionEvaluator`] instance
    fn partition_evaluator(
        &self,
        partition_evaluator_args: PartitionEvaluatorArgs,
    ) -> Result<Box<dyn PartitionEvaluator>>;

}
  1. Define PartitionEvaluatorArgs:
/// Defines the state of the user-defined window function during
/// physical execution.
#[derive(Debug, Default)]
pub struct PartitionEvaluatorArgs<'a> {
    /// The expressions passed as arguments to the user-defined window
    /// function.
    input_exprs: &'a [Arc<dyn PhysicalExpr>],
    /// The corresponding data types of expressions passed as arguments
    /// to the user-defined window function.
    input_types: &'a [DataType],
    /// Set to `true` if the user-defined window function is reversed.
    is_reversed: bool,
    /// Set to `true` if `IGNORE NULLS` is specified.
    ignore_nulls: bool,
}

Describe alternatives you've considered

No response

Additional context

Part of #12802.

@jcsherin jcsherin added the enhancement New feature or request label Oct 8, 2024
Michael-J-Ward added a commit to Michael-J-Ward/datafusion-python that referenced this issue Oct 15, 2024
Michael-J-Ward added a commit to Michael-J-Ward/datafusion-python that referenced this issue Oct 28, 2024
Michael-J-Ward added a commit to Michael-J-Ward/datafusion-python that referenced this issue Oct 28, 2024
Michael-J-Ward added a commit to apache/datafusion-python that referenced this issue Nov 10, 2024
* patch datafusion deps

* migrate from deprecated RuntimeEnv::new to RuntimeEnv::try_new

Ref: apache/datafusion#12566

* remove Arc from create_udf call

Ref: apache/datafusion#12489

* doc typo

* migrage new UnnestOptions API

Ref: https://github.com/apache/datafusion/pull/12836/files

* update API for logical expr Limit

Ref: apache/datafusion#12836

* remove logical expr CrossJoin

It was removed upstream.

Ref: apache/datafusion#13076

* update PyWindowUDF

Ref: apache/datafusion#12803

* migrate window functions lead and lag to udwf

Ref: apache/datafusion#12802

* migrate window functions rank, dense_rank, and percent_rank to udwf

Ref: apache/datafusion#12648

* convert window function cume_dist to udwf

Ref: apache/datafusion#12695

* convert window function ntile to udwf

Ref: apache/datafusion#12694

* clean up functions_window invocation

* Only one column was being passed to udwf

* Update to DF 43.0.0

* Update tests to look for string_view type

* String view is now the default type for strings

* Making a variety of adjustments in wrappers and unit tests to account for the switch from string to string_view as default

* Resolve errors in doc building

---------

Co-authored-by: Tim Saucer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant