pref(rust!, python): Unify sort with SortOptions and SortMultipleOptions

[CanglongCl]

Unify all sort, arg_sort and related functions in rust. Sort by single series functions use SortOptions as options whereas sort by multi series functions use SortMultipleOptions. Add related options for python side.

Closes #9820, closes #10365.

SortOptions and SortMultipleOptions

SortOptions

pub struct SortOptions {
    /// If true sort in descending order.
    /// Default `false`.
    pub descending: bool,
    /// Whether place null values last.
    /// Default `false`.
    pub nulls_last: bool,
    /// If true sort in multiple threads.
    /// Default `true`.
    pub multithreaded: bool,
    /// If true maintain the order of equal elements.
    /// Default `false`.
    pub maintain_order: bool,
}

Example:

let s = Series::new("a", [Some(5), Some(2), Some(3), Some(4), None].as_ref());
let sorted = s
    .sort(
        SortOptions::default()
            .with_order_descending(true)
            .with_nulls_last(true)
            .with_multithreaded(false),
    )
    .unwrap();
assert_eq!(
    sorted,
    Series::new("a", [Some(5), Some(4), Some(3), Some(2), None].as_ref())
);

SortMultipleOptions

pub struct SortMultipleOptions {
    /// Order of the columns. Default all `false``.
    pub descending: Vec<bool>,
    /// Whether place null values last. Default `false`.
    pub nulls_last: bool,
    /// Whether sort in multiple threads. Default `true`.
    pub multithreaded: bool,
    /// Whether maintain the order of equal elements. Default `false`.
    pub maintain_order: bool,
}

Example:

let df = df! {
    "a" => [Some(1), Some(2), None, Some(4), None],
    "b" => [Some(5), None, Some(3), Some(2), Some(1)]
}?;

let out = df
    .sort(
        ["a", "b"],
        SortMultipleOptions::default()
            .with_maintain_order(true)
            .with_multithreaded(false)
            .with_order_descendings([false, true])
            .with_nulls_last(true),
    )?;

let expected = df! {
    "a" => [Some(1), Some(2), Some(4), None, None],
    "b" => [Some(5), None, Some(2), Some(3), Some(1)]
}?;

assert_eq!(out, expected);

Rust API changes

Deprecation: DataFrame::sort_with_options.

New implementation:

For DataFrame

pub fn sort_in_place(
    &mut self,
    by: impl IntoVec<SmartString>,
    sort_options: SortMultipleOptions,
)  -> PolarsResult<&mut Self>

pub fn sort(
    &self,
    by: impl IntoVec<SmartString>,
    sort_options: SortMultipleOptions,
) -> PolarsResult<Self>

pub fn top_k(
    &self,
    k: usize,
    by_column: impl IntoVec<SmartString>,
    sort_options: SortMultipleOptions,
) -> PolarsResult<DataFrame>

For LazyFrame

pub fn sort(self, by: impl IntoVec<SmartString>, sort_options: SortMultipleOptions) -> Self

pub fn sort_by_exprs<E: AsRef<[Expr]>>(
    self,
    by_exprs: E,
    sort_options: SortMultipleOptions,
) -> Self

// as well as bottom k
pub fn top_k<E: AsRef<[Expr]>>(
    self,
    k: IdxSize,
    by_exprs: E,
    sort_options: SortMultipleOptions,
) -> Self

For Expr

pub fn sort(self, options: SortOptions) -> Self

pub fn sort_by<E: AsRef<[IE]>, IE: Into<Expr> + Clone>(
    self,
    by: E,
    sort_options: SortMultipleOptions,
) -> Expr

pub fn arg_sort(self, sort_options: SortOptions) -> Self

For Series

pub fn sort(&self, sort_options: SortOptions) -> PolarsResult<Self>

For Functions

pub fn arg_sort_by<E: AsRef<[Expr]>>(by: E, sort_options: SortMultipleOptions) -> Expr

Python API Changes

Added

• nulls_last: bool = False
• multithreaded: bool = True
• maintain_order: bool = False

for all sort related apis if not exists.

[codecov]

Codecov Report

Attention: Patch coverage is 80.57971% with 134 lines in your changes are missing coverage. Please review.

Project coverage is 81.10%. Comparing base (003f4f2) to head (83be101).
Report is 4 commits behind head on main.

❗ Current head 83be101 differs from pull request most recent head 31b6bbf. Consider uploading reports for the commit 31b6bbf to get more accurate results

Files	Patch %	Lines
crates/polars-ops/src/chunked_array/top_k.rs	43.24%	21 Missing ⚠️
...rs-core/src/chunked_array/ops/sort/arg_bottom_k.rs	74.54%	14 Missing ⚠️
.../polars-core/src/chunked_array/ops/sort/options.rs	86.25%	11 Missing ⚠️
...n/src/logical_plan/optimizer/simplify_functions.rs	0.00%	10 Missing ⚠️
...ates/polars-core/src/chunked_array/ops/sort/mod.rs	86.00%	7 Missing ⚠️
crates/polars-plan/src/logical_plan/tree_format.rs	0.00%	7 Missing ⚠️
...s-core/src/series/implementations/binary_offset.rs	0.00%	6 Missing ⚠️
...lars-core/src/series/implementations/dates_time.rs	0.00%	6 Missing ⚠️
...polars-core/src/series/implementations/datetime.rs	0.00%	6 Missing ⚠️
...polars-core/src/series/implementations/duration.rs	0.00%	6 Missing ⚠️
... and 12 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15590      +/-   ##
==========================================
- Coverage   81.15%   81.10%   -0.06%     
==========================================
  Files        1367     1369       +2     
  Lines      174928   175204     +276     
  Branches     2530     2530              
==========================================
+ Hits       141971   142099     +128     
- Misses      32482    32629     +147     
- Partials      475      476       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codspeed-hq]

CodSpeed Performance Report

Merging #15590 will not alter performance

_{Comparing CanglongCl:sort (31b6bbf) with main (c8e26ca)}

Summary

✅ 22 untouched benchmarks

[ritchie46]

Thanks, this looks good. Can you do a rebase?

[wsyxbcl]

Regarding with_order_descendings in SortMultipleOptions, what's the preferred use case? For instance, if one aims to sort three columns ["a", "b", "c"] in a specific order, what should the input be?

[CanglongCl]

Regarding with_order_descendings in SortMultipleOptions, what's the preferred use case? For instance, if one aims to sort three columns ["a", "b", "c"] in a specific order, what should the input be?

Just be consistent with the columns input. e.g. for ["a", "b", "c"] with order acceding, descending, ascending, input should be SortMultipleOptions::default().with_order_descendings([false, true, false])

If all a, b, c are using descending order, we can also use SortMultipleOptions::default().with_order_descending(true) to apply for all columns

[wsyxbcl]

@CanglongCl Oops, my mistake. I didn't notice the descending aspect and assumed the order referred to the sequence in which the columns were being sorted. Thanks for the response anyway!