Expr.first, Expr.last should always produce same size output (PanicException: The column lengths in the DataFrame are not equal)

[Julian-J-S]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

df = pl.DataFrame(
    {
        "a": [1, 2],
        "b": [None, None],
    }
)

df.select(
    pl.all().drop_nulls().sum().name.suffix("_sum"),
    pl.all().drop_nulls().std().name.suffix("_std"),
    # pl.all().drop_nulls().first().name.suffix("_first"), # PanicException: The column lengths in the DataFrame are not equal.
    pl.all().drop_nulls().first().len().name.suffix("_len"),
)

shape: (1, 6)
┌───────┬───────┬──────────┬───────┬───────┬───────┐
│ a_sum ┆ b_sum ┆ a_std    ┆ b_std ┆ a_len ┆ b_len │
│ ---   ┆ ---   ┆ ---      ┆ ---   ┆ ---   ┆ ---   │
│ i64   ┆ f32   ┆ f64      ┆ f32   ┆ u32   ┆ u32   │
╞═══════╪═══════╪══════════╪═══════╪═══════╪═══════╡
│ 3     ┆ 0.0   ┆ 0.707107 ┆ null  ┆ 1     ┆ 0     │ <<<<<<<<<<<<<< should not be of size 0!
└───────┴───────┴──────────┴───────┴───────┴───────┘

Log output

No response

Issue description

first / last should behave like other aggreation functions and return a consistent size output (1 element) even if the column is empty!

Empty column behaviour examples:

• sum -> 0
• std -> null
• str.concat -> "" (was recently fixed: fix(python,rust): str.concat on empty list #12066)

BUT

• first -> empy zero length df but should be null

Those inconsistent 0-size results can cause multiple problems.
A recent one (#12354) was already fixed (#12357) by adding extra logic to when/then but the real problem was actually first returning a 0-size result. By fixing the return value of first no extrac logic is required.

Expected behavior

first / last should behave like other aggreation functions and return a consistent size output (1 element) even if the column is empty!

• correct: [].first -> null
• wrong: [].first -> 0-size df

Installed versions

0.19.12

[reswqa]

I see your point. My only concern is that this could easily make it seem like the first element of this column is Null instead of empty. After all, first and last have such meanings.

[Julian-J-S]

I see your point. My only concern is that this could easily make it seem like the first element of this column is Null instead of empty. After all, first and last have such meanings.

good point!

However, I think being consistent and always returning a value is better for the user.
This "problem" can lead to bugs/panics/crashes that the user cannot control.

The ambiguity is not allways avoidable but also consistent with the other functions.

• [].std = null
• [null, null].std = null

What the user can do is drop_nulls() before and when the result is null you know it is empty

[reswqa]

I think there are some differences between first/last and std/mean etc. It clearly indicates the element at the specific position, but the others do not.

But I truly understand the benefits of being consistent with other agg funcs. So I don't have a strong tendency. :)

[nameexhaustion]

It's reasonable - DuckDB notably does this:

>>> df = pl.DataFrame({"x": []})
>>> df.select(pl.first("x").alias("pl.first(x)"), pl.last("x").alias("pl.last(x)"))
shape: (0, 2)
┌─────────────┬────────────┐
│ pl.first(x) ┆ pl.last(x) │
│ ---         ┆ ---        │
│ f32         ┆ f32        │
╞═════════════╪════════════╡
└─────────────┴────────────┘
>>> duckdb.sql("select first(x), last(x) from df")
┌──────────┬─────────┐
│ first(x) │ last(x) │
│  float   │  float  │
├──────────┼─────────┤
│     NULL │    NULL │
└──────────┴─────────┘