BUG: FutureWarning when splitting a dataframe using np.split

[amanlai]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

To reproduce it:

import numpy as np
import pandas as pd

df = pd.DataFrame({'a': [1, 2, 3]})
lst = np.split(df, 3)

Issue Description

The above code raises a FutureWarning:

FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.

As far as I understand, np.split uses np.swapaxes which is raising this warning.

Expected Behavior

Not show a warning.

Installed Versions

python : 3.11.5
pandas : 2.2.0
numpy : 1.26.3

[VISWESWARAN1998]

swapaxes is called from shape_base.py in numpy package

[rhshadrach]

Thanks for the report - is there a reason you prefer to use np.swapaxes(df) over df.T?

[amanlai]

@rhshadrach the use case here is not np.swapaxes(df) itself, it's np.split(df), which apparently uses np.swapaxes under the hood.

[rhshadrach]

Ah, thanks @amanlai. On main, DataFrame.swapaxes has been removed and the OP gives the output:

[array([[1]]), array([[2]]), array([[3]])]

On 2.2.x, I am seeing

[   a
0  1,    a
1  2,    a
2  3]

cc @jorisvandenbossche @phofl @mroeschke

[rhshadrach]

Marking this as a blocker for 3.0 so a decision is made.

[Aloqeely]

I don't think many people use np.split on a DataFrame to consider reverting this deprecation. (Stackoverflow question did not have many interactions)

And as stated in numpy/numpy#24889 (comment) it's possible to calculate the start/stop slices and then manually slicing using df.iloc[start:stop]

[WillAyd]

I agree with @Aloqeely. Is there an upstream issue for NumPy on this topic? I think should be resolved there and use df.T like @rhshadrach suggests

[WillAyd]

Ah ignore my previous comment - I thought they were calling our swapaxes implementation but misread the OP. Assuming they call swapaxes generically for their internal use, so not as easy as changing the call

Even still I don't think we should revert this deprecation

[Aloqeely]

Is it sensible to implement a DataFrame.split function for convenience? Since np.split doesn't work appropriately on DataFrames anymore.

I can work on it next week if you are all ok with it.

[jorisvandenbossche]

Is there an upstream issue for NumPy on this topic?

numpy/numpy#24889 (comment)

[WillAyd]

Is it sensible to implement a DataFrame.split function for convenience? Since np.split doesn't work appropriately on DataFrames anymore.

I don't think so - generally our goal is to reduce the footprint of our API, and I don't see this as a huge value add over the other method you have suggested:

And as stated in numpy/numpy#24889 (comment) it's possible to calculate the start/stop slices and then manually slicing using df.iloc[start:stop]

[ddelange]

Is it sensible to implement a DataFrame.split function for convenience? Since np.split doesn't work appropriately on DataFrames anymore.

I can work on it next week if you are all ok with it.

+1 👍 splitting a dataframe into equally sized chunks (except for the trailing chunk) is a routine task in ML and other batching applications.

requiring end-users to replace np.split/np.array_split with some bespoke iloc helper function sounds to me like a lot of duplicated boilerplate, wasted brain cycles over time, and overall increased bug surface for the ecosystem

[ddelange]

another big reason to use np.array_split is the optional axis=1 argument, which is probably why it is calling np.swapaxes in the first place.