python310Packages.apache-beam build failure on x86_64-linux as of cf73a86c

[samuela]

Issue description

Build of python310Packages.apache-beam failed on x86_64-linux as of cf73a86. This is currently breaking python3Packages.tensorflow-datasets.

error: builder for '/nix/store/d0x1bw5crk8wrx5i9pc320cg5m36g35f-python3.10-apache-beam-2.45.0.drv' failed with exit code 1;
       last 10 log lines:
       > ERROR apache_beam/runners/interactive/sql/sql_chain_test.py - AttributeError: type object 'Series' has no attribute 'append'. Did you mea...
       > ERROR apache_beam/runners/interactive/sql/sql_chain_test.py - AttributeError: type object 'Series' has no attribute 'append'. Did you mea...
       > ERROR apache_beam/runners/interactive/sql/utils_test.py - AttributeError: type object 'Series' has no attribute 'append'. Did you mea...
       > ERROR apache_beam/runners/interactive/sql/utils_test.py - AttributeError: type object 'Series' has no attribute 'append'. Did you mea...
       > ERROR apache_beam/typehints/pandas_type_compatibility_test.py - AttributeError: module 'pandas' has no attribute 'Int64Index'
       > ERROR apache_beam/typehints/pandas_type_compatibility_test.py - AttributeError: module 'pandas' has no attribute 'Int64Index'
       > ERROR apache_beam/utils/interactive_utils_test.py - AttributeError: type object 'Series' has no attribute 'append'. Did you mea...
       > ERROR apache_beam/utils/interactive_utils_test.py - AttributeError: type object 'Series' has no attribute 'append'. Did you mea...
       > = 3 failed, 4322 passed, 877 skipped, 932 warnings, 70 errors in 1953.82s (0:32:33) =
       > /nix/store/rk3g4mxrs84bv1vanqnzpdkhm73ksr0z-stdenv-linux/setup: line 1596: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/d0x1bw5crk8wrx5i9pc320cg5m36g35f-python3.10-apache-beam-2.45.0.drv'.
error: 1 dependencies of derivation '/nix/store/00z7p8jpd7348s1qs9biy0mrl09qd7dk-python3.10-tensorflow-datasets-4.9.2.drv' failed to build

full build log

python3Packages.tensorflow-datasets maintainers cc: @ndl
python310Packages.apache-beam maintainers cc: @ndl
Other cc: n/a

Steps to reproduce

1. Checkout nixpkgs at commit cf73a86
2. Run nix-build -A python310Packages.apache-beam

Technical details

 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.0-1042-azure, Ubuntu, 22.04.3 LTS (Jammy Jellyfish), nobuild`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.16.1`
 - nixpkgs: `/nix/store/jywvfjw7y8hd9bqwibcr510hnsb4hjcy-source`

Contents of ~/.config/nixpkgs/config.nix:

{ allowUnfree = true; cudaSupport = true; cudaCapabilities = ["8.0"]; }

Misc.

This issue was automatically generated by nixpkgs-upkeep.

• CI workflow that created this issue.
• Internal tags: 110297808603086876 -3196476956928570580

[tjni]

This is not fixable until apache/beam#27221 is resolved upstream. We updated pandas and beam has not. I tried to fix it quickly but soon realized the changes need more time to implement properly. The main issue is that beam essentially leaks panda's API by decorating it, and many things have been removed in the new version of pandas.

[natsukium]

Great progress on pandas2 support. apache/beam#28636
We will be able to fix this in the near future.

[akssri]

Both the tagged issues have been fixed upstream. apache-beam remains broken in nix though.
I tried removing the patch for dill, but the tests fail for some reason.

[alexvorobiev]

Any updates on this issue?