Add support for pandas 2.0

[caneff]

With all the tests now passing I added a tox config for pandas 2.0, and fixed setup.py and the gradle to support it. Explicitly not supporting 2.1 yet because of an issue I can't figure out (will insert issue number here when I make it).

Fixes #27221

[caneff]

R: @tvalentyn

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (7fabc12) 38.34% compared to head (66796bd) 38.28%.
Report is 13 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #28636      +/-   ##
==========================================
- Coverage   38.34%   38.28%   -0.07%     
==========================================
  Files         693      690       -3     
  Lines      102237   102029     -208     
==========================================
- Hits        39199    39058     -141     
+ Misses      61446    61391      -55     
+ Partials     1592     1580      -12     

Flag	Coverage Δ
python	29.87% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tvalentyn]

Seeing an error in coverage suite:

E           ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
E           A suitable version of pyarrow or fastparquet is required for parquet support.
E           Trying to import the above resulted in these errors:
E            - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '4.0.1' currently installed).
E            - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.

I suspect we need to configure some parquet io tests for pyarrow 4 to use only pandas 1.x; alternatively, we'd have to bump the lower bound for supported pyarrow versions.

[tvalentyn]

Run Python 3.8 PostCommit

[tvalentyn]

In postcommits, seeing one relevant error, rest seems flake, need to rerun.

�[31mFAILED�[0m apache_beam/examples/dataframe/flight_delays_it_test.py::�[1mFlightDelaysTest::test_flight_delays�[0m - TypeError: Cannot perform reduction 'mean' with string dtype

[tvalentyn]

Likely some pandas1 code in https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/dataframe/flight_delays_it_test.py

[caneff]

Run Python 3.8 PostCommit

[caneff]

Run Python 3.8 PostCommit

[caneff]

Run Python_Coverage PreCommit

[caneff]

Run Python_Coverage PreCommit

[svetakvsundhar]

Run Python 3.8 PostCommit

[svetakvsundhar]

(running again as the failure should be fixed in #28896)

[damccorm]

@caneff @svetakvsundhar @tvalentyn what are next steps here?

[caneff]

Run Python_Coverage PreCommit

[tvalentyn]

Run Python 3.8 PostCommit

[tvalentyn]

Run Python_PVR_Flink PreCommit

[tvalentyn]

Run Python_Coverage PreCommit

[tvalentyn]

Run Python_Coverage Precommit

[tvalentyn]

Run Python_Runners PreCommit

[tvalentyn]

12:00:35 �[31mERROR: Cannot install pyarrow<4 and >=3 and pyarrow<5 and >=4 because these package versions have conflicting dependencies.
12:00:35   py38-pyarrow-3: FAIL code 1 (2.85 seconds)
12:00:35   evaluation failed :( (3.07 seconds)
12:00:35 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
12:00:35 �[39m
12:00:35 > Task :sdks:python:test-suites:tox:py38:testPy38pyarrow-3 FAILED

[tvalentyn]

it seems like something is not right in definition of coverage test suites; I wonder if we don't follow tox.ini syntax correctly

[tvalentyn]

somehow constraints for different pyarrow versions get conjoined. I wonder if listing pyarrow and pandas on the same line would help. see also: https://stackoverflow.com/questions/57024579/tox-ini-environment-with-multiple-dependencies

[caneff]

Run Python 3.8 PostCommit

[tvalentyn]

looks like prior errors are resolved, checking whether remaining apache_beam.utils.multi_process_shared_test.MultiProcessSharedTest.test_connect error is a flake.

[tvalentyn]

Run Python_Coverage PreCommit

[tvalentyn]

Noting that Py3.8 have passed on https://ci-beam.apache.org/job/beam_PostCommit_Python38_PR/810/

[tvalentyn]

Run Python 3.11 PostCommit

[AnandInguva]

@caneff @damccorm merging this PR. Any objections?

[damccorm]

SGTM

[caneff]

SGTM