feature prepared for py311 #1923

ReimarBauer · 2023-08-23T15:40:02Z

The PR enables python 3.11

Does this PR introduce a breaking change?
This PR enables newer packages of matplotlib, qt and their dependencies

If the changes in this PR are manually verified, list down the scenarios covered::
a build by boa was verified

boa convert meta.yaml
boa build .

Additional information for reviewer? :
Mention if this PR is part of any design or a continuation of previous PRs

Does this PR results in some Documentation changes?
If yes, include the list of Documentation changes

Checklist:

Bug fix. Fixes Build for python 3.11 is blocked #1750
New feature (Non-API breaking changes that adds functionality)
PR Title follows the convention of <type>: <subject>
Commit has unit tests

ReimarBauer · 2023-08-23T15:54:26Z

Tests not suceeded, a segfault stops it in a way that it looks good

tests/_test_mscolab/test_file_manager.py::Test_FileManager::test_fetch_operation_creator
Fatal Python error: Segmentation fault

ReimarBauer · 2023-08-23T16:19:01Z

we need to lookup for grid_b, test_mss_plot_driver.py triggers this

2023-08-23T16:10:27.2407781Z Error message: keyword grid_b is not recognized; valid keywords are ['size', 'width', 'color', 'tickdir', 'pad', 'labelsize', 'labelcolor', 'zorder', 'gridOn', 'tick1On', 'tick2On', 'label1On', 'label2On', 'length', 'direction', 'left', 'bottom', 'right', 'top', 'labelleft', 'labelbottom', 'labelright', 'labeltop', 'labelrotation', 'grid_agg_filter', 'grid_alpha', 'grid_animated', 'grid_antialiased', 'grid_clip_box', 'grid_clip_on', 'grid_clip_path', 'grid_color', 'grid_dash_capstyle', 'grid_dash_joinstyle', 'grid_dashes', 'grid_data', 'grid_drawstyle', 'grid_figure', 'grid_fillstyle', 'grid_gapcolor', 'grid_gid', 'grid_in_layout', 'grid_label', 'grid_linestyle', 'grid_linewidth', 'grid_marker', 'grid_markeredgecolor', 'grid_markeredgewidth', 'grid_markerfacecolor', 'grid_markerfacecoloralt', 'grid_markersize', 'grid_markevery', 'grid_mouseover', 'grid_path_effects', 'grid_picker', 'grid_pickradius', 'grid_rasterized', 'grid_sketch_params', 'grid_snap', 'grid_solid_capstyle', 'grid_solid_joinstyle', 'grid_transform', 'grid_url', 'grid_visible', 'grid_xdata', 'grid_ydata', 'grid_zorder', 'grid_aa', 'grid_c', 'grid_ds', 'grid_ls', 'grid_lw', 'grid_mec', 'grid_mew', 'grid_mfc', 'grid_mfcalt', 'grid_ms'].
2023-08-23T16:10:27.2408088Z Hint: Check used waypoints..
2023-08-23T16:10:27.2408636Z 2023-08-23 16:08:24 ERROR ERROR: <class 'owslib.util.ServiceException'> The data corresponding to your request is not available. Please check the times and/or path you have specified.
2023-08-23T16:10:27.2408644Z 
2023-08-23T16:10:27.2411462Z Error message: keyword grid_b is not recognized; valid keywords are ['size', 'width', 'color', 'tickdir', 'pad', 'labelsize', 'labelcolor', 'zorder', 'gridOn', 'tick1On', 'tick2On', 'label1On', 'label2On', 'length', 'direction', 'left', 'bottom', 'right', 'top', 'labelleft', 'labelbottom', 'labelright', 'labeltop', 'labelrotation', 'grid_agg_filter', 'grid_alpha', 'grid_animated', 'grid_antialiased', 'grid_clip_box', 'grid_clip_on', 'grid_clip_path', 'grid_color', 'grid_dash_capstyle', 'grid_dash_joinstyle', 'grid_dashes', 'grid_data', 'grid_drawstyle', 'grid_figure', 'grid_fillstyle', 'grid_gapcolor', 'grid_gid', 'grid_in_layout', 'grid_label', 'grid_linestyle', 'grid_linewidth', 'grid_marker', 'grid_markeredgecolor', 'grid_markeredgewidth', 'grid_markerfacecolor', 'grid_markerfacecoloralt', 'grid_markersize', 'grid_markevery', 'grid_mouseover', 'grid_path_effects', 'grid_picker', 'grid_pickradius', 'grid_rasterized', 'grid_sketch_params', 'grid_snap', 'grid_solid_capstyle', 'grid_solid_joinstyle', 'grid_transform', 'grid_url', 'grid_visible', 'grid_xdata', 'grid_ydata', 'grid_zorder', 'grid_aa', 'grid_c', 'grid_ds', 'grid_ls', 'grid_lw', 'grid_mec', 'grid_mew', 'grid_mfc', 'grid_mfcalt', 'grid_ms'].
2023-08-23T16:10:27.2411591Z Hint: Check used waypoints.
2023-08-23T16:10:27.2411854Z ______________________ Test_VSec.test_repeated_locations _______________________
2023-08-23T16:10:27.2412205Z [gw6] linux -- Python 3.11.4 /opt/conda/envs/mss-develop-env/bin/python3.11
2023-08-23T16:10:27.2412214Z 
2023-08-23T16:10:27.2412385Z self = <test_mss_plot_driver.Test_VSec object at 0x7fd751515050>
2023-08-23T16:10:27.2412392Z 
2023-08-23T16:10:27.2412517Z     def test_repeated_locations(self):
2023-08-23T16:10:27.2412613Z         p1 = [45.00, 8.]
2023-08-23T16:10:27.2412707Z         p2 = [50.00, 12.]
2023-08-23T16:10:27.2412813Z         self.path = [p1, p1]
2023-08-23T16:10:27.2413013Z >       img = self.plot(mpl_vsec_styles.VS_TemperatureStyle_01(driver=self.vsec))
2023-08-23T16:10:27.2413021Z 
2023-08-23T16:10:27.2413148Z tests/_test_mswms/test_mss_plot_driver.py:95: 
2023-08-23T16:10:27.2413307Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2023-08-23T16:10:27.2413452Z tests/_test_mswms/test_mss_plot_driver.py:89: in plot
2023-08-23T16:10:27.2413575Z     return self.vsec.plot()
2023-08-23T16:10:27.2413898Z mslib/mswms/mss_plot_driver.py:541: in plot
2023-08-23T16:10:27.2414090Z     image = self.plot_object.plot_vsection(data, self.lats, self.lons,
2023-08-23T16:10:27.2414225Z mslib/mswms/mpl_vsec.py:224: in plot_vsection
2023-08-23T16:10:27.2414319Z     self._plot_style()
2023-08-23T16:10:27.2414462Z mslib/mswms/mpl_vsec_styles.py:355: in _plot_style
2023-08-23T16:10:27.2414615Z     self._latlon_logp_setup(orography=curtain_p[0, :])
2023-08-23T16:10:27.2414743Z mslib/mswms/mpl_vsec.py:135: in _latlon_logp_setup
2023-08-23T16:10:27.2414833Z     ax.grid(b=True)
2023-08-23T16:10:27.2415248Z /opt/conda/envs/mss-develop-env/lib/python3.11/site-packages/matplotlib/axes/_base.py:3194: in grid
2023-08-23T16:10:27.2415398Z     self.xaxis.grid(visible, which=which, **kwargs)
2023-08-23T16:10:27.2415715Z /opt/conda/envs/mss-develop-env/lib/python3.11/site-packages/matplotlib/axis.py:1660: in grid
2023-08-23T16:10:27.2416048Z     self.set_tick_params(which='major', **gridkw)
2023-08-23T16:10:27.2416388Z /opt/conda/envs/mss-develop-env/lib/python3.11/site-packages/matplotlib/axis.py:932: in set_tick_params
2023-08-23T16:10:27.2416528Z     kwtrans = self._translate_tick_params(kwargs)
2023-08-23T16:10:27.2416689Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ReimarBauer · 2023-08-23T16:31:42Z

We missed the deprecation in matplotlib 3.5

matplotlib/matplotlib#25267 (comment)

ReimarBauer · 2023-08-24T15:23:22Z

interestingly this version now shows the test which blocks all tests

ReimarBauer · 2023-11-13T13:34:19Z

.github/workflows/testing.yml

@@ -96,7 +96,7 @@ jobs:
        && source /opt/conda/etc/profile.d/conda.sh \
        && source /opt/conda/etc/profile.d/mamba.sh \
        && mamba activate mss-${{ inputs.branch_name }}-env \
-        && pytest -vv -n 6 --dist loadfile --max-worker-restart 0 tests \
+        && pytest -vv -n 6 --dist loadfile --max-worker-restart 4 tests \
        || (for i in {1..5} \


solves the situation when a worker errors. without all tests of that worker in the queue would be skipped

ReimarBauer · 2023-11-13T13:35:25Z

localbuild/meta.yaml

    - pygeoif <1.0.0
    - isodate
    - lxml
    - netcdf4
    - hdf4
    - pillow
    - pytz
-    - pyqt >=5, <5.13
-    - qt >=5.10, <5.13
+    - pyqt >=5.15.0


this is needed to get recent python versions used. This also enables updating a lot dependent packages.

ReimarBauer · 2023-11-13T13:39:26Z

tests/_test_utils/test_auth.py

    # empty http_auth definition
    server_url = "http://example.com"
    http_auth = config_loader(dataset="MSS_auth")
    assert http_auth == {}
    data = auth.get_auth_from_url_and_name(server_url, http_auth, overwrite_login_cache=False)
    assert data == (None, None)
+    # checking if the test setup changes this
+    assert constants.AUTH_LOGIN_CACHE == {}


this test on github sometimes seems like to have a bias from another test, related to TestMigration. It happens only here in serial tests. And it does not happen on each run. In that case at some point constants.AUTH_LOGIN_CACHE is not empty.

matrss · 2023-11-14T09:16:41Z

I cannot get this to build with python 3.11, boa itself seems to break:

boa convert meta.yaml > recipe.yaml
boa build .

leads to:

AttributeError: 
"load()" has been removed, use

  yaml = YAML(typ='rt')
  yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data,
instead of file "/home/icg149/mambaforge/envs/mssdev-py311/lib/python3.11/site-packages/boa/cli/convert.py", line 133

        ruamel.yaml.load("".join(rest_lines), ruamel.yaml.RoundTripLoader)

matrss · 2023-11-14T10:02:37Z

conftest.py

@@ -243,7 +243,6 @@ def fail_if_open_message_boxes_left():
        except RuntimeError:
            pass

-


This change seems wrong to me. Top-level functions should be separated by 2 newlines according to pep8.

ouch, but there are more

#2093

matrss

Using

mamba install python=3.11
mamba mambabuild .
mamba install --use-local mss

worked and after installing the dev dependencies the test suite passes as well. Starting msui looked good too.

Are there any blockers to get this up to py312 as well? I couldn't get my steps to work when I change 3.11 to 3.12, for some reason it tries to install an ancient mss version in the last step (mss-1.2.1-py27_0).

ReimarBauer · 2023-11-14T14:07:39Z

I cannot get this to build with python 3.11, boa itself seems to break:

boa convert meta.yaml > recipe.yaml
boa build .

leads to:

AttributeError: 
"load()" has been removed, use

  yaml = YAML(typ='rt')
  yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data,
instead of file "/home/icg149/mambaforge/envs/mssdev-py311/lib/python3.11/site-packages/boa/cli/convert.py", line 133

        ruamel.yaml.load("".join(rest_lines), ruamel.yaml.RoundTripLoader)

interesting, some other than
the problem is an inconsistancy in xmlschema, see

#2056

in the recipe - xmlschema <2.5.0 helps, but we should have a fix upstream

ReimarBauer · 2023-11-14T14:10:22Z

Using
mamba install python=3.11
mamba mambabuild .
mamba install --use-local mss
worked and after installing the dev dependencies the test suite passes as well. Starting msui looked good too.

Are there any blockers to get this up to py312 as well? I couldn't get my steps to work when I change 3.11 to 3.12, for some reason it tries to install an ancient mss version in the last step (mss-1.2.1-py27_0).

once we can build basemap for py312 we can try
matplotlib/basemap#590

ReimarBauer marked this pull request as draft August 23, 2023 15:53

ReimarBauer changed the title ~~feat prepared for py311~~ feature prepared for py311 Aug 23, 2023

ReimarBauer force-pushed the py311 branch from 59c425b to ea7f33c Compare August 25, 2023 11:32

ReimarBauer added 17 commits August 27, 2023 21:28

prepared for py311

20ea5cb

skipped needs review

330d0e5

enables worker restart

684adef

worker restart

13f6b82

skip test

caacecb

because of deprecation of grid_b and change to visible we need 3.7

1b2e0d4

disabled fixture

1ff0618

fixed deprecation

401acce

disabled test without asserts

231169a

skip test

98bb9e8

module added

acc28eb

test skipped

a15f8df

test skipped

3f5753f

test skipped

05ff577

skip tests

4337d1f

trying other ports

1171779

flake8

45d076d

ReimarBauer force-pushed the py311 branch from 72cc941 to 45d076d Compare August 27, 2023 19:29

ReimarBauer added 5 commits August 27, 2023 23:00

skip test

3c0ff3f

Merge branch 'develop' into py311

865259c

Merge branch 'develop' into py311

e562a18

undo test changes

2e0396c

skip test

f5d1edf

ReimarBauer added 5 commits November 13, 2023 11:15

typo

e34bf02

flake8

615e824

set start condition

c471318

skip early

5cc560b

check added

428c813

ReimarBauer commented Nov 13, 2023

View reviewed changes

ReimarBauer marked this pull request as ready for review November 13, 2023 13:40

ReimarBauer requested review from joernu76 and matrss November 13, 2023 13:40

joernu76 approved these changes Nov 14, 2023

View reviewed changes

matrss reviewed Nov 14, 2023

View reviewed changes

matrss approved these changes Nov 14, 2023

View reviewed changes

ReimarBauer merged commit be3e513 into develop Nov 14, 2023
4 checks passed

ReimarBauer deleted the py311 branch November 14, 2023 14:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature prepared for py311 #1923

feature prepared for py311 #1923

ReimarBauer commented Aug 23, 2023 •

edited

Loading

ReimarBauer commented Aug 23, 2023

ReimarBauer commented Aug 23, 2023 •

edited

Loading

ReimarBauer commented Aug 23, 2023 •

edited

Loading

ReimarBauer commented Aug 24, 2023

ReimarBauer Nov 13, 2023

ReimarBauer Nov 13, 2023

ReimarBauer Nov 13, 2023

matrss commented Nov 14, 2023

matrss Nov 14, 2023

ReimarBauer Nov 14, 2023

matrss left a comment •

edited

Loading

ReimarBauer commented Nov 14, 2023

ReimarBauer commented Nov 14, 2023

		@@ -243,7 +243,6 @@ def fail_if_open_message_boxes_left():
		except RuntimeError:
		pass

feature prepared for py311 #1923

feature prepared for py311 #1923

Conversation

ReimarBauer commented Aug 23, 2023 • edited Loading

ReimarBauer commented Aug 23, 2023

ReimarBauer commented Aug 23, 2023 • edited Loading

ReimarBauer commented Aug 23, 2023 • edited Loading

ReimarBauer commented Aug 24, 2023

ReimarBauer Nov 13, 2023

Choose a reason for hiding this comment

ReimarBauer Nov 13, 2023

Choose a reason for hiding this comment

ReimarBauer Nov 13, 2023

Choose a reason for hiding this comment

matrss commented Nov 14, 2023

matrss Nov 14, 2023

Choose a reason for hiding this comment

ReimarBauer Nov 14, 2023

Choose a reason for hiding this comment

matrss left a comment • edited Loading

Choose a reason for hiding this comment

ReimarBauer commented Nov 14, 2023

ReimarBauer commented Nov 14, 2023

ReimarBauer commented Aug 23, 2023 •

edited

Loading

ReimarBauer commented Aug 23, 2023 •

edited

Loading

ReimarBauer commented Aug 23, 2023 •

edited

Loading

matrss left a comment •

edited

Loading