Geven monkey patching not being set in subprocesses

[ddorian]

Hi,

I have a simple test file of:

from gevent.monkey import is_module_patched, saved


def test_auto():
    if not is_module_patched("threading"):
        print(saved)
        print(saved.keys())
        exit("not patched!")

When I run pytest normally with monkey patching it works:

python -m gevent.monkey --module pytest

But running with xdist it breaks:

❯ python -m gevent.monkey --module pytest -n auto
==================================================================================================================================================================== test session starts ====================================================================================================================================================================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/myhome/Desktop/stream/xdist-gevent
configfile: pyproject.toml
plugins: xdist-3.5.0
12 workers [1 item]       
F                                                                                                                                                                                                                                                                                                                                                     [100%]
========================================================================================================================================================================= FAILURES ==========================================================================================================================================================================
_________________________________________________________________________________________________________________________________________________________________________ test_auto _________________________________________________________________________________________________________________________________________________________________________
[gw0] linux -- Python 3.12.2 /home/myhome/.cache/pypoetry/virtualenvs/xdist-gevent-hYN2y4Ul-py3.12/bin/python

    def test_auto():
        if not is_module_patched("threading"):
            print(saved)
            print(saved.keys())
>           exit("not patched!")

tests/test_some.py:8: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Use exit() or Ctrl-D (i.e. EOF) to exit, code = 'not patched!'

>   ???
E   SystemExit: not patched!

<frozen _sitebuiltins>:26: SystemExit
------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------------------------------------------------------
{}
dict_keys([])
================================================================================================================================================================== short test summary info ==================================================================================================================================================================
FAILED tests/test_some.py::test_auto - SystemExit: not patched!
===================================================================================================================================================================== 1 failed in 0.88s =====================================================================================================================================================================

How can I monkey patch the workers too?

[RonnyPfannschmidt]

Currently there's no built-in way to auto monkepatch on startup of a worker

I think we would need to extend execmodels

[ddorian]

Any hint how I can try to hard fix it? That codebase looks complex, hard to find my way around it.

[RonnyPfannschmidt]

No I'm rather unfamiliar with gevent and similar and currently not in a position to work on extra gestures regarding it

[nicoddemus]

@ddorian perhaps you can implement pytest_configure() and call monkey.patch_all() explicitly:

def pytest_configure():
    if os.environ.get("PYTEST_XDIST_WORKER", "").startswith("gw"):
        from gevent import monkey
        monkey.patch_all()

[ddorian]

That made the process hang. So the underlying module is not compatible (or the patching was done too late).

#conftest.py
def pytest_configure(config):
    import os

    from gevent import monkey

    print("sssssssssssssssssssssssssssssssssssssssss")

    if os.environ.get("PYTEST_XDIST_WORKER", "").startswith("gw"):
        monkey.patch_all()
        exit("CONFIGURING!!!")

python -m gevent.monkey --module pytest -n 2
sssssssssssssssssssssssssssssssssssssssss
==================================================================================================================================================================== test session starts ====================================================================================================================================================================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/guru/Desktop/stream/xdist-gevent
configfile: pyproject.toml
plugins: xdist-3.5.0
initialized: 2/2 workers

[nicoddemus]

or the patching was done too late

Ahh possibly this then. You might try @hookimpl(tryfirst=True) to see if it helps, but other than that I'm out of ideas I'm afraid.

[ddorian]

Ahh possibly this then. You might try @hookimpl(tryfirst=True) to see if it helps,

Still stuck.

but other than that I'm out of ideas I'm afraid.

I just need a way to execute the monkey patching first then when the worker is started, but can't understand the code of execnet to see where it happens.

[RonnyPfannschmidt]

This pretty much needs a new execmodels that invokes the patches on usage

[rafales]

@RonnyPfannschmidt there is a gevent exec model in execnet. The problem is that xdist always tries to override execmodel, so params --tx 6*popen//execmodel=gevent --dist=load cause

INTERNALERROR>     raise ValueError(f"duplicate key: {key!r} in {string!r}")
INTERNALERROR> ValueError: duplicate key: 'execmodel' in 'execmodel=main_thread_only//popen//execmodel=gevent'

[RonnyPfannschmidt]

@rafales that's a different new bug