BUG: Test collection for Transformer fails #30237
Labels
Milestone
Comments
5 tasks
|
It'd be useful to have more info in your log. You might want to add a However, you're right that this is coming from the tags infrastructure, and if I were to guess, it's because you have a transformer which doesn't inherit from In any case, you're right that the message should be better and tell developers that we've switched our tags infrastructure. I'll push a PR for that. |
adrinjalali
added
Developer API
|
Adding this to the milestone since this will affect many people, and it'd be nice for them to get a better error message. |
|
FWIW, I can't get the tests pass on scikit-learn=1.5.2, since I get a segfault: |
|
But I can reproduce the collection error with the nightly release. It comes from Which has: class SlidingEstimator(BaseEstimator, TransformerMixin):
...But it needs to be: class SlidingEstimator(TransformerMixin, BaseEstimator):
...We just added a test to check the right order by the way, so you'd be getting an error for this soon anyway. But I'll fix the test collection as well. |
|
With #30248, I get this on your repo: $ MNE_SKIP_TESTING_DATASET_TESTS=true pytest -m "not (ultraslowtest or pgtest)" --tb=short --cov=mne --cov-report xml -vv -rfE mne/ -l
==================================================================================================================== test session starts =====================================================================================================================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0 -- /home/adrin/micromamba/envs/mne-delete/bin/python3.12
cachedir: .pytest_cache
PySide6 6.8.0.2 -- Qt runtime 6.8.0 -- Qt compiled 6.8.0
MNE 1.9.0.dev92+g460899541 -- /tmp/mne-python/mne
rootdir: /tmp/mne-python
configfile: pyproject.toml
plugins: timeout-2.3.1, cov-6.0.0, anyio-4.6.2.post1, qt-4.4.0
collecting 4086 items / 1 error Using default location ~/mne_data for testing...
Dataset testing version 0.0 out of date, latest version is 0.156
Dataset out of date but force_update=False and download=False, returning empty data_path
collected 4729 items / 1 error / 70 deselected / 4659 selected
=========================================================================================================================== ERRORS ===========================================================================================================================
__________________________________________________________________________________________________ ERROR collecting mne/decoding/tests/test_search_light.py __________________________________________________________________________________________________
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/pluggy/_hooks.py:513: in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
firstresult = True
kwargs = {'collector': <Module test_search_light.py>,
'name': 'test_sklearn_compliance',
'obj': <function test_sklearn_compliance at 0x7e35a9886de0>}
self = <HookCaller 'pytest_pycollect_makeitem'>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/pluggy/_manager.py:120: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
firstresult = True
hook_name = 'pytest_pycollect_makeitem'
kwargs = {'collector': <Module test_search_light.py>,
'name': 'test_sklearn_compliance',
'obj': <function test_sklearn_compliance at 0x7e35a9886de0>}
methods = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py'>>,
<HookImpl plugin_name='unittest', plugin=<module '_pytest.unittest' from '/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/unittest.py'>>,
<HookImpl plugin_name='anyio', plugin=<module 'anyio.pytest_plugin' from '/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/anyio/pytest_plugin.py'>>]
self = <_pytest.config.PytestPluginManager object at 0x7e36616396a0>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py:245: in pytest_pycollect_makeitem
return list(collector._genfunctions(name, obj))
collector = <Module test_search_light.py>
name = 'test_sklearn_compliance'
obj = <function test_sklearn_compliance at 0x7e35a9886de0>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py:462: in _genfunctions
self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
cls = None
clscol = None
definition = <FunctionDefinition test_sklearn_compliance>
fixtureinfo = FuncFixtureInfo(argnames=('estimator', 'check'),
initialnames=('matplotlib_config',
'qt_config',
'protect_config',
'add_mne',
'check_verbose',
'close_all',
'estimator',
'check'),
names_closure=['matplotlib_config',
'qt_config',
'protect_config',
'doctest_namespace',
'add_mne',
'check_verbose',
'close_all',
'estimator',
'check',
'request'],
name2fixturedefs={'add_mne': (<FixtureDef argname='add_mne' scope='function' baseid='mne'>,),
'check_verbose': (<FixtureDef argname='check_verbose' scope='function' baseid='mne'>,),
'close_all': (<FixtureDef argname='close_all' scope='function' baseid='mne'>,),
'doctest_namespace': (<FixtureDef argname='doctest_namespace' scope='session' baseid=''>,),
'matplotlib_config': (<FixtureDef argname='matplotlib_config' scope='session' baseid='mne'>,),
'protect_config': (<FixtureDef argname='protect_config' scope='session' baseid='mne'>,),
'qt_config': (<FixtureDef argname='qt_config' scope='session' baseid='mne'>,)})
funcobj = <function test_sklearn_compliance at 0x7e35a9886de0>
metafunc = <_pytest.python.Metafunc object at 0x7e35a97111c0>
methods = []
module = <module 'mne.decoding.tests.test_search_light' from '/tmp/mne-python/mne/decoding/tests/test_search_light.py'>
modulecol = <Module test_search_light.py>
name = 'test_sklearn_compliance'
self = <Module test_search_light.py>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/pluggy/_hooks.py:574: in call_extra
return self._hookexec(self.name, hookimpls, kwargs, firstresult)
firstresult = False
hookimpls = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py'>>,
<HookImpl plugin_name='funcmanage', plugin=<_pytest.fixtures.FixtureManager object at 0x7e35c2bb2210>>]
kwargs = {'metafunc': <_pytest.python.Metafunc object at 0x7e35a97111c0>}
methods = []
opts = {'hookwrapper': False,
'optionalhook': False,
'specname': None,
'tryfirst': False,
'trylast': False,
'wrapper': False}
self = <HookCaller 'pytest_generate_tests'>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/pluggy/_manager.py:120: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
firstresult = False
hook_name = 'pytest_generate_tests'
kwargs = {'metafunc': <_pytest.python.Metafunc object at 0x7e35a97111c0>}
methods = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py'>>,
<HookImpl plugin_name='funcmanage', plugin=<_pytest.fixtures.FixtureManager object at 0x7e35c2bb2210>>]
self = <_pytest.config.PytestPluginManager object at 0x7e36616396a0>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py:115: in pytest_generate_tests
metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)
marker = Mark(name='parametrize',
args=('estimator, check',
<generator object parametrize_with_checks.<locals>.checks_generator at 0x7e35a9afa200>),
kwargs={'ids': <function _get_check_estimator_ids at 0x7e35a9982480>})
metafunc = <_pytest.python.Metafunc object at 0x7e35a97111c0>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/python.py:1206: in parametrize
argnames, parametersets = ParameterSet._for_parametrize(
_param_mark = Mark(name='parametrize',
args=('estimator, check',
<generator object parametrize_with_checks.<locals>.checks_generator at 0x7e35a9afa200>),
kwargs={'ids': <function _get_check_estimator_ids at 0x7e35a9982480>})
argnames = 'estimator, check'
argvalues = <generator object parametrize_with_checks.<locals>.checks_generator at 0x7e35a9afa200>
ids = <function _get_check_estimator_ids at 0x7e35a9982480>
indirect = False
scope = None
self = <_pytest.python.Metafunc object at 0x7e35a97111c0>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/mark/structures.py:159: in _for_parametrize
parameters = cls._parse_parametrize_parameters(argvalues, force_tuple)
argnames = ['estimator', 'check']
argvalues = <generator object parametrize_with_checks.<locals>.checks_generator at 0x7e35a9afa200>
cls = <class '_pytest.mark.structures.ParameterSet'>
config = <_pytest.config.Config object at 0x7e366196a930>
force_tuple = False
func = <function test_sklearn_compliance at 0x7e35a9886de0>
nodeid = 'mne/decoding/tests/test_search_light.py::test_sklearn_compliance'
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/_pytest/mark/structures.py:146: in _parse_parametrize_parameters
ParameterSet.extract_from(x, force_tuple=force_tuple) for x in argvalues
argvalues = <generator object parametrize_with_checks.<locals>.checks_generator at 0x7e35a9afa200>
force_tuple = False
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/sklearn/utils/estimator_checks.py:520: in checks_generator
for check in _yield_all_checks(estimator, legacy=legacy):
check = <function check_f_contiguous_array_estimator at 0x7e35a9983600>
check_instance = <SlidingEstimator(allow_2d=True, base_estimator=LogisticRegression())>
check_with_name = functools.partial(<function check_f_contiguous_array_estimator at 0x7e35a9983600>, 'SlidingEstimator')
estimator = <SlidingEstimator(allow_2d=True, base_estimator=LogisticRegression())>
estimators = [<SlidingEstimator(allow_2d=True, base_estimator=LogisticRegression())>]
legacy = True
name = 'SlidingEstimator'
pytest = <module 'pytest' from '/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/pytest/__init__.py'>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/sklearn/utils/estimator_checks.py:371: in _yield_all_checks
for check in _yield_transformer_checks(estimator):
check = <function check_f_contiguous_array_estimator at 0x7e35a9983600>
estimator = <SlidingEstimator(allow_2d=True, base_estimator=LogisticRegression())>
legacy = True
name = 'SlidingEstimator'
tags = Tags(estimator_type=None,
target_tags=TargetTags(required=False,
one_d_labels=False,
two_d_labels=False,
positive_only=False,
multi_output=False,
single_output=True),
transformer_tags=None,
classifier_tags=None,
regressor_tags=None,
array_api_support=False,
no_validation=False,
non_deterministic=False,
requires_fit=True,
_skip_test=False,
_xfail_checks={},
input_tags=InputTags(one_d_array=False,
two_d_array=True,
three_d_array=False,
sparse=False,
categorical=False,
string=False,
dict=False,
positive_only=False,
allow_nan=False,
pairwise=False))
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/sklearn/utils/estimator_checks.py:263: in _yield_transformer_checks
_raise_for_missing_tags(transformer, "transformer_tags", TransformerMixin)
transformer = <SlidingEstimator(allow_2d=True, base_estimator=LogisticRegression())>
/home/adrin/micromamba/envs/mne-delete/lib/python3.12/site-packages/sklearn/utils/estimator_checks.py:103: in _raise_for_missing_tags
raise RuntimeError(
E RuntimeError: Estimator SlidingEstimator seems to be a Transformer, but the `transformer_tags` tag is not set. Either set the tag manually or inherit from the TransformerMixin. Note that the order of inheritance matters, the TransformerMixin should come before BaseEstimator.
Mixin = <class 'sklearn.base.TransformerMixin'>
estimator = <SlidingEstimator(allow_2d=True, base_estimator=LogisticRegression())>
estimator_type = 'Transformer'
tag_name = 'transformer_tags'
tags = Tags(estimator_type=None,
target_tags=TargetTags(required=False,
one_d_labels=False,
two_d_labels=False,
positive_only=False,
multi_output=False,
single_output=True),
transformer_tags=None,
classifier_tags=None,
regressor_tags=None,
array_api_support=False,
no_validation=False,
non_deterministic=False,
requires_fit=True,
_skip_test=False,
_xfail_checks={},
input_tags=InputTags(one_d_array=False,
two_d_array=True,
three_d_array=False,
sparse=False,
categorical=False,
string=False,
dict=False,
positive_only=False,
allow_nan=False,
pairwise=False))
--------------------------------------------------------------------------------------------------- generated xml file: /tmp/mne-python/junit-results.xml ----------------------------------------------------------------------------------------------------
---------- coverage: platform linux, python 3.12.7-final-0 -----------
Coverage XML written to file coverage.xml
================================================================================================================== short test summary info ===================================================================================================================
ERROR mne/decoding/tests/test_search_light.py - RuntimeError: Estimator SlidingEstimator seems to be a Transformer, but the `transformer_tags` tag is not set. Either set the tag manually or inherit from the TransformerMixin. Note that the order of inheritance matters, the TransformerMixin should come before BaseEstimator.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================================== 70 deselected, 1 error in 16.30s ==============================================================================================================which seems rather nice and informative enough. |
So on this branch regardless of whether or not I change the inheritance order in MNE-Python to where This also based on |
|
Your implementation is a bit odd to me. The issue comes from here: class LinearModel(BaseEstimator):
_model_attr_wrap = (
...
"_estimator_type",
...
)
...
def __getattr__(self, attr):
"""Wrap to model for some attributes."""
if attr in LinearModel._model_attr_wrap:
return getattr(self.model, attr)
elif attr == "fit_transform" and hasattr(self.model, "fit_transform"):
return super().__getattr__(self, "_fit_transform")
return super().__getattr__(self, attr)you're delegating Your class definition here needs to be class LinearModel(ClassifierMixin, BaseEstimator):
...And when you hack into python's machinery to get attributes from the sub-estimator like you're doing, it's not something we really support 😅 |
|
Yeah... 🤦 Thanks for the additional help here! In the meantime because |
|
The right way to handle this is more like: class MyEstimator(MetaEstimatorMixin, BaseEstimator):
...
def __sklearn_tags__(self):
tags = super().__sklearn_tags__()
sub_tags = get_tags(self.estimator)
tags.estimator_type = sub_tags.estimator_type
...
return tags |
|
Thanks! Added a new issue in our repo mne-tools/mne-python#12952 to remind us to fix it properly at some point. |
github-project-automation
bot
moved this to Done
in Developer API and third party library support
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
On latest
scientific-python-nightly-wheelswheel things were passing yesterday but now we now get the following when using parametrize_with_checks:Full traceback
According to a local
git bisectthis was introduced by #30122. I see some API entries and maybe we need to adjust some code, but it seems like an API change should at least emit a future or deprecation warning rather than fail hard like this.Steps/Code to Reproduce
Expected Results
Tests pass (or at least we get an informative error about our misuse of something!)
Actual Results
☝️
Versions
System: python: 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] executable: /home/larsoner/python/virtualenvs/base/bin/python machine: Linux-6.8.0-48-generic-x86_64-with-glibc2.39 Python dependencies: sklearn: 1.6.dev0 pip: 24.0 setuptools: 75.3.0 numpy: 2.2.0.dev0 scipy: 1.15.0.dev0 Cython: 3.0.10 pandas: 3.0.0.dev0+1597.g9e10119dc8 matplotlib: 3.10.0.dev491+gf7b3def52b joblib: 1.4.2 threadpoolctl: 3.5.0 Built with OpenMP: True threadpoolctl info: user_api: blas internal_api: openblas num_threads: 1 prefix: libscipy_openblas filepath: /home/larsoner/python/virtualenvs/base/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-6bb31eeb.so version: 0.3.28 threading_layer: pthreads architecture: Haswell user_api: blas internal_api: openblas num_threads: 1 prefix: libscipy_openblas filepath: /home/larsoner/python/virtualenvs/base/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so version: 0.3.28 threading_layer: pthreads architecture: Haswell user_api: openmp internal_api: openmp num_threads: 8 prefix: libgomp filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0 version: NoneThe text was updated successfully, but these errors were encountered: