Error combining coverage when running with pytest-xdist and a custom coverage plugin

[slanzmich]

Summary

I am working on a custom coverage plugin that returns custom file tracers for Python files. When I use this coverage plugin together with pytest-xdist, it fails when combining the coverage data from the xdist workers (see traceback below).

To debug this, I added keep=True to the calls to Coverage.combine in pytest_cov/engine.py.

In an example with three xdist workers:

• Four .coverage.* files are created: one from each worker proceses, and one from the master process.
• All four coverage files contain the same set of files in the "files" table.
• The coverage file generated from the master process has an empty "arc" table, and no entry in the "tracer" table.

When combining the coverage files, coverage checks if the "tracer" entries of all coverage files match. If a file is listed in the "files" table, but does not have an entry in the "tracer" table, it uses "" as the default tracer. This conflicts with the "tracer" info from the other data files, which causes coverage to error out.

To resolve this issue, pytest-cov could not create that extra, empty coverage data file from the master process.

Traceback

Traceback (most recent call last):
  File "<venv_path>/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "<venv_path>/lib/python3.10/site-packages/_pytest/main.py", line 325, in _main
    config.hook.pytest_runtestloop(session=session)
  File "<venv_path>/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "<venv_path>/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "<venv_path>/lib/python3.10/site-packages/pluggy/_callers.py", line 155, in _multicall
    teardown[0].send(outcome)
  File "<venv_path>/lib/python3.10/site-packages/pytest_cov/plugin.py", line 339, in pytest_runtestloop
    self.cov_controller.finish()
  File "<venv_path>/lib/python3.10/site-packages/pytest_cov/engine.py", line 46, in ensure_topdir_wrapper
    return meth(self, *args, **kwargs)
  File "<venv_path>/lib/python3.10/site-packages/pytest_cov/engine.py", line 355, in finish
    self.cov.combine()
  File "<venv_path>/lib/python3.10/site-packages/coverage/control.py", line 836, in combine
    combine_parallel_data(
  File "<venv_path>/lib/python3.10/site-packages/coverage/data.py", line 179, in combine_parallel_data
    data.update(new_data, aliases=aliases)
  File "<venv_path>/lib/python3.10/site-packages/coverage/sqldata.py", line 760, in update
    raise DataError(
coverage.exceptions.DataError: Conflicting file tracer name for '<project_path>/tests/test_example.py': '' vs 'example_plugin.ExampleCoveragePlugin'

Versions

Python 3.10.11 on Windows

coverage                         7.4.4
pytest                           7.4.3
pytest-cov                       5.0.0
pytest-xdist                     3.5.0

The same also fails on Linux with the same package versions.

Config

pytest is run via python -m pytest --cov --cov-config=pyproject.toml -v --numprocesses=3 --maxschedchunk=1.

coverage is configured as follows:

[tool.coverage.run]
branch = true
plugins = ["example_plugin"]
concurrency = ["thread", "multiprocessing"]

[nedbat]

I don't have a solution to the problem, but I'm curious what your custom plugin does for Python files?

[slanzmich]

It tries to address what is also reported as nedbat/coveragepy#856: Find calls to functions that were compiled by TensorFlow's autograph functionality, and report the original function as covered.

[nedbat]

It tries to address what is also reported as nedbat/coveragepy#856: Find calls to functions that were compiled by TensorFlow's autograph functionality, and report the original function as covered.

Interesting! Keep me posted on the progress.

[ionelmc]

I have created an test for this but can't reproduce the problem: b9c99e7

Perhaps it's not enough to have an empty plugin and needs a custom tracer or something? I need a plugin that can reproduce the problem.