Test coverage wrong when importing large library

[MattTheCuber]

Type: Bug

1. Create a simple project that imports MONAI with a test
2. Open the Testing tab and run tests with coverage
3. Module-level lines (e.g. imports, class definitions, and function definitions) will show as missed in the coverage report

Extension version: 2024.16.0
VS Code version: Code 1.94.0 (d78a74bcdfad14d5d3b1b782f87255d802b57511, 2024-10-02T13:08:12.626Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64
Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64
Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64
Remote OS version: Linux x64 4.18.0-348.23.1.el8_5.x86_64

System Info

Item	Value
CPUs	AMD Ryzen 7 5825U with Radeon Graphics (16 x 1996)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	38.83GB (21.36GB free)
Process Argv	--file-uri file:///c%3A/Users/mvine/Documents/Programming/goliath-fragct.code-workspace
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: 10.10.10.137
OS	Linux x64 4.18.0-348.23.1.el8_5.x86_64
CPUs	AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)
Memory (System)	125.82GB (89.95GB free)
VM	0%

Item	Value
Remote	SSH: 10.10.10.137
OS	Linux x64 4.18.0-348.23.1.el8_5.x86_64
CPUs	AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)
Memory (System)	125.82GB (89.95GB free)
VM	0%

Item	Value
Remote	SSH: 10.10.10.137
OS	Linux x64 4.18.0-348.23.1.el8_5.x86_64
CPUs	AMD Ryzen 9 5950X 16-Core Processor (32 x 3650)
Memory (System)	125.82GB (89.95GB free)
VM	0%

Item	Value
Remote	SSH: 10.10.10.137
OS	Linux x64 4.18.0-348.23.1.el8_5.x86_64
CPUs	AMD Ryzen 9 5950X 16-Core Processor (32 x 3589)
Memory (System)	125.82GB (89.95GB free)
VM	0%

[eleanorjboyd]

thank you for the bug report! Could you include what is shown if you run coverage from the terminal as comparison?

[MattTheCuber]

thank you for the bug report! Could you include what is shown if you run coverage from the terminal as comparison?

Sure thing:

---------- coverage: platform linux, python 3.11.6-final-0 -----------
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
app/__init__.py              0      0   100%
app/main.py                  1      0   100%
app/tests/__init__.py        0      0   100%
app/tests/test_main.py       3      0   100%
------------------------------------------------------
TOTAL                        4      0   100%

[MattTheCuber]

@eleanorjboyd any update on this? Anything more I can provide to help debug?

[eleanorjboyd]

What is your root dir when you run it from the terminal? You can see what one is selected for running tests in vscode and so im trying to figure out what setting could be different between what you are putting in the terminal and the extension might be passing to run coverage in the UI.

[MattTheCuber]

My root directory is test_coverage.

I also ran the full command exactly as it shows in the terminal excluding the -p argument: pytest --cov=. --cov-report term-missing --rootdir=/home/mvine/Documents/test_coverage /home/mvine/Documents/test_coverage/app/tests/test_main.py::test and got the same results. My suspicions are that either the plugin is doing something or the --cov=. is not working because it's using a relative path. However, I tested running the previous command from /home/mvine/Documents/test_coverage/app/tests, /home/mvine/Documents/test_coverage/app, /home/mvine/Documents/test_coverage, and /home/mvine/Documents without replicating the problem.

[eleanorjboyd]

I wondering if it has to do with the plugins you have enabled because I am now seeing the error "unable to read testIds from temp fileNo...". Also I see you added --cov-report term-missing to your run in terminal, are you expecting this to be run in the extension because I don't see that in the logs.

[MattTheCuber]

I will create a new virtual environment trying to replicate VS Code's as closely as possible. But the test coverage is correct in the terminal, VS Code's is wrong. Do you know how I can get the vscode pytest plugin?

I think I can remove the term missing stuff from the command as well. That's just the command I normally run.

[MattTheCuber]

I created a new virtual environment, ran pip install monai pytest pytest-cov and the integrated test reported 100% coverage. I then ran pip install faker==22.2.0 pytest-timeout==2.3.1 anyio==4.4.0 pytest-cov==5.0.0 dash==2.16.1 pytest-xdist==3.5.0 pytest-drop-dup-tests==1.0.0 pytest-mock==3.14.0 typeguard==4.3.0 to install the identical plugins and it still reported 100% coverage. I am now going through every library in my massive virtual environment to find which is causing the problem 😩

[MattTheCuber]

It's opencv-python!

1. pip install opencv-python pytest pytest-cov
2. Have a file with the line import cv2 (change monai to cv2)
3. Test coverage breaks

pip freeze

coverage==7.6.4
iniconfig==2.0.0
numpy==2.1.2
opencv-python==4.10.0.84
packaging==24.1
pluggy==1.5.0
pytest==8.3.3
pytest-cov==6.0.0

This now throws a couple warnings.

Pretty sure it can be ignored because it still works properly in the terminal.

[MattTheCuber]

Yeah, the warnings are fine: nedbat/coveragepy#1653

[MattTheCuber]

Aha, problem solved! Adding the config file as described in this comment resolves the problem: nedbat/coveragepy#1653 (comment)! (I opened an issue on the opencv-python repo as well)

This is probably still a problem though because my terminal doesn't match the integrated test coverage.

[eleanorjboyd]

Hi @MattTheCuber, thanks for all your investigation help! I have just put in the a fix for the NoSource issue. Are you seeing the same issues with your terminal not matching like above or does it look different now? If it looks different and you could send an updated screenshot that would be great. One note is you might see rounding discrepancies in the coverage numbers between terminal and the sidebar so just an fyi there.

[MattTheCuber]

Which version of the extension should I test?