Console reporting even when saving to json file and other questions

[echoix]

I'm trying to integrate slipcover instead of a coverage.py, as coverage.py takes an already too long 4*35 minutes without coverage (test suite split into 4 CI jobs), to about 45-50+ min without coverage.py.

With slipcover, the times taken are about the same as without coverage, so it's great.

I'll have a couple questions, as I did my best to understand how this tool works, as there isn't a documentation for the project yet, and I tried to follow the couple issues beforehand.

I've been able to upload to codecov, as long as I set the output file name to coverage.json (if it is called .coverage.json, codecov tries to call coverage.py to merge files as it looks like if it were multiple files created with the parallel option, since it is like the .coverage.* pattern).

So, knowing that there is a way to have a human, tabular report in slipcover, how is it possible to request it to be shown?
The relevant function I think is print_coverage located here

slipcover/src/slipcover/slipcover.py

Lines 96 to 148 in 7f435c1

	def print_coverage(coverage, *, outfile=sys.stdout, missing_width=None, skip_covered=False) -> None:
	"""Prints coverage information for human consumption."""
	from tabulate import tabulate
	if not coverage.get('files', None): # includes empty coverage['files']
	return
	branch_coverage = coverage.get('meta', {}).get('branch_coverage', False)
	def table():
	for f, f_info in sorted(coverage['files'].items()):
	exec_l = len(f_info['executed_lines'])
	miss_l = len(f_info['missing_lines'])
	if branch_coverage:
	exec_b = len(f_info['executed_branches'])
	miss_b = len(f_info['missing_branches'])
	pct_b = 100*exec_b/(exec_b+miss_b) if (exec_b+miss_b) else 0
	pct = f_info['summary']['percent_covered']
	if skip_covered and pct == 100.0:
	continue
	yield [f, exec_l+miss_l, miss_l,
	*([exec_b+miss_b, miss_b, round(pct_b)] if branch_coverage else []),
	round(pct),
	format_missing(f_info['missing_lines'], f_info['executed_lines'],
	f_info['missing_branches'] if 'missing_branches' in f_info else [])]
	if len(coverage['files']) > 1:
	yield ['---'] + [''] * (6 if branch_coverage else 4)
	s = coverage['summary']
	if branch_coverage:
	exec_b = s['covered_branches']
	miss_b = s['missing_branches']
	pct_b = 100*exec_b/(exec_b+miss_b) if (exec_b+miss_b) else 0
	yield ['(summary)', s['covered_lines']+s['missing_lines'], s['missing_lines'],
	*([exec_b+miss_b, miss_b, round(pct_b)] if branch_coverage else []),
	round(s['percent_covered']), '']
	print("", file=outfile)
	headers = ["File", "#lines", "#l.miss",
	*(["#br.", "#br.miss", "brCov%", "totCov%"] if branch_coverage else ["Cover%"]),
	"Missing"]
	maxcolwidths = [None] * (len(headers)-1) + [missing_width]
	print(tabulate(table(), headers=headers, maxcolwidths=maxcolwidths), file=outfile)

Up to now, I wasn't able to see that it in action yet.

It might be related to the logic in the function printit inside atexit here:

slipcover/src/slipcover/__main__.py

Lines 202 to 210 in 7f435c1

	def sci_atexit():
	global output_tmpfile
	def printit(coverage, outfile):
	if args.json:
	print(json.dumps(coverage, indent=(4 if args.pretty_print else None)), file=outfile)
	else:
	sc.print_coverage(coverage, outfile=outfile, skip_covered=args.skip_covered,
	missing_width=args.missing_width)

• If I understood correctly, on my first try, just replacing python interpreter by python -m slipcover ran some coverage, but didn't save any data of it, and didn't display anything too. (Hopefully I only ran a subset of the tests instead of 35 minutes ;) )
• If I use python -m slipcover --json in order to have a file output (or as I thought it would), it only outputs a json to the terminal, it doesn't save it with a default name. There is still no file created that could be used to post-process what slipcover collected, either with slipcover or any custom-purpose tool.
• If I use python -m slipcover --out coverage.json, then a json file is saved, but no terminal output. However, that file is empty (0 bytes).
• If I use python -m slipcover --json --out coverage.json, then there is a file saved, no terminal output, and the file is valid, but didn't contain any files covered.
• If I use python -m slipcover --json --out coverage.json --source ., then there is a file saved, no terminal output, and the file is valid, and contains some files, (with missing lines reported), but no lines covered (that's on my side, as the source should be better tuned with the paths like I needed with coverage.py)

Other questions, in no particular order

• Does slipcover save the data collected somewhere?
• Can slipcover work without specifying any --source argument, by only saving what is in under the current directory, a bit like other coverage tools?
• Is slipcover's json output summarized? If not, and the complete data is there, it might be possible to use the couple functions in https://coverage.readthedocs.io/en/latest/api_coveragedata.html and be able to create a .coverage file just like coverage.py uses, and use coverage.py to use all their reporting facilities. For example, I already had to create a custom path mapping function since the mapping available from their config file worked on path prefixes, and I needed to change some file names too. https://github.com/OSGeo/grass/blob/b082d422ef215c21a0fcdaf3a0a5b38cc7c51ea1/utils/coverage_mapper.py Their format is now an SQLite database, and is quite simple to understand. If it's realistic, I could try to make the missing integration if you point me to what your data structures are equivalent to their arcs and lines. Maybe slipcover could be used as a tracer plugin even, to see...
• Is there any mechanism to map paths back to files in the repo, like multiple other coverage tools (outside of Python, like dotnet and Java)? If not, creating a coverage.py file from the api as above would allow to use their mechanisms with a simple coverage combine or even directly with any of their reporting options.
• Can the --source option be used multiple times, or it must be used once with comma separating the different paths?

[echoix]

Sorry if it's a bit unordered, I wrote it on my phone, launching extra CI runs while writing to get more answers

[echoix]

I've did a little PoC for the following:

python -m slipcover --json --out cov_data/slipcover_raytrace_orig.json benchmarks/bm_raytrace.py
python benchmarks/create_bm_raytrace_slipcover.py
coverage json --data-file=cov_data/.coverage_raytrace_slipcover -o cov_data/slipcover_raytrace_created.json
python -m coverage run --include=benchmarks/bm_raytrace.py benchmarks/bm_raytrace.py

The file (benchmarks/create_bm_raytrace_slipcover.py):

from coverage import CoverageData
import json
from pathlib import Path


def main():
    b = json.load(Path("cov_data/slipcover_raytrace_orig.json").open("r"))
    a = CoverageData("cov_data/.coverage_raytrace_slipcover")
    a.erase()
    a.write()

    files_before = b["files"]
    print('b["files"]: ', b["files"])
    line_data = {
        str(Path().joinpath(file).resolve()): data["executed_lines"]
        for file, data in b["files"].items()
    }
    print("line_data: ", line_data)
    a.add_lines(line_data)
    a.write()


if __name__ == "__main__":
    main()

The json files are the same:

I'll try with branch coverage enable next.

[echoix]

I didn't end up finishing converting json file with branch coverage yesterday. I ended up that I had to convert a list of lists (pairs) (in the "executed_branches" of each file) to a collection of tuples (it could still remain a list).

But anyways, the format seemed the correct way, so it means the json output of slipcover contains enough info to recreate a coveragepy data file, that allows to fully use coverage.py's reporting facilities. And thus would allow me to remap paths using coverage.py's config file and using their commands.