Retrieve SARIF errors and warnings correctly

[bdovaz]

With @nvuillam's request I have:

• Return the number of warnings (which it didn't do)
• Correct the calculation of counting the errors or warnings according to: https://github.com/microsoft/sarif-tutorials/blob/main/docs/2-Basics.md#results
  • If the property “level” exists in the object “result” (some linters like gitleaks does not fill it) it counts the number of objects in the array “locations”.
  • If the property “level” does not exist, it only counts the results for the level “warning”, because by default in the specification it is set like this: https://json.schemastore.org/sarif-2.1.0.json
• Add sarif to cli_lint_errors_count in json scheme (was implemented in code but not added to schema)
• Add sarif to cli_lint_warnings_count in json scheme
• Share in two functions (regex and sarif) this logic

There are linters that do not fill out the SARIF level field:

llaville/sarif-php-converters#1 (Fixed!)

pmd/pmd#5573 (Fixed!!)

anchore/grype#2511 (Fixed!!)

gitleaks/gitleaks#1858

In these cases if they do not fill in the “level” field as I have put in the first message, according to the specification the default value is “warning” so it will be reported as such.

There are other cases Python Bandit that fills levels that we do not show in the reporters like “note” level, in this case I make it to be interpreted as warning that is the closest one.

That is, before this PR everything was interpreted as error, now:

• If they do not fill in the property “level” it will be reported as warning as described in the specification.
• If they fill it as “error” it will be reported as error.
• If they fill it as “warning” it will be reported as warning.
• If they fill it as “note” it will be reported as a warning because it is the closest one we have in the reporters.

[github-actions]

🦙 MegaLinter status: ⚠️ WARNING

Descriptor	Linter	Files	Fixed	Errors	Warnings	Elapsed time
✅ API	spectral	1		0	0	1.6s
⚠️ BASH	bash-exec	6		1	0	0.02s
✅ BASH	shellcheck	6		0	0	0.13s
✅ BASH	shfmt	6	0	0	0	0.66s
✅ COPYPASTE	jscpd	yes		no	no	2.81s
✅ DOCKERFILE	hadolint	131		0	0	28.74s
✅ JSON	jsonlint	20		0	0	0.22s
✅ JSON	v8r	22		0	0	12.49s
⚠️ MARKDOWN	markdownlint	269	0	306	0	20.74s
✅ MARKDOWN	markdown-table-formatter	269	0	0	0	140.63s
⚠️ PYTHON	bandit	222		67	0	5.42s
✅ PYTHON	black	222	0	0	0	5.3s
✅ PYTHON	flake8	222		0	0	2.08s
✅ PYTHON	isort	222	0	0	0	1.4s
✅ PYTHON	mypy	222		0	0	11.34s
✅ PYTHON	pylint	222		0	0	28.68s
✅ PYTHON	ruff	222	0	0	0	0.66s
✅ REPOSITORY	checkov	yes		no	no	37.86s
✅ REPOSITORY	git_diff	yes		no	no	0.62s
⚠️ REPOSITORY	grype	yes		27	no	30.03s
✅ REPOSITORY	secretlint	yes		no	no	8.76s
✅ REPOSITORY	syft	yes		no	no	2.39s
✅ REPOSITORY	trivy	yes		no	no	11.27s
✅ REPOSITORY	trivy-sbom	yes		no	no	22.06s
✅ REPOSITORY	trufflehog	yes		no	no	4.64s
✅ SPELL	cspell	730		0	0	14.19s
⚠️ SPELL	lychee	351		43	0	28.04s
✅ XML	xmllint	3	0	0	0	1.33s
✅ YAML	prettier	160	0	0	0	4.36s
✅ YAML	v8r	103		0	0	20.26s
✅ YAML	yamllint	161		0	0	3.8s

See detailed report in MegaLinter reports

[bdovaz]

@nvuillam @echoix is ready for review!

It's a PR that I've been working on for 3 months and that has led me to do a lot of research linter by linter due to the fact that there were linters that didn't report the “level” and I've even had to do PRs to correct it in several of them.

Right now the builds are failing because of 2 things:

• Trivy that is unrelated to this PR
• gitleaks which fails because it doesn't report the level but I doubt it will because the linter itself doesn't allow to change the log level, it always reports everything as warning.

In the second case I'm afraid we will have to make a workaround to ignore this part in the test because it is impossible that it can report errors as expected by the test:

megalinter/megalinter/utilstest.py

Line 306 in 6e89c16

mega_linter_linter.total_number_errors > 1,

[echoix]

Looks fine, but indeed, gitleaks should be addressed to merge. Why does it need to report an error, if the tool doesn't report as an error? + How does it handle the setting warning as errors? Like with this PR, how can someone configure ML to fail when gitleaks finds something?

[nvuillam]

Amazing job @bdovaz , i'll do my best to look at it this weekend :)

[bdovaz]

Looks fine, but indeed, gitleaks should be addressed to merge. Why does it need to report an error, if the tool doesn't report as an error?

This code I mention was already there, I have not modified it:

megalinter/megalinter/utilstest.py

Line 306 in 5a91fb0

mega_linter_linter.total_number_errors > 1,

That's why I say, for it to pass the gitleaks test, we need to put a workaround "hack" condition (that I see that in these and similar classes there are already several) in that test because it is impossible for it to report errors as I mentioned.

How does it handle the setting warning as errors? Like with this PR, how can someone configure ML to fail when gitleaks finds something?

As you can see in my changes, I have only touched sarif's error and warnings calculation nothing else, everything you mention I guess it won't affect it because it will be in a higher layer that interprets those settings you mention but @nvuillam will know better.