Add support for fetching meta data from deps.dev

[n1ckl0sk0rtge]

Description

This PR adds basic capabilities to fetch meta data from deps.dev for a given PURL. With this code, only the related source code repositories will be fetched and stored.

• I have read and understand the contributing guidelines
• This PR fixes a defect, and I have provided tests to verify that the fix is effective
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces changes to the database model, and I have updated the migration changelog accordingly
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +2.69% (target: -1.00%)	✅ 87.50% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (41867be)	172	140	81.40%
Head commit (472841c)	6912 (+6740)	5812 (+5672)	84.09% (+2.69%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1457)	48	42	87.50%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

[nscuro]

Thanks @n1ckl0sk0rtge! I've not forgotten about this PR, I'll try to get it reviewed this weekend! Apologies for the delay.

[nscuro]

The analyzer would still need to be registered with RepositoryAnalyzerFactory, based on the PURL types it supports:

hyades/repository-meta-analyzer/src/main/java/org/dependencytrack/repometaanalyzer/repositories/RepositoryAnalyzerFactory.java

Lines 33 to 44 in 941dd0f

	private static final Map<String, Supplier<IMetaAnalyzer>> ANALYZER_SUPPLIERS = Map.of(
	PackageURL.StandardTypes.COMPOSER, ComposerMetaAnalyzer::new,
	PackageURL.StandardTypes.GEM, GemMetaAnalyzer::new,
	PackageURL.StandardTypes.GOLANG, GoModulesMetaAnalyzer::new,
	PackageURL.StandardTypes.HEX, HexMetaAnalyzer::new,
	PackageURL.StandardTypes.MAVEN, MavenMetaAnalyzer::new,
	PackageURL.StandardTypes.NPM, NpmMetaAnalyzer::new,
	PackageURL.StandardTypes.NUGET, NugetMetaAnalyzer::new,
	PackageURL.StandardTypes.PYPI, PypiMetaAnalyzer::new,
	PackageURL.StandardTypes.CARGO, CargoMetaAnalyzer::new,
	"cpan", CpanMetaAnalyzer::new
	);

The current model assumes at most one analyzer per PURL type though, so we'd need to adjust this to support multiple. Question then is, should one take priority over the other? Do we execute all applicable analyzers, and if so, how do we merge all results back into one?

[nscuro]

I'm wondering if perhaps we should switch to deps.dev entirely for all public components. The internal status is already provided to the repository meta analyzer as per Protobuf definition:

hyades/proto/src/main/proto/org/dependencytrack/repometaanalysis/v1/repo_meta_analysis.proto

Lines 45 to 47 in 941dd0f

	// Whether the component is internal to the organization.
	// Internal components will only be looked up in internal repositories.
	optional bool internal = 2;

In that case, we would not only source the repository from deps.dev, but also:

• Latest version
• Publish timestamp of latest version
• Publish timestamp of current version
• Hashes of current version (not sure if deps.dev provides that?)

I think that would be simpler than trying to support multiple analyzers per PURL.

Does that sound reasonable? Granted that change would be a bit larger.

[n1ckl0sk0rtge]

@nscuro took me a while to review. Yes, thats a good point. So do you think the Map can hold List<Supplier<IMetaAnalyzer>> and that we add some priority mechanism to IMetaAnalyzer? What we could also do is having DepsDevMetaAnalyzer as a base class for all the other MetaAnayzer. This would result in having a basic implementation for all PURL typs in DepsDevMetaAnalyzer which can be specialised and overridden. Then using the same ANALYZER_SUPPLIERS-map we can select and migrate step by step for each purl type.

The analyzer would still need to be registered with RepositoryAnalyzerFactory, based on the PURL types it supports:

hyades/repository-meta-analyzer/src/main/java/org/dependencytrack/repometaanalyzer/repositories/RepositoryAnalyzerFactory.java

Lines 33 to 44 in 941dd0f

	private static final Map<String, Supplier<IMetaAnalyzer>> ANALYZER_SUPPLIERS = Map.of(
	PackageURL.StandardTypes.COMPOSER, ComposerMetaAnalyzer::new,
	PackageURL.StandardTypes.GEM, GemMetaAnalyzer::new,
	PackageURL.StandardTypes.GOLANG, GoModulesMetaAnalyzer::new,
	PackageURL.StandardTypes.HEX, HexMetaAnalyzer::new,
	PackageURL.StandardTypes.MAVEN, MavenMetaAnalyzer::new,
	PackageURL.StandardTypes.NPM, NpmMetaAnalyzer::new,
	PackageURL.StandardTypes.NUGET, NugetMetaAnalyzer::new,
	PackageURL.StandardTypes.PYPI, PypiMetaAnalyzer::new,
	PackageURL.StandardTypes.CARGO, CargoMetaAnalyzer::new,
	"cpan", CpanMetaAnalyzer::new
	);

The current model assumes at most one analyzer per PURL type though, so we'd need to adjust this to support multiple. Question then is, should one take priority over the other? Do we execute all applicable analyzers, and if so, how do we merge all results back into one?

[nscuro]

What we could also do is having DepsDevMetaAnalyzer as a base class for all the other MetaAnayzer. This would result in having a basic implementation for all PURL typs in DepsDevMetaAnalyzer which can be specialised and overridden. Then using the same ANALYZER_SUPPLIERS-map we can select and migrate step by step for each purl type.

Yes, I really like this idea!