GitHub tag matching

[16Martin]

This PR addresses #99 and introduces code intended to replace the current combination of get_github_info() followed by get_matching_tag() which exists only in capycli.bom.findsources.

This approach first tries to match a tag using the original get_matching_tag(). If all the guessing does not yield any results, the algo implicitly falls back to analyzing each tag with get_matching_tag().

[16Martin]

I cannot make the unittest work.

In test_find_golang_url_github() (https://github.com/sw360/capycli/blob/main/tests/test_find_sources.py#L329-L339) we expect the result of find_golang_url() to be 'https://pkg.go.dev/github.com/opencontainers/runc'. I do not understand this.

find_golang_url() has essentially two ways to set source_url, the variable that it ultimately returns:

1. if len(split_version) == 3, in this case the source_url would contain /archive/
2. in any other case source_url is set to the result of get_matching_tags, which is mocked to 'https://github.com/opencontainers/runc/archive/refs/tags/v1.0.1.zip', which also contains /archive/
3. Every non-empty result of get_matching_tag() ends with .zip

How and why does this test work?

As a user, I would not feel good about getting https://pkg.go.dev/github.com/opencontainers/runc as a source URL, but maybe the variable names are misleading?

[16Martin]

Found it. The order in which the unittest sets up its mocks does not match the mock naming.

[16Martin]

Some functionality has been moved to protected methods in order to split the task into smaller, more focused parts. There is only one new public method: FindSources.version_to_github_tag().

The core of this PR are the lines in https://github.com/sw360/capycli/blob/martin/fix-github-tag-matching/capycli/bom/findsources.py#L278-L290:

While the current approach first fetch all tags that belong to a specific project and then passes the full list to get_matching_tag(), this new approach calls get_matching_tag() for each tag for each page of tags. As a result, if the new tag-guessing does not yield any matches,, the algo will match (or not match) the same tag the current approach matches.

The most important additions are lines 290 and following. If get_matching_tag() was unable to match the version to the tag, the logic generates eight possible candidates from the version and then checks if any of these candidates exist in the repo, before moving on to the next tag in the list.

The logic to create the candidates is (to some extend) the inverse of to_semver_string(). The idea is that if any of these candidates exists, get_matching_tag() would yield a positive match. The logic generating the candidates is to some extend the inverse of to_semver_string().

The algo then looks for each candidate in the current result-page and if that local lookup does not yield a match, then the algo queries the GitHub API and specifically asks if a tag with the candidate's name exists. If we can find a match through either of these two lookups, we use that match and stop the search.

With my BOMs, I notice a tremendous speedup. On average the guessing part finds a positive match immediately on the first results page. If it doesn't the API query is successful. Using my BOMs, the algo never fetches the second page of tags from GitHub.

[16Martin]

The first draft of this contribution to address the scaling issue of the current code described in #99

I went through several iterations and attempts and I always ended up in a way to big change to become a sensible contribution. This draft is now a very reduced implementation of the orignal idea. Essentially it runs get_matching_tag() (unchanged!) per results page instead of once on the complete set of results. That way it is still get_matching_tag() that makes all important decisions and therefore we can have a high confidence that the new code produces the same outcome as the old code.

The guessing tags idea has been replaced with a guessing prefix search using the /git/refs endpoint. The idea is that in most cases this will yield a much smaller result than the original get_github_info() and will therefore consume fewer resources and be quicker. In the worst case it makes even more API calls than the original implementation, but usually it is a lot quicker.

I tested this approach in the context of another project of mine where I actually had both implementations enabled concurrently, so I was able to compate the old approach with the new approach. In my test run, there were 1215 calls to get_matching_source_url. In 1210 cases it did yield the exact same result as the original implementation.

In five cases the new approach did yield results while the old apporach didn't yield any results. Since all matching is still done using the unchenged get_matching_tag() I concluded the reason for these 5 additional results is that in the original implementation these tags are not retrieved from Github. It turned out that only 2 out of the 3 use cases use get_github_into() to retrieve the full set of tags. In the third use case get_matching_tag is only passed the first page of results. I assumed this is a bug.

I structured the contribution into smaller commits with extensive commit messages to assist the reviewing.

[tngraf]

Hi @16Martin,
I had a look at your code:

• you may want to run poetry run flake8 and fix all issues.
• you may want to run poetry run codespell . and fix all issues.
• using your code in real live did not work: please run
  poetry run capycli bom findsources -i .\testbom1.json -o .\testbom1s.json -v
  with the attached SBOM ... and make it work.

testbom1.json

[tngraf]

@16Martin
change line 143 to elif "message" in response.json(): should help...

[16Martin]

@16Martin change line 143 to elif "message" in response.json(): should help...

But it will run into problems if we ever see a list with 'message' as an entry. I put isinstance(res, dict) that covers more cases. The issue with my code is that I never tested it without a github token. I am working on a proper unittest for github_request(). Obviously we are lacking coverage.