Some PRs listed on Dependency Dashboard are not linked to individual PRs when more than 1000 PRs old

[blowery]

What Renovate type are you using?

App, Github

Describe the bug

See Automattic/wp-calypso#37419
Some renovate updates there are not linked to the individual PRs. It would be nice if they were all linked.

Did you see anything helpful in debug logs?

Nope

To Reproduce

See Automattic/wp-calypso#37419

Additional context

[rarkins]

First check: are all the unlinked ones actually opened? Or perhaps are they mistakenly listed as opened when they're not?

[blowery]

Some are for sure. For instance Automattic/wp-calypso#35623 is listed on the master issue, but not linked. Also Automattic/wp-calypso#34935

[rarkins]

Up to date master issue link: Automattic/wp-calypso#37527

Screenshot of master issue open issues (some are unliked):

Screenshot of actual open issues, sorted from newest to oldest:

[rarkins]

I don't think it's a coincidence that the unlinked ones are the oldest ones. I'm guessing it's because we don't fetch every PR, and this repository has a really high number of PRs.

[rarkins]

Confirmed, we fetch only 1000 PRs:

{"level":20,"msg":"Retrieving PR list","time":"2019-12-11T09:02:58.885Z"}
{"level":20,"msg":"Retrieved 1000 Pull Requests","time":"2019-12-11T09:03:11.304Z"}

Unfortunately I can't fix this without first refactoring how we fetch PRs, which is a little bit of technical debt I've been meaning to get to. I'll leave this open for addressing afterwards.

[blowery]

Thanks for looking into this!

[MaronHatoum]

This should be resolved in Renovate release 25.37.1 (issue: #5668)
@rarkins FYI

[rarkins]

@MaronHatoum are you sure? I thought we limit our fetch to 1000 PRs, so not sure how we would otherwise find them?

[MaronHatoum]

After checking the code I don't see any limitation and getting PRs from GitHub has been refactored recently,
is there another place where there is a limitation?

[rarkins]

Our pagination defaults to 10 pages by default and I think it's used for PRs too

[MaronHatoum]

hi @rarkins
the code was refactored it calls an unlimited number of pages and gets 20 PR per page.
it is a while loop I don't see any restriction to 10 pages at all.

[MaronHatoum]

renovate/lib/modules/platform/github/pr.ts

Lines 93 to 107 in ae9714a

	let pageIdx = 1;
	while (needNextPageFetch && needNextPageSync) {
	const urlPath = `repos/${repo}/pulls?per_page=20&state=all&sort=updated&direction=desc&page=${pageIdx}`;
	const opts: GithubHttpOptions = { paginate: false };
	if (pageIdx === 1) {
	if (is.emptyArray(prApiCache.getItems())) {
	// Speed up initial fetch
	opts.paginate = true;
	}
	}
	const res = await http.getJson<GhRestPr[]>(urlPath, opts);
	apiQuotaAffected = true;
	requestsTotal += 1;

This is the refactored code and it is using the new HTTP(GOT) request instead of the old github.ts request and I don't see any limitation in the new request.

I can see the pagination limitation in the old github.ts request but the github/pr.ts is not calling it anymore.

[rarkins]

It should fetch 20 per page only until a date is hit. Otherwise if the cache is empty it uses opts.paginate=true which is passed to the HTTP layer and limits to 10 pages.

@zharinov please confirm. Also can you check that we're still fetching 100 per page when the cache is missing?