Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend Maven datasource caching #30419

Closed
rarkins opened this issue Jul 27, 2024 · 7 comments
Closed

Extend Maven datasource caching #30419

rarkins opened this issue Jul 27, 2024 · 7 comments
Assignees
Labels
core:cache datasource:maven priority-2-high Bugs impacting wide number of users or very important features type:feature Feature (new functionality)

Comments

@rarkins
Copy link
Collaborator

rarkins commented Jul 27, 2024

Describe the proposed change(s).

The Maven datasource does not appear to have caching = true set. When run on a repo like https://github.com/renovate-reproductions/net.twisterrob.gradle caching does not seem to be working or efficient, e.g. repeated runs still take 42 seconds for lookup on my laptop, even though all registries are public ones.

Logs should 0 set cache methods performed. Ideally all results should be cached such that potentially 0 requests to registries are required when the cache is "hot". On the other hand I note a comment // cache non-null responses unless marked as private and I'm not sure why we're being so conservative about not caching null.

We need to:

  • Review the Maven datasource to ensure isPrivate is set reliably, and
  • Enable caching
@rarkins rarkins added type:feature Feature (new functionality) priority-2-high Bugs impacting wide number of users or very important features datasource:maven labels Jul 27, 2024
@rarkins
Copy link
Collaborator Author

rarkins commented Jul 27, 2024

Related: #30420

@rarkins
Copy link
Collaborator Author

rarkins commented Jul 27, 2024

If we achieve 100% caching for this repo but it still seems that the lookup is subjectively slow, create a new issue for further investigation.

@rarkins
Copy link
Collaborator Author

rarkins commented Jul 27, 2024

I played around with this diff:

diff --git a/lib/modules/datasource/index.ts b/lib/modules/datasource/index.ts
index 006a0023c..4f677016c 100644
--- a/lib/modules/datasource/index.ts
+++ b/lib/modules/datasource/index.ts
@@ -70,7 +70,7 @@ async function getRegistryReleases(
       cacheKey,
     );
     // istanbul ignore if
-    if (cachedResult) {
+    if (cachedResult !== undefined) {
       logger.trace({ cacheKey }, 'Returning cached datasource response');
       return cachedResult;
     }
@@ -80,13 +80,13 @@ async function getRegistryReleases(
     res.registryUrl ??= registryUrl;
   }
   // cache non-null responses unless marked as private
-  if (datasource.caching && res) {
+  if (datasource.caching) {
     const cachePrivatePackages = GlobalConfig.get(
       'cachePrivatePackages',
       false,
     );
-    if (cachePrivatePackages || !res.isPrivate) {
       logger.trace({ cacheKey }, 'Caching datasource response');
+    if (cachePrivatePackages || !res?.isPrivate) {
       const cacheMinutes = 15;
       await packageCache.set(cacheNamespace, cacheKey, res, cacheMinutes);
     }
diff --git a/lib/modules/datasource/maven/index.ts b/lib/modules/datasource/maven/index.ts
index 3476ded21..4d2f18de7 100644
--- a/lib/modules/datasource/maven/index.ts
+++ b/lib/modules/datasource/maven/index.ts
@@ -68,6 +68,8 @@ export class MavenDatasource extends Datasource {
 
   override readonly defaultVersioning: string = mavenVersioning.id;
 
+  override readonly caching = true;
+
   override readonly registryStrategy: RegistryStrategy = 'merge';
 
   override readonly releaseTimestampSupport = true;

@zharinov
Copy link
Collaborator

For null values, we can't determine whether it's private or not, so I think it's better not to cache nulls

@rarkins
Copy link
Collaborator Author

rarkins commented Aug 15, 2024

You mean private but missing credentials? Why not cache that?

@zharinov
Copy link
Collaborator

I mean null results for private packages

@rarkins
Copy link
Collaborator Author

rarkins commented Aug 15, 2024

What can go wrong if we cache those?

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
core:cache datasource:maven priority-2-high Bugs impacting wide number of users or very important features type:feature Feature (new functionality)
Projects
None yet
Development

No branches or pull requests

3 participants