feat: add option to disable access to cloud metadata services #26411

fgreinacher · 2023-12-22T08:34:53Z

Changes

Adds a config option to disable access to cloud metadata services.

Context

Should Renovate use the GCP/AWS Metadata Service implicitely? #26107

Documentation (please check one with an [x])

I have updated the documentation, or
No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

Code inspection only, or
Newly added/modified unit tests, or
No unit tests but ran on a real repository, or
Both unit tests + ran on a real repository

🛠️ with ❤️ by @siemens

lib/workers/global/initialize.ts

viceice

seems okay to me

lib/workers/global/initialize.ts

fgreinacher

Sorry this took a while. I went ahead with the suggested approach. Let me know what you think 🙇

lib/workers/global/initialize.ts

viceice

you need to do some real tests to validate that the variables are not removed to late. so maybe some of those libs are already initialized.

fgreinacher · 2024-01-12T16:01:21Z

you need to do some real tests to validate that the variables are not removed to late. so maybe some of those libs are already initialized.

Sure, here you go:

GCP

Default behavior (DEBUG_AUTH is for GCP SDK):

❯ DEBUG_AUTH=true pnpm start fgreinacher/renovate-cloud-metadata-services
 INFO: Repository started (repository=fgreinacher/renovate-cloud-metadata-services)
       "renovateVersion": "0.0.0-semantic-release"
 INFO: Dependency extraction complete (repository=fgreinacher/renovate-cloud-metadata-services, baseBranch=main)
       "stats": {
         "managers": {"dockerfile": {"fileCount": 1, "depCount": 2}},
         "total": {"fileCount": 1, "depCount": 2}
       }
GaxiosError: request to http://metadata.google.internal./computeMetadata/v1/instance failed, reason: getaddrinfo ENOTFOUND metadata.google.internal.
    at Gaxios._request (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/gaxios/src/gaxios.ts:183:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async metadataAccessor (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/gcp-metadata/src/index.ts:147:15)
    at async Object.isAvailable (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/gcp-metadata/src/index.ts:376:5)
    at async GoogleAuth._checkIsGCE (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:277:51)
    at async GoogleAuth.getApplicationDefaultAsync (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:248:13)
    at async GoogleAuth.getClient (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:673:17)
    at async GoogleAuth.getAccessToken (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/google-auth-library/build/src/auth/googleauth.js:695:24)
    at async getGoogleAuthToken (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/util.ts:21:25)
    at async getAuthHeaders (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/docker/common.ts:119:20)
    at async DockerDatasource.getManifestResponse (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/docker/index.ts:98:23)
    at async DockerDatasource.getManifest (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/docker/index.ts:276:30)
    at async DockerDatasource.getLabels (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/docker/index.ts:453:24)
    at async /Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/util/cache/package/decorator.ts:119:18
    at async DockerDatasource.getReleases (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/docker/index.ts:1019:20)
    at async getRegistryReleases (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/index.ts:77:15)
    at async fetchReleases (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/modules/datasource/index.ts:289:15)
    at async lookupUpdates (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/workers/repository/process/lookup/index.ts:115:56)
    at async withLookupStats (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/workers/repository/process/fetch.ts:27:18)
    at async fetchDepUpdates (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/lib/workers/repository/process/fetch.ts:72:30) {
  config: {
    url: 'http://metadata.google.internal./computeMetadata/v1/instance',
    headers: { 'Metadata-Flavor': 'Google' },
    retryConfig: {
      noResponseRetries: 0,
      currentRetryAttempt: 0,
      retry: 3,
      httpMethodsToRetry: [Array],
      statusCodesToRetry: [Array]
    },
    params: {},
    responseType: 'text',
    timeout: 3000,
    paramsSerializer: [Function: paramsSerializer],
    validateStatus: [Function: validateStatus],
    method: 'GET',
    errorRedactor: [Function: defaultErrorRedactor]
  },
  response: undefined,
  error: FetchError: request to http://metadata.google.internal./computeMetadata/v1/instance failed, reason: getaddrinfo ENOTFOUND metadata.google.internal.
      at ClientRequest.<anonymous> (/Users/fgreinacher/Code/github.com/fgreinacher/renovate/node_modules/.pnpm/[email protected]/node_modules/node-fetch/lib/index.js:1501:11)
      at ClientRequest.emit (node:events:514:28)
      at ClientRequest.emit (node:domain:488:12)
      at Socket.socketErrorListener (node:_http_client:495:9)
      at Socket.emit (node:events:514:28)
      at Socket.emit (node:domain:488:12)
      at emitErrorNT (node:internal/streams/destroy:151:8)
      at emitErrorCloseNT (node:internal/streams/destroy:116:3)
      at processTicksAndRejections (node:internal/process/task_queues:82:21) {
    type: 'system',
    errno: 'ENOTFOUND',
    code: 'ENOTFOUND'
  },
  code: 'ENOTFOUND'
}
 WARN: Package lookup failures (repository=fgreinacher/renovate-cloud-metadata-services)
       "warnings": [
         "Failed to look up docker package test-account.dkr.ecr.eu-central-1.amazonaws.com/test-image"
       ],
       "files": ["Dockerfile"]
 INFO: Repository finished (repository=fgreinacher/renovate-cloud-metadata-services)
       "cloned": true,
       "durationMs": 6625

Behavior with new option set:

❯ DEBUG_AUTH=true RENOVATE_USE_CLOUD_METADATA_SERVICES=false pnpm start fgreinacher/renovate-cloud-metadata-services
 INFO: Repository started (repository=fgreinacher/renovate-cloud-metadata-services)
       "renovateVersion": "0.0.0-semantic-release"
 INFO: Dependency extraction complete (repository=fgreinacher/renovate-cloud-metadata-services, baseBranch=main)
       "stats": {
         "managers": {"dockerfile": {"fileCount": 1, "depCount": 2}},
         "total": {"fileCount": 1, "depCount": 2}
       }
 WARN: Package lookup failures (repository=fgreinacher/renovate-cloud-metadata-services)
       "warnings": [
         "Failed to look up docker package test-account.dkr.ecr.eu-central-1.amazonaws.com/test-image"
       ],
       "files": ["Dockerfile"]
 INFO: Repository finished (repository=fgreinacher/renovate-cloud-metadata-services)
       "cloned": true,
       "durationMs": 6271

AWS

Default behavior (getInstanceImdsProvider is output I added to the relevant function in the AWS SDK):

❯ pnpm start fgreinacher/renovate-cloud-metadata-services

 INFO: Repository started (repository=fgreinacher/renovate-cloud-metadata-services)
       "renovateVersion": "0.0.0-semantic-release"
 INFO: Dependency extraction complete (repository=fgreinacher/renovate-cloud-metadata-services, baseBranch=main)
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 1, "depCount": 1},
           "regex": {"fileCount": 1, "depCount": 1}
         },
         "total": {"fileCount": 2, "depCount": 2}
       }
getInstanceImdsProvider
 WARN: Package lookup failures (repository=fgreinacher/renovate-cloud-metadata-services)
       "warnings": [
         "Failed to look up aws-machine-image package [{\"Name\":\"owner-id\",\"Values\":[\"602401143452\"]},{\"Name\":\"name\",\"Values\":[\"amazon-eks-node-1.20-*\"]}]"
       ],
       "files": ["amis.txt"]
 INFO: Repository finished (repository=fgreinacher/renovate-cloud-metadata-services)
       "cloned": true,
       "durationMs": 6424

Behavior with new option set:

❯ RENOVATE_USE_CLOUD_METADATA_SERVICES=false pnpm start fgreinacher/renovate-cloud-metadata-services
 INFO: Repository started (repository=fgreinacher/renovate-cloud-metadata-services)
       "renovateVersion": "0.0.0-semantic-release"
 INFO: Dependency extraction complete (repository=fgreinacher/renovate-cloud-metadata-services, baseBranch=main)
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 1, "depCount": 1},
           "regex": {"fileCount": 1, "depCount": 1}
         },
         "total": {"fileCount": 2, "depCount": 2}
       }
 WARN: Package lookup failures (repository=fgreinacher/renovate-cloud-metadata-services)
       "warnings": [
         "Failed to look up aws-machine-image package [{\"Name\":\"owner-id\",\"Values\":[\"602401143452\"]},{\"Name\":\"name\",\"Values\":[\"amazon-eks-node-1.20-*\"]}]"
       ],
       "files": ["amis.txt"]
 INFO: Repository finished (repository=fgreinacher/renovate-cloud-metadata-services)
       "cloned": true,
       "durationMs": 6141

renovate-release · 2024-01-12T16:46:36Z

🎉 This PR is included in version 37.130.0 🎉

The release is available on:

GitHub release
37.130.0

Your semantic-release bot 📦🚀

…tebot#26411)

feat: add option to disable access to cloud metadata services

7eb6b12

fgreinacher marked this pull request as draft December 22, 2023 08:35

fgreinacher commented Dec 22, 2023

View reviewed changes

lib/workers/global/initialize.ts Outdated Show resolved Hide resolved

viceice reviewed Dec 26, 2023

View reviewed changes

lib/workers/global/initialize.ts Show resolved Hide resolved

chore: add tests and documentation

9fa8199

fgreinacher marked this pull request as ready for review January 9, 2024 11:25

fgreinacher commented Jan 9, 2024

View reviewed changes

lib/workers/global/initialize.ts Show resolved Hide resolved

fgreinacher requested a review from viceice January 9, 2024 11:27

fgreinacher force-pushed the feat/disable-cloud-metadata-services branch from 2f69963 to 9fa8199 Compare January 9, 2024 22:45

viceice reviewed Jan 10, 2024

View reviewed changes

fgreinacher added 2 commits January 10, 2024 16:24

chore: add comment for deviation from early return style

3065376

Merge branch 'main' into feat/disable-cloud-metadata-services

f29eccc

fgreinacher requested a review from viceice January 12, 2024 16:21

viceice approved these changes Jan 12, 2024

View reviewed changes

viceice added this pull request to the merge queue Jan 12, 2024

Merged via the queue into renovatebot:main with commit 83b545f Jan 12, 2024
34 checks passed

fgreinacher deleted the feat/disable-cloud-metadata-services branch January 12, 2024 19:57

HonkingGoose mentioned this pull request Jan 15, 2024

docs: rewrite useCloudMetadataServices #26646

Merged

6 tasks

zT-1337 pushed a commit to scm-manager/renovate that referenced this pull request Jan 22, 2024

feat: add option to disable access to cloud metadata services (renova…

f3cd86c

…tebot#26411)

zT-1337 pushed a commit to scm-manager/renovate that referenced this pull request Jan 24, 2024

feat: add option to disable access to cloud metadata services (renova…

2c1322a

…tebot#26411)

zT-1337 pushed a commit to scm-manager/renovate that referenced this pull request Jan 24, 2024

feat: add option to disable access to cloud metadata services (renova…

4d28c9f

…tebot#26411)

github-actions bot locked as resolved and limited conversation to collaborators Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add option to disable access to cloud metadata services #26411

feat: add option to disable access to cloud metadata services #26411

fgreinacher commented Dec 22, 2023 •

edited by viceice

Loading

viceice left a comment

fgreinacher left a comment

viceice left a comment

fgreinacher commented Jan 12, 2024 •

edited

Loading

renovate-release commented Jan 12, 2024

feat: add option to disable access to cloud metadata services #26411

feat: add option to disable access to cloud metadata services #26411

Conversation

fgreinacher commented Dec 22, 2023 • edited by viceice Loading

Changes

Context

Documentation (please check one with an [x])

How I've tested my work (please select one)

viceice left a comment

Choose a reason for hiding this comment

fgreinacher left a comment

Choose a reason for hiding this comment

viceice left a comment

Choose a reason for hiding this comment

fgreinacher commented Jan 12, 2024 • edited Loading

GCP

AWS

renovate-release commented Jan 12, 2024

fgreinacher commented Dec 22, 2023 •

edited by viceice

Loading

fgreinacher commented Jan 12, 2024 •

edited

Loading