Preview with refresh failing with Authentication errors #1339

flostadler · 2025-01-16T15:29:17Z

Describe what happened

A user reported that v4.6.0 of the provider seems to have a bug in how authentication is working during preview with refresh.

They've configured the provider with temporary tokens exposed via env variables:

    docker_provider = docker.Provider(
            name,
            registry_auth=[
                docker.ProviderRegistryAuthArgs(
                    address=os.environ.get("CI_REGISTRY"),
                    username=os.environ.get("CI_REGISTRY_USER"),
                    password=os.environ.get("CI_REGISTRY_PASSWORD"),
                ),
            ],
        )

With the provider < 4.6.0, everything seems fine, but once they upgrade, the pulumi preview job with automatic refresh keeps failing with a 401 error when trying to connect to the gitlab registry:

error: Program failed with an unhandled exception:
Traceback (most recent call last):
[...]
    gitlab_image = docker.get_registry_image(name=f"{args.docker_image_source}:{args.docker_image_tag}", opts=default_invoke_options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulumi_docker/get_registry_image.py", line 114, in get_registry_image
    __ret__ = pulumi.runtime.invoke('docker:index/getRegistryImage:getRegistryImage', __args__, opts=opts, typ=GetRegistryImageResult).value
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/invoke.py", line 127, in invoke
    return _sync_await(awaitableInvokeResult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/sync_await.py", line 66, in _sync_await
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/invoke.py", line 356, in wait_for_fut
    return await asyncio.ensure_future(do_rpc())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/invoke.py", line 352, in do_rpc
    raise error
Exception: invoke of docker:index/getRegistryImage:getRegistryImage failed: invocation of docker:index/getRegistryImage:getRegistryImage returned an error: invoking docker:index/getRegistryImage:getRegistryImage: 1 error occurred:
	* Got error when attempting to fetch image version [...] from registry: Got bad response from registry: 401 Unauthorized

Sample program

n/a

Log output

No response

Affected Resource(s)

No response

Output of `pulumi about`

tbd

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

flostadler · 2025-01-16T15:33:01Z

Hey @asvinours, thanks for bringing this issue up. Could you please run pulumi about in the directory of your pulumi project and post the output here?

Could you also help me reproduce this issue? Ideally a sample pulumi program and necessary steps that will show the broken behavior.

flostadler · 2025-01-16T16:36:38Z

I tried reproducing this, but couldn't. I used ECR as an example to test:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as docker from "@pulumi/docker";

const repo = new aws.ecr.Repository("my-repo", {
    forceDelete: true,
});

const prov = new docker.Provider("docker-prov", {
    registryAuth: [{
        address: process.env.CI_REGISTRY!,
        username: process.env.CI_REGISTRY_USER!,
        password: process.env.CI_REGISTRY_PASSWORD!,
    }]
});

const image = new docker.Image("my-image", {
    build: {
        context: "app",
    },
    imageName: pulumi.interpolate`${repo.repositoryUrl}:latest`,
    registry: {
        server: process.env.CI_REGISTRY!,
        username: process.env.CI_REGISTRY_USER!,
        password: process.env.CI_REGISTRY_PASSWORD!,
    }
});

export const remoteImage = docker.getRegistryImageOutput({
    name: pulumi.interpolate`${repo.repositoryUrl}:latest`,
}, { provider: prov, dependsOn: [image] }).sha256Digest;

I exported the following env variables (replace AWS_ACCOUNT_ID):

export CI_REGISTRY_USER="AWS"
export CI_REGISTRY="https://AWS_ACCOUNT_ID.dkr.ecr.eu-central-1.amazonaws.com"
export CI_REGISTRY_PASSWORD="$(aws ecr get-login-password --region eu-central-1)"

After that I did the following:

pulumi up to deploy everything
export CI_REGISTRY_PASSWORD="$(aws ecr get-login-password --region eu-central-1)" to fetch and expose a new password
PULUMI_DEBUG_GRPC="logs.jsonl" pulumi preview --refresh
After inspecting the GRPC logs, I can see that the docker provider receives the newly updated credentials in the Configure GRPC call.

asvinours · 2025-01-16T17:35:45Z

Hey, here's the output of pulumi about:

pulumi about
CLI          
Version      3.136.1
Go Version   go1.23.2
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  aws     6.66.3
resource  docker  4.6.0
language  python  3.136.1
resource  random  4.16.8
resource  vault   6.4.0

Host     
OS       darwin
Version  14.6.1
Arch     x86_64

This project is written in python: executable='/Users/fbaumann/projects/poc/pulumi-docker-provider/.venv/bin/python3' version='3.12.6'

Backend        
Name           MP16FBAUMANN1.local
URL            file://
User           fbaumann
Organizations  
Token type     personal

Dependencies:
NAME           VERSION
pulumi_aws     6.66.3
pulumi_docker  4.6.0
pulumi_random  4.16.8
pulumi_vault   6.4.0

Here's the code to reproduce the issue, I tried to keep it somewhat close to our real application:

from __future__ import annotations

import os

from pulumi import ComponentResource
from pulumi import InvokeOptions
from pulumi import ResourceOptions
import pulumi_docker as docker


class CustomComponentTwo(ComponentResource):
    def __init__(self, name: str, opts: ResourceOptions = None) -> None:
        super().__init__("custom:component:two", name, None, opts)

        default_resource_options = ResourceOptions(parent=self)
        default_invoke_options = InvokeOptions(parent=self)

        # removed the actual image name here - needs to be replaced with a proper image reference
        gitlab_image = docker.get_registry_image(name="[...]", opts=default_invoke_options)
        docker.RemoteImage(
            name,
            name=gitlab_image.name,
            pull_triggers=[gitlab_image.sha256_digest],
            opts=default_resource_options,
        )


class CustomComponentOne(ComponentResource):
    def __init__(self, name: str, opts: ResourceOptions = None) -> None:
        super().__init__("custom:component:one", name, None, opts)

        docker_provider = docker.Provider(
            "this",
            registry_auth=[
                docker.ProviderRegistryAuthArgs(
                    address=os.environ.get("CI_REGISTRY"),
                    username=os.environ.get("CI_REGISTRY_USER"),
                    password=os.environ.get("CI_REGISTRY_PASSWORD"),
                ),
            ],
        )

        default_resource_options = ResourceOptions(parent=self, providers={"docker": docker_provider})

        CustomComponentTwo(
            name="this",
            opts=default_resource_options,
        )


CustomComponentOne(name="test-gitlab-docker")

Pulumi config:

---
name: test-pulumi-docker-provider

runtime:
  name: python

options:
  refresh: always

disable-default-providers: ['*']

I was able to reproduce the issue with the following steps:

export CI_REGISTRY_USER="[...]" - this is my Gitlab username
export CI_REGISTRY_PASSWORD="[...]" - this is a Gitlab access token attached to my user with access to the registry
pulumi login file://
pulumi up
run pulumi up again just to ensure things still work properly
Browse to the Gitlab access token page and revoke the token exported in step 2
Create a new access token with the same access
export CI_REGISTRY_PASSWORD="[...]" - export token created in step 7
pulumi preview --diff

pulumi preview --diff                                   
Please choose a stack, or create a new one: default
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):  
Enter your passphrase to unlock config/secrets
Previewing update (default):
    ~ custom:component:one: (refresh)
        [urn=urn:pulumi:default::test-pulumi-docker-provider::custom:component:one::test-gitlab-docker]
~ pulumi:pulumi:Stack: (refresh)
    [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:pulumi:Stack::test-pulumi-docker-provider-default]
        ~ custom:component:two: (refresh)
            [urn=urn:pulumi:default::test-pulumi-docker-provider::custom:component:one$custom:component:two::this]
    ~ pulumi:providers:docker: (refresh)
        [id=9a5ecf06-6282-47f3-916d-ec3161e854f6]
        [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:providers:docker::this]
            ~ docker:index/remoteImage:RemoteImage: (refresh)
                [id=sha256:c58b1e54efd8db1b900a6fb5ec07236c1b326c1cd463f45bdd79709f52a7377e[...]]
                [urn=urn:pulumi:default::test-pulumi-docker-provider::custom:component:one$custom:component:two$docker:index/remoteImage:RemoteImage::this]
                [provider=urn:pulumi:default::test-pulumi-docker-provider::pulumi:providers:docker::this::9a5ecf06-6282-47f3-916d-ec3161e854f6]
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:pulumi:Stack::test-pulumi-docker-provider-default]
error: Program failed with an unhandled exception:
Traceback (most recent call last):
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/__main__.py", line 52, in <module>
    CustomComponentOne(name="test-gitlab-docker")
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/__main__.py", line 46, in __init__
    CustomComponentTwo(
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/__main__.py", line 20, in __init__
    gitlab_image = docker.get_registry_image(name="[...]", opts=default_invoke_options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/.venv/lib/python3.12/site-packages/pulumi_docker/get_registry_image.py", line 114, in get_registry_image
    __ret__ = pulumi.runtime.invoke('docker:index/getRegistryImage:getRegistryImage', __args__, opts=opts, typ=GetRegistryImageResult).value
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/.venv/lib/python3.12/site-packages/pulumi/runtime/invoke.py", line 127, in invoke
    return _sync_await(awaitableInvokeResult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/.venv/lib/python3.12/site-packages/pulumi/runtime/sync_await.py", line 66, in _sync_await
    return fut.result()
           ^^^^^^^^^^^^
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/.venv/lib/python3.12/site-packages/pulumi/runtime/invoke.py", line 357, in wait_for_fut
    return await asyncio.ensure_future(do_rpc())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fbaumann/projects/poc/pulumi-docker-provider/.venv/lib/python3.12/site-packages/pulumi/runtime/invoke.py", line 353, in do_rpc
    raise error
Exception: invoke of docker:index/getRegistryImage:getRegistryImage failed: invocation of docker:index/getRegistryImage:getRegistryImage returned an error: invoking docker:index/getRegistryImage:getRegistryImage: 1 error occurred:
        * Got error when attempting to fetch image version [...] from registry: Got bad response from registry: 401 Unauthorized

then I downgraded the pulumi-docker provider to < 4.6.0 and the preview works again:

pulumi preview --diff                     
Please choose a stack, or create a new one: default
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):  
Enter your passphrase to unlock config/secrets
Previewing update (default):
~ custom:component:one: (refresh)
    [urn=urn:pulumi:default::test-pulumi-docker-provider::custom:component:one::test-gitlab-docker]
    ~ custom:component:two: (refresh)
        [urn=urn:pulumi:default::test-pulumi-docker-provider::custom:component:one$custom:component:two::this]
~ pulumi:pulumi:Stack: (refresh)
    [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:pulumi:Stack::test-pulumi-docker-provider-default]
    ~ pulumi:providers:docker: (refresh)
        [id=9a5ecf06-6282-47f3-916d-ec3161e854f6]
        [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:providers:docker::this]
            ~ docker:index/remoteImage:RemoteImage: (refresh)
                [id=sha256:c58b1e54efd8db1b900a6fb5ec07236c1b326c1cd463f45bdd79709f52a7377e[...]]
                [urn=urn:pulumi:default::test-pulumi-docker-provider::custom:component:one$custom:component:two$docker:index/remoteImage:RemoteImage::this]
                [provider=urn:pulumi:default::test-pulumi-docker-provider::pulumi:providers:docker::this::9a5ecf06-6282-47f3-916d-ec3161e854f6]
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:pulumi:Stack::test-pulumi-docker-provider-default]
    ~ pulumi:providers:docker: (update)
        [id=9a5ecf06-6282-47f3-916d-ec3161e854f6]
        [urn=urn:pulumi:default::test-pulumi-docker-provider::pulumi:providers:docker::this]
        registryAuth: [secret]
      ~ version     : "4.6.0" => "4.5.8"
Resources:              
    ~ 1 to update
    4 unchanged

flostadler · 2025-01-17T11:55:11Z

Thanks @asvinours, I was able to reproduce it with your example!

The problem arises under the following circumstances:

No default provider used for pulumi-docker (e.g. disable-default-providers: ['*'])
Running preview with refresh (e.g. pulumi preview --refresh or stack config option refresh: always)
Using changed credentials (i.e. ones that are different then the current creds in state)

What triggers this behavior under the hood is that pulumi creates a provider during the refresh phase using the credentials from state (Configure with values from state). During the following preview, it'll diff the provider (DiffConfig) and if there's no diff, it'll not issue another Configure call using the inputs, instead it'll reuse the existing provider from the refresh phase. But this one has the old/wrong credentials from state.

This is definitely a regression introduced in #1327, I'm sorry!

I'll pick this up right away. For other users running into this issue, please either use an earlier version (e.g. v4.5.8) or run refresh separately from pulumi up/pulumi preview as a workaround in the meantime.

…" (#1341) This reverts commit 8d722c6. Revert in order to fix preview with refresh regression (#1339) introduced in #1327 Fixes #1339

flostadler · 2025-01-17T15:20:27Z

We've reverted the offending change. The release should go out shortly

pulumi-bot · 2025-01-17T15:44:42Z

This issue has been addressed in PR #1341 and shipped in release v4.6.1.

flostadler · 2025-01-17T15:46:44Z

@asvinours the fix has been shipped. Could you verify whether everything works as expected again?

asvinours · 2025-01-17T16:58:36Z

I upgraded to 4.6.1 and ran the gitlab pipeline couple of times and it is successful now. Thank you for the quick fix 🙏

flostadler added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Jan 16, 2025

flostadler self-assigned this Jan 16, 2025

flostadler added the needs-repro Needs repro steps before it can be triaged or fixed label Jan 16, 2025

flostadler mentioned this issue Jan 16, 2025

Ignore username/password changes in registryAuth config #1327

Merged

flostadler added awaiting-feedback Blocked on input from the author and removed needs-triage Needs attention from the triage team labels Jan 16, 2025

pulumi-bot added needs-triage Needs attention from the triage team and removed awaiting-feedback Blocked on input from the author labels Jan 16, 2025

flostadler added p1 A bug severe enough to be the next item assigned to an engineer impact/regression Something that used to work, but is now broken and removed needs-triage Needs attention from the triage team needs-repro Needs repro steps before it can be triaged or fixed labels Jan 17, 2025

flostadler mentioned this issue Jan 17, 2025

Revert "Ignore username/password changes in registryAuth config (#1327)" #1341

Merged

flostadler closed this as completed in #1341 Jan 17, 2025

flostadler added a commit that referenced this issue Jan 17, 2025

Revert "Ignore username/password changes in registryAuth config (#1327)…

8a0c1a4

…" (#1341) This reverts commit 8d722c6. Revert in order to fix preview with refresh regression (#1339) introduced in #1327 Fixes #1339

pulumi-bot added the resolution/fixed This issue was fixed label Jan 17, 2025

flostadler reopened this Jan 17, 2025

flostadler added the awaiting-feedback Blocked on input from the author label Jan 17, 2025

pulumi-bot added needs-triage Needs attention from the triage team and removed awaiting-feedback Blocked on input from the author labels Jan 17, 2025

flostadler closed this as completed Jan 21, 2025

flostadler removed the needs-triage Needs attention from the triage team label Jan 21, 2025

flostadler mentioned this issue Jan 22, 2025

Registry auth shows changes on every run #952

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preview with refresh failing with Authentication errors #1339

Preview with refresh failing with Authentication errors #1339

flostadler commented Jan 16, 2025

flostadler commented Jan 16, 2025

flostadler commented Jan 16, 2025

asvinours commented Jan 16, 2025 •

edited

Loading

flostadler commented Jan 17, 2025

flostadler commented Jan 17, 2025

pulumi-bot commented Jan 17, 2025

flostadler commented Jan 17, 2025

asvinours commented Jan 17, 2025

Preview with refresh failing with Authentication errors #1339

Preview with refresh failing with Authentication errors #1339

Comments

flostadler commented Jan 16, 2025

Describe what happened

Sample program

Log output

Affected Resource(s)

Output of pulumi about

Additional context

Contributing

flostadler commented Jan 16, 2025

flostadler commented Jan 16, 2025

asvinours commented Jan 16, 2025 • edited Loading

flostadler commented Jan 17, 2025

flostadler commented Jan 17, 2025

pulumi-bot commented Jan 17, 2025

flostadler commented Jan 17, 2025

asvinours commented Jan 17, 2025

Output of `pulumi about`

asvinours commented Jan 16, 2025 •

edited

Loading