error: failed to load language plugin nodejs: could not read plugin [/usr/bin/pulumi-language-nodejs]

[cdwheatley]

What happened?

I am encountering the following error when I am trying to bring up a Nodejs stack with the Pulumi Kubernetes Operator. Any help would be greatly appreciated.

I have verified the plugin appears to be available in the pod at /usr/bin/pulumi-language-nodejs.

I'm unsure where to go next to troubleshoot this issue.

I0821 20:53:54.750044       7 leaderelection.go:243] attempting to acquire leader lease default/pulumi-kubernetes-operator-lock...
I0821 20:54:12.555983       7 leaderelection.go:253] successfully acquired lease default/pulumi-kubernetes-operator-lock
{"level":"error","ts":"2024-08-21T20:55:00.297Z","logger":"controller_stack","msg":"Failed to update Stack","Request.Namespace":"default","Request.Name":"curl-pod-stack-chrisdev-d8d9eb01","Stack.Name":"organization/curl-pod/chrisdev","error":"failed to run update: exit status 255\ncode: 255\nstdout: Updating (chrisdev):\n    pulumi:pulumi:Stack curl-pod-chrisdev  flag provided but not defined: -packagemanager\n    pulumi:pulumi:Stack curl-pod-chrisdev  Usage of /usr/bin/pulumi-language-nodejs:\n    pulumi:pulumi:Stack curl-pod-chrisdev    -alsologtostderr\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tlog to standard error as well as files\n    pulumi:pulumi:Stack curl-pod-chrisdev    -dirs value\n    pulumi:pulumi:Stack curl-pod-chrisdev      \toptional list of regexes to use to select integration tests to run\n    pulumi:pulumi:Stack curl-pod-chrisdev    -list-dirs\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tlist available integration tests without running them\n    pulumi:pulumi:Stack curl-pod-chrisdev    -log_backtrace_at value\n    pulumi:pulumi:Stack curl-pod-chrisdev      \twhen logging hits line file:N, emit a stack trace\n    pulumi:pulumi:Stack curl-pod-chrisdev    -log_dir string\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tIf non-empty, write log files in this directory\n    pulumi:pulumi:Stack curl-pod-chrisdev    -log_link string\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tIf non-empty, add symbolic links in this directory to the log files\n    pulumi:pulumi:Stack curl-pod-chrisdev    -logbuflevel int\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tBuffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; ...). Has limited applicability on non-prod platforms.\n    pulumi:pulumi:Stack curl-pod-chrisdev    -logtostderr\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tlog to standard error instead of files\n    pulumi:pulumi:Stack curl-pod-chrisdev    -nodeargs string\n    pulumi:pulumi:Stack curl-pod-chrisdev      \t[obsolete] Arguments for the Node process\n    pulumi:pulumi:Stack curl-pod-chrisdev    -root string\n    pulumi:pulumi:Stack curl-pod-chrisdev      \t[obsolete] Project root path to use\n    pulumi:pulumi:Stack curl-pod-chrisdev    -sdk.no-checks\n    pulumi:pulumi:Stack curl-pod-chrisdev      \twhen set, skips all post-SDK-generation checks\n    pulumi:pulumi:Stack curl-pod-chrisdev    -stderrthreshold value\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tlogs at or above this threshold go to stderr (default 2)\n    pulumi:pulumi:Stack curl-pod-chrisdev    -tracing string\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tEmit tracing to a Zipkin-compatible tracing endpoint\n    pulumi:pulumi:Stack curl-pod-chrisdev    -tsconfig string\n    pulumi:pulumi:Stack curl-pod-chrisdev      \t[obsolete] Path to tsconfig.json to use\n    pulumi:pulumi:Stack curl-pod-chrisdev    -typescript\n    pulumi:pulumi:Stack curl-pod-chrisdev      \t[obsolete] Use ts-node at runtime to support typescript source natively (default true)\n    pulumi:pulumi:Stack curl-pod-chrisdev    -v value\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tlog level for V logs\n    pulumi:pulumi:Stack curl-pod-chrisdev    -vmodule value\n    pulumi:pulumi:Stack curl-pod-chrisdev      \tcomma-separated list of pattern=N settings for file-filtered logging\n    pulumi:pulumi:Stack curl-pod-chrisdev  36 messages\nDiagnostics:\n  pulumi:pulumi:Stack (curl-pod-chrisdev):\n    flag provided but not defined: -packagemanager\n    Usage of /usr/bin/pulumi-language-nodejs:\n      -alsologtostderr\n        \tlog to standard error as well as files\n      -dirs value\n        \toptional list of regexes to use to select integration tests to run\n      -list-dirs\n        \tlist available integration tests without running them\n      -log_backtrace_at value\n        \twhen logging hits line file:N, emit a stack trace\n      -log_dir string\n        \tIf non-empty, write log files in this directory\n      -log_link string\n        \tIf non-empty, add symbolic links in this directory to the log files\n      -logbuflevel int\n        \tBuffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; ...). Has limited applicability on non-prod platforms.\n      -logtostderr\n        \tlog to standard error instead of files\n      -nodeargs string\n        \t[obsolete] Arguments for the Node process\n      -root string\n        \t[obsolete] Project root path to use\n      -sdk.no-checks\n        \twhen set, skips all post-SDK-generation checks\n      -stderrthreshold value\n        \tlogs at or above this threshold go to stderr (default 2)\n      -tracing string\n        \tEmit tracing to a Zipkin-compatible tracing endpoint\n      -tsconfig string\n        \t[obsolete] Path to tsconfig.json to use\n      -typescript\n        \t[obsolete] Use ts-node at runtime to support typescript source natively (default true)\n      -v value\n        \tlog level for V logs\n      -vmodule value\n        \tcomma-separated list of pattern=N settings for file-filtered logging\n\n\nstderr: error: failed to load language plugin nodejs: could not read plugin [/usr/bin/pulumi-language-nodejs] stdout: EOF\n\n","stacktrace":"github.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:892\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214"}

Example

See above

Output of pulumi about

Pulumi about from inside the Pulumi Kubernetes Pod

pulumi-kubernetes-operator@pulumi-kubernetes-operator-default-chrisdev-90b20901-778fcsfsrp:/$ pulumi about
CLI          
Version      3.109.0
Go Version   go1.22.0
Go Compiler  gc

Host     
OS       debian
Version  11.9
Arch     x86_64

Backend        
Name           pulumi.com
URL            https://app.pulumi.com
User           Unknown
Organizations  
Token type     personal

Pulumi locates its logs in /tmp by default
warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from /). If you have not created a project yet, use `pulumi new` to do so: no project file found
warning: Failed to get information about the current stack: no Pulumi.yaml project file found (searching upwards from /). If you have not created a project yet, use `pulumi new` to do so: no project file found

Here is the Pulumi Kubernetes Operator project's Pulumi about:

pulumi about
2024/08/21 16:06:05 Enabling CSM
CLI          
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws         6.47.0
resource  aws         6.36.0
resource  command     1.0.1
resource  kubernetes  4.15.0
language  nodejs      unknown

Host     
OS       darwin
Version  14.2.1
Arch     x86_64

This project is written in nodejs: executable='/usr/local/bin/node' version='v20.12.0'

Current Stack: organization/deploy-pulumi-operator/chrisdev

TYPE                                                         URN
pulumi:pulumi:Stack                                          urn:pulumi:chrisdev::deploy-pulumi-operator::pulumi:pulumi:Stack::deploy-pulumi-operator-chrisdev
kubernetes:yaml:ConfigFile                                   urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:yaml:ConfigFile::stackcrd
pulumi:providers:pulumi                                      urn:pulumi:chrisdev::deploy-pulumi-operator::pulumi:providers:pulumi::default
kubernetes:yaml:ConfigFile                                   urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:yaml:ConfigFile::programcrd
pulumi:pulumi:StackReference                                 urn:pulumi:chrisdev::deploy-pulumi-operator::pulumi:pulumi:StackReference::organization/service-account/dev
pulumi:providers:command                                     urn:pulumi:chrisdev::deploy-pulumi-operator::pulumi:providers:command::default_1_0_1
command:local:Command                                        urn:pulumi:chrisdev::deploy-pulumi-operator::command:local:Command::gitBranch-chrisdev
pulumi:providers:kubernetes                                  urn:pulumi:chrisdev::deploy-pulumi-operator::pulumi:providers:kubernetes::default_4_15_0
kubernetes:rbac.authorization.k8s.io/v1:Role                 urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:rbac.authorization.k8s.io/v1:Role::operator-role-default-chrisdev
kubernetes:core/v1:ServiceAccount                            urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:core/v1:ServiceAccount::operator-service-account-default-chrisdev
pulumi:providers:aws                                         urn:pulumi:chrisdev::deploy-pulumi-operator::pulumi:providers:aws::default_6_47_0
kubernetes:core/v1:Secret                                    urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:core/v1:Secret::gitlab-credentials-chrisdev
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:yaml:ConfigFile$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::programs.pulumi.com
kubernetes:core/v1:Secret                                    urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:core/v1:Secret::aws-credentials-chrisdev
kubernetes:rbac.authorization.k8s.io/v1:RoleBinding          urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::operator-role-binding-default-chrisdev
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:yaml:ConfigFile$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::stacks.pulumi.com
kubernetes:apps/v1:Deployment                                urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:apps/v1:Deployment::pulumi-kubernetes-operator-default-chrisdev
kubernetes:pulumi.com/v1:Stack                               urn:pulumi:chrisdev::deploy-pulumi-operator::kubernetes:pulumi.com/v1:Stack::curl-pod-stack-chrisdev


Found no pending operations associated with chrisdev

Backend        
Name           TCM-xxxxxx
URL            s3://xxxxxxxxxxx-pulumi-state?region=us-east-1&awssdk=v2&profile=xxxxxxxxx-dev
User           xxxxxx
Organizations  
Token type     personal

Dependencies:
NAME                VERSION
pulumi-util         
@pulumi/aws         6.47.0
@pulumi/command     1.0.1
@pulumi/kubernetes  4.15.0
@pulumi/pulumi      3.127.0
@types/node         18.19.42

Pulumi locates its logs in /var/folders/v5/z5rcjv5n5lgfmplydjgnm9sxcqwq1k/T/ by default


### Additional context

_No response_

### Contributing

Vote on this issue by adding a 👍 reaction. 
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already). 

[rquitales]

I noticed this line while reviewing the error log you provided: flag provided but not defined: -packagemanager. This flag was introduced in v3.121.0 of the Pulumi CLI (reference). However, the pulumi about command indicates that your Operator pod is running v3.109.0 of the CLI. We'll need to cut a release of the Operator using a newer version of the Pulumi CLI to support this option.

As a temporary solution, you can remove the packagemanager option from your Pulumi.yaml config file for now.

[rquitales]

This should now be resolved with v1.16.0 of the Pulumi Kubernetes Operator.

[cdwheatley]

Thanks very much @rquitales it is greatly appreciated!