AssumeRoleWithWebIdentity issues cross partition

[wattie-canva]

What happened?

When trying to create a resource in a different partition with AssumeRoleWithWebIdentity the following error occurs:

"error": "failed to create the Instance resource: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}"

How can we reproduce it?

There is an EKS cluster in the global partition
EKS cluster in another partition such as aws-cn
Create OIDC IDP in aws-cn which trusts the global clusters OIDC

Try to create any resource with:

spec:
  assumeRoleWithWebIdentity:
    roleARN: "arn:aws:iam::${TARGET_AWS_ACCOUNT_ID}:role/${IAM_ROLE_NAME}"

What environment did it happen in?

Crossplane version:
index.docker.io/crossplanecontrib/provider-aws:v0.37.1

[wattie-canva]

@nabuskey

[nabuskey]

@wattie-canva

I've pushed a container image with a fix for this here: public.ecr.aws/m8u6z8z4/manabu-test:web-token-partition-v1

Please test it out and see if it fixes the issue for you. Source code is available here: https://github.com/nabuskey/provider-aws/tree/bugfix/web-token-partition. If this works, I will make a PR based on this.
You should be able to specify the image in your controller config.

[wattie-canva]

Hey @nabuskey I updated the image version but the provider doesn't come up healthy.
Do you know what might cause this or how to fix it?

# k describe providers
...
Events:
  Type     Reason                  Age                    From                                 Message
  ----     ------                  ----                   ----                                 -------
  Warning  InstallPackageRevision  11m (x7 over 12m)      packages/provider.pkg.crossplane.io  current package revision health is unknown
  Warning  InstallPackageRevision  11m (x7 over 12m)      packages/provider.pkg.crossplane.io  cannot apply package revision: cannot create object: the server could not find the requested resource (post providerrevisions.pkg.crossplane.io)
  Warning  InstallPackageRevision  9m8s                   packages/provider.pkg.crossplane.io  cannot apply package revision: cannot patch object: Operation cannot be fulfilled on providerrevisions.pkg.crossplane.io "provider-aws-f619b9b5a5d9": the object has been modified; please apply your changes to the latest version and try again
  Warning  InstallPackageRevision  9m7s (x4 over 9m8s)    packages/provider.pkg.crossplane.io  current package revision health is unknown
  Warning  InstallPackageRevision  8m58s (x2 over 8m58s)  packages/provider.pkg.crossplane.io  current package revision is unhealthy

I'm using crossplane v1.11.1-stable and crossplane-provider-aws public.ecr.aws/m8u6z8z4/manabu-test:web-token-partition-v1

Do you know what might cause this or how to fix it?

[nabuskey]

Can you get events from the provider revision object? Also can you post your controller config?

[wattie-canva]

Providerrevision events:

Events:
  Type     Reason             Age                     From                                         Message
  ----     ------             ----                    ----                                         -------
  Normal   ApplyClusterRoles  33m (x4 over 98m)       rbac/providerrevision.pkg.crossplane.io      Applied RBAC ClusterRoles
  Warning  ParsePackage       4m18s (x1029 over 20h)  packages/providerrevision.pkg.crossplane.io  cannot initialize parser backend: failed to open package stream file: EOF

Controllerconfig with some specifics such as IRSA removed:
Happy to provide full dump privately if that helps.

kind: ControllerConfig
metadata:
  annotations:
    eks.amazonaws.com/role-arn: 'arn:aws:iam::<ID>:role/...crossplane-provider-aws'
  creationTimestamp: '2023-06-25T23:35:04Z'
  generation: 1
  labels:
    app.kubernetes.io/managed-by: argocd
    app.kubernetes.io/name: crossplane-provider-aws
    app.kubernetes.io/version: 1.0.0
    argocd.argoproj.io/instance: crossplane-provider-aws.cluster-name
  name: provider-aws-config
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                pkg.crossplane.io/provider: provider-aws
            topologyKey: topology.kubernetes.io/zone
          weight: 50
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                pkg.crossplane.io/provider: provider-aws
            topologyKey: kubernetes.io/hostname
          weight: 50
  args:
    - '--debug'
  env:
    - name: LEADER_ELECTION
      value: 'true'
  replicas: 2

[nabuskey]

The image I linked is for the controller only so you need to specify it in the spec.image field. e.g.

kind: ControllerConfig
metadata:
  name: provider-aws-config
spec:
  image: public.ecr.aws/m8u6z8z4/manabu-test:web-token-partition-v1

[wattie-canva]

Got it running with that image now but the pods provider-aws- crashloopbackoff with the logs

exec /usr/local/bin/crossplane-aws-provider: exec format error

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.