Issues pulling nginx-plus-ingress:2.3.0

[BrianVinsonF5]

Hello, I am having an issue pulling the nginx-plus-ingress:2.3.0 image from the private registry. Any note on what I am doing wrong?

Followed the steps in https://docs.nginx.com/nginx-ingress-controller/installation/using-the-jwt-token-docker-secret/

Ubuntu K8s 1.24.4 cluster

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
     #annotations:
       #prometheus.io/scrape: "true"F
       #prometheus.io/port: "9113"
       #prometheus.io/scheme: http
    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress:2.3.0
        imagePullPolicy: IfNotPresent
        name: nginx-plus-ingress
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: readiness-port
          containerPort: 8081
        - name: prometheus
          containerPort: 9113
        readinessProbe:
          httpGet:
            path: /nginx-ready
            port: readiness-port
          periodSeconds: 1
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-plus
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
          - -enable-cert-manager
          - -enable-app-protect
         #- -enable-app-protect-dos
         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
         #- -enable-prometheus-metrics
         #- -global-configuration=$(POD_NAMESPACE)/nginx-configuration

apiVersion: v1
data:
  .dockerconfigjson: <Removed>
kind: Secret
metadata:
  creationTimestamp: "2022-08-05T20:58:53Z"
  name: regcred
  namespace: nginx-ingress
  resourceVersion: "650867"
  uid: b8e08a4e-d807-4299-a21b-7012b028ac0b
type: kubernetes.io/dockerconfigjson

Name:                nginx-ingress
Namespace:           nginx-ingress
Labels:              <none>
Annotations:         <none>
Image pull secrets:  regcred
Mountable secrets:   nginx-ingress-token-z89gx
Tokens:              nginx-ingress-token-z89gx
Events:              <none>

[brianehlert]

Hey @BrianVinsonF5 , can you share the error message?

Also, did you try the cURL command to test your credential by listing the images?

[BrianVinsonF5]

When I use just the serviceAccount linked to the Secret, the error contains docker.io. If I place the imagePullSecrets in the deployment spec. I get the same error with the private-registry.nginx.com as the domain name.

Events: Normal Scheduled 56m default-scheduler Successfully assigned nginx-ingress/nginx-ingress-596b7f6945-hjbkj to kub3 Normal Pulling 55m (x4 over 56m) kubelet Pulling image "nginx-plus-ingress:2.3.0" Warning Failed 55m (x4 over 56m) kubelet Failed to pull image "nginx-plus-ingress:2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx-plus-ingress:2.3.0": failed to resolve reference "docker.io/library/nginx-plus-ingress:2.3.0": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 55m (x4 over 56m) kubelet Error: ErrImagePull Warning Failed 54m (x6 over 56m) kubelet Error: ImagePullBackOff Normal BackOff 85s (x246 over 56m) kubelet Back-off pulling image "nginx-plus-ingress:2.3.0"

curl https://private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress/tags/list --key nginx-repo.key --cert nginx-repo.crt | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 382 100 382 0 0 644 0 --:--:-- --:--:-- --:--:-- 644
{
"name": "nginx-ic-nap/nginx-plus-ingress",
"tags": [
"1.12.0-ubi",
"1.12.0",
"1.12.1-ubi",
"1.12.1",
"1.12.2-ubi",
"1.12.2",
"1.12.3-ubi",
"1.12.3",
"1.12.4-ubi",
"1.12.4",
"2.0.0-ubi",
"2.0.0",
"2.0.1-ubi",
"2.0.1",
"2.0.2-ubi",
"2.0.2",
"2.0.3-ubi",
"2.0.3",
"2.1.0-ubi",
"2.1.0",
"2.1.1-ubi",
"2.1.1",
"2.1.2-ubi",
"2.1.2",
"2.2.0-ubi",
"2.2.0",
"2.2.1-ubi",
"2.2.1",
"2.2.2-ubi",
"2.2.2",
"2.3.0-ubi",
"2.3.0"
]
}

I think the CURL looks right

[ciarams87]

It looks to me that the issue is probably the image name you have in the deployment - there is an erroneous v2 which is required for the curl command (because it is querying the docker registry API), but it is not part of the image name. Try changing
- image: private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress:2.3.0 -> - image: private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0. The rest looks OK I think!

[BrianVinsonF5]

Thanks for your reply. Looks perhaps a bit closer 401 is better than a 403!

state:
waiting:
message: 'rpc error: code = Unknown desc = failed to pull and unpack image
"private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0": failed
to resolve reference "private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0":
pulling from host private-registry.nginx.com failed with status code [manifests
2.3.0]: 401 Unauthorized'

Normal Scheduled 58s default-scheduler Successfully assigned nginx-ingress/nginx-ingress-77f8c77f66-57vgc to kub3
Normal BackOff 26s (x2 over 54s) kubelet Back-off pulling image "private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0"
Warning Failed 26s (x2 over 54s) kubelet Error: ImagePullBackOff
Normal Pulling 12s (x3 over 57s) kubelet Pulling image "private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0"
Warning Failed 10s (x3 over 55s) kubelet Failed to pull image "private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0": failed to resolve reference "private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:2.3.0": pulling from host private-registry.nginx.com failed with status code [manifests 2.3.0]: 401 Unauthorized
Warning Failed 10s (x3 over 55s) kubelet Error: ErrImagePull

Something with the format of my token perhaps?

PS C:\Users\vinson\OneDrive - F5, Inc\Code\Kub> kubectl get secret regcred -o json
{
"apiVersion": "v1",
"data": {
".dockerconfigjson":
},
"kind": "Secret",
"metadata": {
"creationTimestamp": "2022-08-05T20:58:53Z",
"name": "regcred",
"namespace": "nginx-ingress",
"resourceVersion": "650867",
"uid": "b8e08a4e-d807-4299-a21b-7012b028ac0b"
},
"type": "kubernetes.io/dockerconfigjson"
}

{"auths":{"private-registry.nginx.com":{"username":"","password":"none","auth":""}}}

[ciarams87]

There might be.

What was the command you used to create your secret? Using this kubectl create secret docker-registry regcred --docker-server=private-registry.nginx.com --docker-username=<JWT Token> --docker-password=none -n nginx-ingress (being sure to enter the JWT Token string copied from MyF5 correctly in the --docker-username field) and adding this as an imagePullSecret to my ServiceAccount works for me.

> kubectl get secret regcred -n nginx-ingress -o json
{
    "apiVersion": "v1",
    "data": {
        ".dockerconfigjson": <REDACTED - JWT TOKEN>
    },
    "kind": "Secret",
    "metadata": {
        "creationTimestamp": "2022-08-22T16:50:54Z",
        "name": "regcred",
        "namespace": "nginx-ingress",
        "resourceVersion": "652",
        "uid": "7161e32e-6c2e-4627-9a44-43e05e06f40e"
    },
    "type": "kubernetes.io/dockerconfigjson"
}

[BrianVinsonF5]

Unsure; what I did wrong. when I deleted the secret and recreated it the issue resolved. Thanks for your help!

[nguyenphu123]

Hello, can I ask how can I customize the image path from private-registry.nginx.com to my personal docker hub private image ?