How to disable proxy when pulling image in a Pod

[zzxwill]

I found Pods failed due to imagePullError.

Events:
  Type     Reason          Age                From                         Message
  ----     ------          ----               ----                         -------
  Normal   Scheduled       <unknown>                                       Successfully assigned cert-manager/cert-manager-7dd5854bb4-lx59h to crc-gm7cm-master-0
  Normal   AddedInterface  23s                multus                       Add eth0 [10.217.0.49/23]
  Normal   Pulling         13s                kubelet, crc-gm7cm-master-0  Pulling image "quay.io/jetstack/cert-manager-controller:v1.3.1"
  Warning  Failed          13s                kubelet, crc-gm7cm-master-0  Failed to pull image "quay.io/jetstack/cert-manager-controller:v1.3.1": rpc error: code = Unknown desc = error pinging docker registry quay.io: Get "https://quay.io/v2/": proxyconnect tcp: dial tcp 127.0.0.1:7890: connect: connection refused
  Warning  Failed          13s                kubelet, crc-gm7cm-master-0  Error: ErrImagePull
  Normal   BackOff         11s (x2 over 12s)  kubelet, crc-gm7cm-master-0  Back-off pulling image "quay.io/jetstack/cert-manager-controller:v1.3.1"
  Warning  Failed          11s (x2 over 12s)  kubelet, crc-gm7cm-master-0  Error: ImagePullBackOff

I'd like to disable proxy using in CRC.

• crc config proxy isn't configured.

➜  /Users/zhouzhengxi/.crc crc config view
- autostart-tray                        : false
- consent-telemetry                     : yes

• My local proxy 127.0.0.1 isn't on system level.

[anjannath]

You might have the http_proxy environment variable set, which crc picks up and configures in the cluster.

[zzxwill]

launchctl getenv HTTP_PROXY

➜  /Users/zhouzhengxi launchctl getenv http_proxy
➜  /Users/zhouzhengxi launchctl getenv HTTP_PROXY

I use the proxy in Chrome. If I need use in cli, I export it by export https_proxy=http://127.0.0.1:7890 http_proxy=http://127.0.0.1:7890 all_proxy=socks5://127.0.0.1:7890.

[zzxwill]

For now as a workaround if you have the proxy config in cluster (can check this by oc get proxy/cluster -oyaml) you can remove it by following https://docs.openshift.com/container-platform/4.7/networking/enable-cluster-wide-proxy.html#nw-proxy-remove_config-cluster-wide-proxy

Okay, let me try it later as currently my cluster keeps becoming unreachable.

[zzxwill]

oc get proxy/cluster -oyaml

➜  /Users/zhouzhengxi oc get proxy/cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  creationTimestamp: "2021-04-29T07:14:07Z"
  generation: 1
  managedFields:
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:trustedCA:
          .: {}
          f:name: {}
      f:status: {}
    manager: cluster-bootstrap
    operation: Update
    time: "2021-04-29T07:14:07Z"
  name: cluster
  resourceVersion: "364"
  selfLink: /apis/config.openshift.io/v1/proxies/cluster
  uid: 86082f5e-9abd-42ab-8923-e6af54df2ea1
spec:
  trustedCA:
    name: ""
status: {}

[anjannath]

Ahh, so no proxy configured in the cluster.. you can try to see if the node has a proxy config, still don't know how it could've landed there.

# Get the name of the node
$ oc get nodes
NAME                 STATUS   ROLES           AGE    VERSION
crc-m89r2-master-0   Ready    master,worker   6d3h   v1.20.0+75370d3

# start a debug pod
$ oc debug node/crc-m89r2-master-0
$ chroot /host

# get the env variables set
$ env

/cc @praveenkumar have a look

[guillaumerose]

Can you look at the content of /etc/systemd/system/crio.service.d/10-default-env.conf and /etc/systemd/system/kubelet.service.d/10-default-env.conf ?
This files are written before kubelet is started. So if a proxy was configured before launch, it will show up here.

[praveenkumar]

This looks like similar to kubernetes-sigs/kind#1459 one, does you have direct internet access on that machine without any proxy or all the traffic goes through this proxy?

@guillaumerose Can it be something to do with vsock?