the controller doesn't update EP after the first run

[dolphyvn]

The controller unlikely to update EP after the first run. I have to restart it so the EP get update.
Anyone have same issue?

[dolphyvn]

The controller actually did the update but its like minutes after. And when scaling up/down. There is a gap when the service EP got revert back to cluster IP.

[wardenlym]

@dolphyvn

i meet the same issue, and by my test it can cost up to ~10min to update secondary endpoints.
it seems k8s deliver update event to the controller after a long lag.
anyone knows the reason?

[wardenlym]

hi, i'm not sure if this PR could solve the issue, but it works fine for me

i guess the origin processNextWorkItem() discard some update events which not get ready because sync() returns errors for them

@dolphyvn i built a image to fix this and hope useful to you:

docker pull wardenlym/k8s-net-attach-def-controller:v0.1.1

[infinitydon]

@wardenlym I just tried using your docker image but the EP addresses are not showing up.

POD:

apiVersion: v1
kind: Pod
metadata:
  name: test-3
  labels:
     app: test
  annotations:
          k8s.v1.cni.cncf.io/networks: '[ { "name": "ipvlan-multus-sub-1", "interface": "net1" },
                                          { "name": "ipvlan-multus-sub-2", "interface": "net2" }
                                        ]'
spec:
  containers:
  - name: test
    command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: praqma/network-multitool

POD status:

Name:         test-3
Namespace:    default
Priority:     0
Node:         ip-172-32-4-151.eu-west-1.compute.internal/172.32.4.151
Start Time:   Wed, 02 Dec 2020 09:14:01 +0000
Labels:       app=test
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "ips": [
                        "172.32.4.186"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/ipvlan-multus-sub-1",
                    "interface": "net1",
                    "ips": [
                        "172.32.5.62"
                    ],
                    "mac": "06:7d:99:29:73:95",
                    "dns": {}
                },{
                    "name": "default/ipvlan-multus-sub-2",
                    "interface": "net2",
                    "ips": [
                        "172.32.6.50"
                    ],
                    "mac": "06:a1:64:bf:c1:2f",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks:
                [ { "name": "ipvlan-multus-sub-1", "interface": "net1" }, { "name": "ipvlan-multus-sub-2", "interface": "net2" } ]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "ips": [
                        "172.32.4.186"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/ipvlan-multus-sub-1",
                    "interface": "net1",
                    "ips": [
                        "172.32.5.62"
                    ],
                    "mac": "06:7d:99:29:73:95",
                    "dns": {}
                },{
                    "name": "default/ipvlan-multus-sub-2",
                    "interface": "net2",
                    "ips": [
                        "172.32.6.50"
                    ],
                    "mac": "06:a1:64:bf:c1:2f",
                    "dns": {}
                }]
              kubernetes.io/psp: eks.privileged

service:

kind: Service
apiVersion: v1
metadata:
  name: test-multus
  annotations:
    k8s.v1.cni.cncf.io/networks: ipvlan-multus-sub-1
spec:
  clusterIP: None
  selector:
    app: test
  ports:
  - protocol: TCP
    port: 80

service status:

Name:              test-multus
Namespace:         default
Labels:            <none>
Annotations:       k8s.v1.cni.cncf.io/networks: ipvlan-multus-sub-1
Selector:          app=test
Type:              ClusterIP
IP:                None
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         <none>
Session Affinity:  None
Events:
  Type    Reason                                      Age                    From                           Message
  ----    ------                                      ----                   ----                           -------
  Normal  Updated to use network ipvlan-multus-sub-1  5m22s (x2 over 5m22s)  k8s-net-attach-def-controller  Endpoints update successful

Any idea why the EP IP is not showing?