Apparently no support for docker runtime

[tardich]

Using this network-attachment in a docker container runtime environment:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
  name: ipvlan-bond0-213-0
spec:
  config: '{
             "cniVersion": "0.3.0",
             "name": "myipvlan",
             "plugins": [
               {
                 "type": "ipvlan",
                 "master": "bond0.213",
                 "mode": "l2",
                 "ipam": {
                   "type": "static",
                    "addresses": [
                      {
                        "address": "192.168.213.252/24"
                      }
                    ]
                 }
               },
               {
                 "type": "route-override",
                 "flushroutes": true,
                 "delroutes": [
                   {
                     "dst": "10.42.0.0/16"
                   },
                   {
                     "dst": "10.43.0.0/16"
                   }
                 ],
                 "addroutes": [
                   {
                     "dst": "10.42.0.0/16",
                     "gw": "169.254.1.1"
                   },
                   {
                     "dst": "10.43.0.0/16",
                     "gw": "169.254.1.1"
                   }
                 ]
               }
             ]
           }'

I'm getting the net1 address added as expected, but no routes are being added. I'm changing the default gateway from the deployment annotation as per multus documentation (which works) but then I need to add 10.43.0.0/16 and 10.42.0.0/16 networks to point to 169.254.1.1 (which used to be the default gateway without multus) to allow the pod to reach internal kubernetes stuff (including DNS, which is at 10.43.0.10).

Routes are not being added. My runtime is docker (kubernetes deployment is rke-based).

[tardich]

Finally, my assumption was wrong. route-override IS working on docker. While looking at kubelet logs, I realized that the routes were sent to the pod, but got refused to apply because of network unreacheable, which is, in fact, true.

What would be needed maybe rather be able to point to a device (eth0) rather than IP

[nsaeris]

I am running into the same issue i.e. using this with docker. The kubelet log shows "failed to add route...network is unreachable". It seems the plugin attempts to add route even before the interface is attached and has IP address. Any solution or workaround?

[s1061123]

As far as I know of, when you use plugins chain of cni and first CNI plugin assigned IP address, second and later plugins are invoked after IP address assignment. Could you please show your net-attach-def?

[nsaeris]

Here is my net-attach-def. I am using dhcp ipam plugin which takes couple of seconds to get the IP address. Are these plugins invoked sequentially or in parallel? I am not able to find enough documentation on understand the order of execution.
`
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: test-macvlan-conf
namespace: test
spec:
config: '{
"cniVersion": "0.3.1",
"name": "test-macvlan-conf",
"plugins": [
{
"type": "macvlan",
"master": "ens192.121",
"mode": "bridge",
"ipam": {
"type": "dhcp"
}
},
{
"type": "route-override",
"flushroutes": true,
"addroutes": [
{
"dst": "172.16.0.0/12",
"gw": "x.y.z.1"
},
{
"dst": "192.168.0.0/16",
"gw": "x.y.z.1"
},
{
"dst": "0.0.0.0/0",
"gw": "a.b.c.d"
}
]
}
]
}'

`

[s1061123]

Q.
Are the dhcp assigned ip address, "x.y.z.1", and "a.b.c.d" in same subnet, or different? As far as I know, gateway ip address should be in same subnet of interface address.

I recommend you to configure your configuration in container, by your hand.
Following container image has ip route tools so you can run this image with privilege and try your configuration is valid or not.
https://hub.docker.com/r/centos/tools/

[nsaeris]

@s1061123
You were right. I referred the wrong interface in the config. The error message in the logs threw me off. I can confirm that the plugin works after updating the interface ("master") in the config.

Thank you also for closing the issue on multus-cni #602