Calico Operator should support running different dataplanes on different nodes in the same Kubernetes cluster

[VadimEisenberg]

Expected Behavior

When heterogeneous worker nodes are used in a Kubernetes cluster, it can make sense to use different dataplanes for different nodes, for example, iptables on some nodes and VPP on other nodes. Some nodes may require a custom dataplane driver (not the one provided by Calico).

Another use case for supporting multiple dataplanes simultaneously is migrating from one dataplane to another - it can be performed by switching nodes one by one to a new dataplane.

From VPP documentation:

The VPP dataplane is entirely compatible with the other Calico dataplanes, meaning you can have a cluster with VPP-enabled nodes along with regular nodes. This makes it possible to migrate a cluster from Linux or eBPF networking to VPP networking.

Current Behavior

Currently, I do not see how I can instruct the operator to install different dataplanes on different nodes. To specify a dataplane, one creates an Installation CR named "default", and specifies in spec.calicoNetwork.linuxDataplane the following values Iptables, BPF or VPP.

From Installation documentation:

At most one instance of this resource is supported. It must be named “default”.

It seems I cannot specify multiple dataplanes in an Installation CR and I cannot create multiple Installation CRs.

Context

I need to run different dataplanes on different nodes.

Your Environment

• Kubernetes: 1.27
• My project: neureality.ai

[VadimEisenberg]

cc @sknat @tmjd

[tmjd]

As I mentioned in this slack thread, I'm not sure that the doc's intention was that a persistent state is available with VPP and another data plane running.

But I do think you might be able to achieve what you are looking for but not with the operator managing both calico-node daemonsets. I think you could specify a nodeSelector in the calicoNodeDaemonSet field for the operator to have the calico-node it deploys only go to specific nodes, then you could deploy a 2nd calico node daemonset manually and target the other nodes. I haven't tried this but I believe it would work.

I have marked this issue as a feature request (kind/enhancement) as I can certainly understand wanting the operator to support a mixed environment like you've suggested.