Potential Latency Issues and Local Affinity in Public MetalnetLB

[hardikdr]

Summary

In the current implementation of the metalnet, which serves as a bridge between Virtlet and DP-service, there is a potential issue related to latency and load balancing. The Public MetalnetLB currently acquires an exhaustive list of all backend MetalnetNICs targeted by the load balancer, which could be spread across an entire VNI, within a single cluster, or across multiple AZs/clusters.

Theoretically, a single MetalnetLB copy could be created on each host machine where at least one targeted MetalnetNIC exists. This approach would allow for the installation of multiple public MetalnetLB copies within the same VNI, enabling HA mode for load balancers.

However, this setup could lead to high latency for packets in certain scenarios:

1. When a VNI is spread across multiple AZs/clusters, traffic from the MetalnetLB in AZ1 could regularly be forwarded to the MetalnetLB in AZ3, resulting in high latency.

2. Even within the same cluster, a MetalnetLB may decide to forward traffic to a remote host, despite the fact that the traffic could be served by the MetalNIC located on the local host. While this could optimize latency, it is a more advanced optimization and less of a critical concern compared to the first scenario.

Motivation

Discussion on whether there should be an affinity towards a local-AZ or local-host when choosing the target MetalnetNIC to forward traffic to. It could potentially reduce latency and optimize traffic flow.

[hardikdr]

cc @FlorinPeter @gehoern @marekbendzala