How to configure Keepalived as L4 load balancer via Direct Routing (DR) method?

[moaali]

Trying to achieve L4 load balancing via Keepalived in front of HAProxy that will act as L7 load balancer. Both Keepalived and HAProxy are on separate machines. I managed to get everything in the below image working but when I try to send a request to the public virtual IP (i.e. 115.101.1.17), the connection always times out.

As per my understanding from the documentation that states:

In order for the real servers to directly respond to the public users’ requests, each real server must use the VIP as its source address when sending replies.

I tried to reset the source IP using SNAT on the HAProxy machine but still the same thing happens.

Rules used for iptables on real servers (HAProxy machines)

iptables \
  -A POSTROUTING \
  -t nat \
  -p tcp \
  --dport 80 \
  -j SNAT \
  --to-source 115.101.1.17

keepalived.conf

vrrp_instance VI_1 {
  state       MASTER     # [1]
  interface   eth0       # [2]
  advert_int  1          # [3]
  priority    100        # [4]

  virtual_router_id  92  # [5]

  # Authentication for VRRP messages
  authentication {
    auth_type  PASS
    auth_pass  pass123
  }

  virtual_ipaddress {
    115.101.1.17 dev eth0  # [6]
  }
}

virtual_server 115.101.1.17 80 {
  lb_algo   rr             # [1]
  lb_kind   DR             # [2]
  protocol  TCP

  delay_loop           10  # [3]
  persistence_timeout  60  # [4]

  # Backend Server (HAProxy LB-01)
  real_server 10.0.1.2 80 {
    weight 100

    TCP_CHECK {
      connect_timeout 5
      nb_get_retry 3
      delay_before_retry 2
    }
  }

  # Backend Server (HAProxy LB-02)
  real_server 10.0.1.3 80 {
    weight 100

    TCP_CHECK {
      connect_timeout 5
      nb_get_retry 3
      delay_before_retry 2
    }
  }
}

sysctl.conf

net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1

Any help will be appreciated.

[nser77]

Hi, in your case, connection time out is an expected behavior; the main reason is that keepalived and haproxy servers are on different subnets (10.0.0.0/24 and 10.0.1.0/24) and loadbalancig direct routing works at MAC Address level; based on your scenario, haproxy servers will never recive traffic beacuse keepalived is not able to learn the (real) MAC Address of those real servers - they are separated by a L3 device.

We can confirm that from your virtual_server configuration block:

[...]

  real_server 10.0.1.2 80 {

[...]

  real_server 10.0.1.3 80 {

[...]

Besides this, I think it is not enough and there might be more suggestions as well:

1. If lvs_method (or lb_kind) is set to DR, keepalived will obviously forward packets as a transparent proxy (see above) so the source and destination IP Addresses will not be changed; in that case, your real server haproxy would receive an IP packet like this: 23.4.1.2:44732 -> 10.0.0.2:443; you must accept that traffic on all real servers to create the socket; RedHat has an interesting article [1].

2. You might need to specify/clarify which device acts as L3.

3. You may want to traverse keepalived [trasparent] proxy with SSL.

4. Checks from keepalived for real servers should be on port 443 and not port 80.

5. net.ipv4.ip_forward=1 is not needed in this case, leave it disabled.

---

[1] 3.2. Load Balancer Using Direct Routing: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/s1-lvs-direct-vsa#doc-wrapper

[pqarmitage]

@moaali Do you still need assistance with this, or have you resolved your problems?

As a starting point I would look at the Linux Virtual Server web site, e.g. http://linuxvirtualserver.org/VS-DRouting.html for direct routing.

When you use Direct Routing, IPVS forwards the incoming packets without modifying the IP layer, so a TCP packet destination port 80 received from 1.2.3.4 to 115.101.1.17 will be forwarded to the chosen real server using the same IP addresses. It uses the configured IP address of the real server in order to set the destination MAC address for the forwarded packet. This means two things:

1. The system on which keepalived is running must be able to find the MAC address associated with IP addresses 10.0.1.2, 10.0.1.3 and 10.0.1.4, This mean that the system on which keepalived is running probably needs an IP address on 10.0.1.0/24, such as 10.0.1.1/24 which does not appear to be being used.

2. When a real server receives a packet forwarded by IPVS, the destination MAC address will be 115.101.1.17. This means that 115.101.1.17 must be configured as a local address on each of the real servers (I tend to create an additional loopback interface and configure the address on that) or alternatively use iptables/nftables DNAT for 115.101.1.17 to a local address. Either way, since HAProxy will reply on the same socket as it received the packet on, return packets will have the source IP address 115.101.1.17, and so an iptables SNAT entry is not required (BTW the iptables entry you show above should use --sport rather than --dport (you can check whether the rule is being used by looking at the counters in the iptables -t nat -nvL POSTROUTING output)).

The final issue that occurs to me is the return path for packets back to the internet. The example shown on the Linux Virtual Server web page referred to above shows return packets not going back via the keepalived server, whereas it is not clear what the return path is in your configuration. One potential issue to check, if the return path is via the keepalived server, is whether it is happy forwarding packets that it receives with the source address of the packet matching one of its own IP addresses (i.e. 115.101.1.17).