Support services defined by External IPs

[brianehlert]

A popular use case is the ability to use NGINX Ingress Controller as a general purpose software based proxy, that runs on Kubernetes compute.
This has becomes popular with companies that have abandoned machines & virtual machines and have standardized on Kubernetes as their compute platform of choice. This is primarily driven by a desire for common operational practices, flexibility that comes with containers, and 'git ops' practices such as 'infrastructure as code.'

NIC supports externalName services today as upstream / backend targets.
ExternalName Services require the resolver directive and an external DNS server / source be defined.
They are limited to one DNS entry.
The single DNS entry limitation is acceptable for customers that have the ability to manage their own internal DNS service. As a single DNS name can return an array of IP addresses back to NGINX which will simply populate the list if IP addresses as the upstream targets.
This is how resolver functions.

One possible way to address a limitation like this is to add support for ExternalIPs services
This would give customers similar control as adding multiple servers to a single DNS record, but covers the case where the folks that need the flexibility do not have this level of control to DNS, but can be granted this level of control in Kubernetes.
I would say this is preferred.

When implementing this, it is recommended that at a minimum passive health checks also be implemented (this requires the ability to send a ping (aka ICMP) on the network outside the cluster to the target. More reliable would be an active healthcheck, which is available with N+. Either way, NGINX would eventually stop sending traffic to any single failed upstream on its own after multiple failed requests.
Health checks are available and are applied to each upstream that is resolved from the K8s API.

Alternative ideas:
Allow multiple ExternalNames services be defined
Without significant logic changes, this would result in a single upstream group be defined for each ExternalName Service. While this is not an absolute functional deficiency, it requires the use of splits.
A customer can do this today. While not ideal, it is possible. And each upstream group would contain one upstream service and the individual server entries resolved by resolver.

Support a customer being able to define a list of DNS or IP names as upstreams (instead of services)
This is a pattern that while valid for NGINX does not follow Kubernetes patterns at all.
Due to that, we don't recommend this.
This is similar to not supporting snippets for upstreams. Arbitrary injection of a block of conf does not align with how upstreams are managed by system (ingress controller).

[leonseng]

just did some tests with split - one problematic behaviour is it traffic is always split between the multiple external upstreams, even to the ones that are unhealthy or do not have valid DNS names. Ability to somehow define multiple ExternalNames to a single upstream with multiple servers would be ideal.

[leonseng]

Here's my config for split test with 1 valid upstream, and another invalid (dummy DNS name)

Config

apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: v1
kind: Service
metadata:
  name: dummy
spec:
  type: ExternalName
  externalName: thisdoesnotexist.foo
---
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: ext-lb
spec:
  host: ext-lb.example.com
  upstreams:
    - name: httpbin
      service: httpbin
      port: 80
      healthCheck:
        enable: true
        path: /ip
        headers:
          - name: Host
            value: httpbin.org
        mandatory: true
        persistent: true
    - name: dummy
      service: dummy
      port: 80
      healthCheck:
        enable: true
        path: /
        headers:
          - name: Host
            value: thisdoesnotexist.foo
        mandatory: true
        persistent: true
  routes:
    - path: /
      splits:
        - weight: 50
          action:
            proxy:
              upstream: httpbin
        - weight: 50
          action:
            proxy:
              upstream: dummy

The resulting nginx config

upstream vs_temp_ext-lb_dummy {
    zone vs_temp_ext-lb_dummy 512k;
    random two least_conn;
    server thisdoesnotexist.foo:80 max_fails=1 fail_timeout=10s max_conns=0 resolve;
}
upstream vs_temp_ext-lb_httpbin {
    zone vs_temp_ext-lb_httpbin 512k;
    random two least_conn;
    server httpbin.org:80 max_fails=1 fail_timeout=10s max_conns=0 resolve;
}
split_clients $request_id $vs_temp_ext_lb_splits_0 {
    50% /internal_location_splits_0_split_0;
    50% /internal_location_splits_0_split_1;
}
server {
    listen 80;
    listen [::]:80;
    server_name ext-lb.example.com;
    status_zone ext-lb.example.com;
    set $resource_type "virtualserver";
    set $resource_name "ext-lb";
    set $resource_namespace "temp";
    server_tokens "on";
    location / {
        rewrite ^ $vs_temp_ext_lb_splits_0 last;
    }
    location @hc-vs_temp_ext-lb_httpbin {
        proxy_set_header Host "httpbin.org";
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        proxy_pass http://vs_temp_ext-lb_httpbin;
        health_check uri=/ip interval=5s jitter=0s
            fails=1 passes=1
             mandatory persistent
             keepalive_time=60s;
    }
    location @hc-vs_temp_ext-lb_dummy {
        proxy_set_header Host "thisdoesnotexist.foo";
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        proxy_pass http://vs_temp_ext-lb_dummy;
        health_check uri=/ interval=5s jitter=0s
            fails=1 passes=1
             mandatory persistent
             keepalive_time=60s;
    }
    location /internal_location_splits_0_split_0 {
        set $service "httpbin";
        status_zone "httpbin";
        internal;
        set $default_connection_header close;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_buffering on;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $vs_connection_header;
        proxy_pass_request_headers on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host "$host";
        proxy_pass http://vs_temp_ext-lb_httpbin$request_uri;
        proxy_next_upstream error timeout;
        proxy_next_upstream_timeout 0s;
        proxy_next_upstream_tries 0;
    }
    location /internal_location_splits_0_split_1 {
        set $service "dummy";
        status_zone "dummy";
        internal;
        set $default_connection_header close;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_buffering on;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $vs_connection_header;
        proxy_pass_request_headers on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host "$host";
        proxy_pass http://vs_temp_ext-lb_dummy$request_uri;
        proxy_next_upstream error timeout;
        proxy_next_upstream_timeout 0s;
        proxy_next_upstream_tries 0;
    }
}

Results

Traffic is being split between a real upstream (user agent returned in response) and the dummy upstream (502 in response). Expectation is NIC would only send traffic to healthy upstream

$ curl --resolve ext-lb.example.com:80:192.168.255.100 ext-lb.example.com/user-agent
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.25.3</center>
</body>
</html>
$ curl --resolve ext-lb.example.com:80:192.168.255.100 ext-lb.example.com/user-agent
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.25.3</center>
</body>
</html>
$ curl --resolve ext-lb.example.com:80:192.168.255.100 ext-lb.example.com/user-agent
{
  "user-agent": "curl/7.68.0"
}

NIC Logs

Logs show NIC proxying traffic to both the valid upstream and dummy upstream

1. Valid upstream

   192.168.0.115 - - [17/Jun/2024:03:20:10 +0000] ext-lb.example.com 18.211.234.122:80 "GET /user-agent HTTP/1.1" 200 34 "-" "curl/7.68.0" "-"
   

2. Upstream with dummy

   2024/06/17 03:20:10 [error] 175#175: thisdoesnotexist.foo could not be resolved (3: Host not found)
   192.168.0.115 - - [17/Jun/2024:03:20:11 +0000] ext-lb.example.com 18.211.234.122:80 "GET /user-agent HTTP/1.1" 200 34 "-" "curl/7.68.0" "-"
   2024/06/17 03:20:12 [error] 175#175: *775 no live upstreams while connecting to upstream, client: 192.168.0.115, server: ext-lb.example.com, request: "GET /user-agent HTTP/1.1", upstream: "http://vs_temp_ext-lb_dummy/user-agent", host: "ext-lb.example.com"