UDP Based LB is not working for the example(basic-tcp-udp) specified in examples folder

[githubstoreforme]

Dear Team,

I am trying to establish UDP based Load balancing, as part of it I am trying out the example specified in the below link.
I have tried all the steps specified in the example as is (tried multiple times) on AKS but couldn't succeed (I don't see any logs been logged either in NGINX or CoreDNS also).

Could please you check once and let me know.

Path:

https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources)/basic-tcp-udp/

and also I have tried below example also in another AKS setup

https://github.com/nginxinc/kubernetes-ingress/tree/v2.2.2/examples/tcp-udp

[jasonwilliams14]

@githubstoreforme can you share your deployment of Ingress (including configmap)?
DId you apply all of our CRDs?

[githubstoreforme]

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
replicas: 1
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
#prometheus.io/scheme: http
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:2.2.2
imagePullPolicy: IfNotPresent
name: nginx-ingress
ports:
- name: dns-udp
containerPort: 5353
- name: http
containerPort: 80
- name: https
containerPort: 443
- name: readiness-port
containerPort: 8081
- name: prometheus
containerPort: 9113
readinessProbe:
httpGet:
path: /nginx-ready
port: readiness-port
periodSeconds: 1
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
- -global-configuration=$(POD_NAMESPACE)/nginx-configuration
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-prometheus-metrics

---

apiVersion: v1
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":null,"kind":"ConfigMap","metadata":{"annotations":{},"name":"nginx-config","namespace":"nginx-ingress"}}
creationTimestamp: "2022-06-06T18:49:46Z"
name: nginx-config
namespace: nginx-ingress
resourceVersion: "168116"
uid: 604c5d95-54ef-4d25-9eb6-8b70c2901e25

---

and created below CRD's

$ kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml

I am struggling to bring this up for last 3days. Any help would be much appreciated.

[jasonwilliams14]

@githubstoreforme did you create your globalconfiguration to setup the UDP listeners as well as your transport server todo L4 UDP loadbalancing?
If so, can you run:

kubectl get transportservers -A
kubectl describe transportservers <name_of_ts>

[githubstoreforme]

Yes, I did.

Below is the result of the kubectl describe transportservers dns-udp

---

Name: dns-udp
Namespace: default
Labels:
Annotations:
API Version: k8s.nginx.org/v1alpha1
Kind: TransportServer
Metadata:
Creation Timestamp: 2022-06-06T18:57:12Z
Generation: 1
Managed Fields:
API Version: k8s.nginx.org/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:message:
f:reason:
f:state:
Manager: Go-http-client
Operation: Update
Subresource: status
Time: 2022-06-06T18:57:12Z
API Version: k8s.nginx.org/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:action:
.:
f:pass:
f:listener:
.:
f:name:
f:protocol:
f:upstreamParameters:
.:
f:udpRequests:
f:udpResponses:
f:upstreams:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-06-06T18:57:12Z
Resource Version: 169998
UID: bc6bf543-c94a-42a7-bfa1-1bdad038ca9a
Spec:
Action:
Pass: dns-app
Listener:
Name: dns-udp
Protocol: UDP
Upstream Parameters:
Udp Requests: 1
Udp Responses: 1
Upstreams:
Name: dns-app
Port: 5353
Service: coredns
Status:
Message: Configuration for default/dns-udp was added or updated
Reason: AddedOrUpdated
State: Valid
Events:

[githubstoreforme]

Any clue?

[jasonwilliams14]

@githubstoreforme have you seen this recent article on this topic?

TCP/UDP Load balancing with NGINX Ingress

That has good information for what you are trying to do.

[jasonwilliams14]

@githubstoreforme did you open up the port on your upstream load balancer? DO you have that file available to review?
I did a quick test this morning. I checked with and without my test port (5353) open on the LB. With the port open, I can successfully load balancer layer 4 connections. With the port NOT open on the LB, it does not work.
Let us know

[githubstoreforme]

I did that also. Did you see the expected log content in CoreDNS container as well?
(I have a feeling that there is UDP data loss)

Below is the resource.

apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
externalTrafficPolicy: Local
type: LoadBalancer
ports:

• port: 5353
  targetPort: 5353
  protocol: UDP
  name: dns-udp
  selector:
  app: nginx-ingress

[githubstoreforme]

I have tried with below service instead of CoreDNS, I see that occasionally data is getting reached the service.
It is very clear that there is some data loss. Please check once .

Thanks a lot for your help.

https://github.com/nginxinc/kubernetes-ingress/blob/main/tests/data/transport-server-udp-load-balance/standard/service_deployment.yaml

[githubstoreforme]

Any suggestions?