Replies: 95 comments 59 replies
-
|
I can validate that this works properly for me with Traefik 2.2 in a 3 node Ubuntu cluster with |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond I don't believe so. However, I'm using a managed cloud k3s service. |
Beta Was this translation helpful? Give feedback.
-
|
Just 3 bare metal nodes running k3s on Ubuntu 19.10. Flannel is in host-gw mode, using Traefik 2.2 for ingress, and MetalLB in bgp mode for external services. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @brandond |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond , any update on this? |
Beta Was this translation helpful? Give feedback.
-
|
@jawabuu I'm now using Traefik 3.1 with the KubernetesCRD provider here's my traefik service: ---
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: traefik
labels:
app: traefik
chart: "traefik-3.1.0"
release: "traefik"
heritage: "Helm"
spec:
type: LoadBalancer
selector:
app: traefik
release: traefik
ports:
- port: 80
name: web
targetPort: "web"
- port: 443
name: websecure
targetPort: "websecure"
- port: 9000
name: traefik
targetPort: "traefik"
loadBalancerIP: 10.0.3,80
loadBalancerSourceRanges:
- "0.0.0.0/0"
externalTrafficPolicy: LocalFor metallb, I found that using bgp works best. I have a Ubiquiti USG as my router, and set it up to peer with all three of my nodes. ---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 10.0.1.1
peer-asn: 64512
my-asn: 64512
hold-time: 120s
address-pools:
- name: manual
auto-assign: false
protocol: bgp
addresses:
- 10.0.3.10-10.0.3.99
- name: default
auto-assign: true
protocol: bgp
addresses:
- 10.0.3.100-10.0.3.254 |
Beta Was this translation helpful? Give feedback.
-
|
Lucky me, I also use the Ubiquiti USG as my router so I am also interested how to peer with all three of your nodes. |
Beta Was this translation helpful? Give feedback.
-
|
Here's what {
"protocols": {
"bgp": {
"64512": {
"neighbor": {
"10.0.1.20": {
"remote-as": "64512"
},
"10.0.1.21": {
"remote-as": "64512"
},
"10.0.1.22": {
"remote-as": "64512"
}
},
"parameters": {
"router-id": "10.0.1.1"
}
}
}
},
"service": {
"dns": {
"forwarding": {
"cache-size": "10000",
"except-interface": [
"eth0"
],
"options": [
"filterwin2k",
"local-ttl=60",
"host-record=usg.khaus,10.0.1.1",
"host-record=sw-core.khaus,10.0.1.2",
"host-record=ap-house.khaus,10.0.1.6",
"host-record=ap-garage.khaus,10.0.1.7",
"host-record=seago.khaus,10.0.1.20",
"host-record=maersk.khaus,10.0.1.21",
"host-record=sealand.khaus,10.0.1.22",
"srv-host=_etcd-client._tcp,seago.khaus,2379",
"srv-host=_etcd-client._tcp,maersk.khaus,2379",
"srv-host=_etcd-client._tcp,sealand.khaus,2379",
"srv-host=_etcd-server._tcp,seago.khaus,2380",
"srv-host=_etcd-server._tcp,maersk.khaus,2380",
"srv-host=_etcd-server._tcp,sealand.khaus,2380",
"server=/k3s.khaus/10.0.3.53",
"server=75.75.75.75",
"server=75.75.76.76"
]
}
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond I have narrowed down the issue - as best as I could :-) to using flannel as CNI. Calico works with Traefik 2.2 and MetalLB to resolve Real Client IP without any further configuration. |
Beta Was this translation helpful? Give feedback.
-
|
@jawabuu Those comments all seem to be about routing issues, not obtaining the original client IP? Either way, I'm using flannel in host-gw mode with traefik and metallb and getting the original address in the headers, so I know that it is doable. All you have to do is use |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond you are correct. The comments are just meant to provide context that the default installation of the latest k3s with flannel may be having some issues.
Ideally points 2 and 3 should be tested to identify edge cases. |
Beta Was this translation helpful? Give feedback.
-
|
thanks to both of you @brandond and @jawabuu |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond, upon further testing with flannel, I have found out that it is not necessary to use |
Beta Was this translation helpful? Give feedback.
-
|
i have the same issue using |
Beta Was this translation helpful? Give feedback.
-
|
Hey @mschneider82 You will not be able to get source ip using the embedded flannel in k3s. containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq=false
- --kube-subnet-mgr
- --iface=${interface} |
Beta Was this translation helpful? Give feedback.
-
|
Mhh works for me, but not if I have multiple wireguard interfaces :( |
Beta Was this translation helpful? Give feedback.
-
|
If you use haproxy as the load balancer. We did this for a while. Works
great actually. Very few load balancer/ingress can actually set proxy
protocol headers. ( That's why I filed the issue on kube-vip to incorporate
support for proxy protocol injection.)
Haproxy is a fully battle tested ingress+lb.
Then we moved to AWS built in lb.
…On Mon, 5 Apr, 2021, 15:40 Ulrich Mayring, ***@***.***> wrote:
You have been mentioning before that the HAProxy Ingress not only reads
the Proxy Protocol headers, but can also set them, if they are not present.
What would be a scenario, where I use this feature?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYU2SFA4J7XEIZL7AYHDTHGECTANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
Do you mean HAProxy external LB or HAProxy Ingress as LB within K3s? I suppose you mean HAProxy Ingress, which then would mean no HA or load balancing, because you always have to point to one node. Which is why you moved to AWS LB. If HAProxy external LB, then you would set the Proxy Protocol headers there and could also use Traefik or NGINX Ingress, since they can also read the headers. I haven't quite gotten my head around how Kube-VIP is supposed to work, but I have upvoted :) |
Beta Was this translation helpful? Give feedback.
-
|
Do you mean HAProxy external LB or HAProxy Ingress as LB within K3s? I
suppose you mean HAProxy Ingress, which then would mean no HA or load
balancing, because you always have to point to one node. Which is why you
moved to AWS LB.
either ways (and yes - ingress would mean no HA) - you will get proxy
protocol headers with haproxy.
If HAProxy external LB, then you would set the Proxy Protocol headers
there and could also use Traefik or NGINX Ingress, since they can also read
the headers.
yes you could. But no use really having a different software for lb and a
different one for ingress. Haproxy is just as impressive as traefik in
features...and possibly more battle tested.
I haven't quite gotten my head around how Kube-VIP is supposed to work,
but I have upvoted :)
kube-vip is a replacement for klipper (the built in LB in k3s). Its fairly
well supported. Just like MetalLB basically.
…On Mon, Apr 5, 2021 at 4:04 PM Ulrich Mayring ***@***.***> wrote:
Do you mean HAProxy external LB or HAProxy Ingress as LB within K3s? I
suppose you mean HAProxy Ingress, which then would mean no HA or load
balancing, because you always have to point to one node. Which is why you
moved to AWS LB.
If HAProxy external LB, then you would set the Proxy Protocol headers
there and could also use Traefik or NGINX Ingress, since they can also read
the headers.
I haven't quite gotten my head around how Kube-VIP is supposed to work,
but I have upvoted :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYUYZ2NCL7ORVB5R5MM3THGG25ANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
As far as I have understood Kube-VIP wants to solve HA, which MetalLB can't (it needs an external BGP service). Kube-VIP apparently can work with a virtual IP, which many cloud providers offer (I believe only Vultr offers BGP and you still have to install your own BGP daemon). |
Beta Was this translation helpful? Give feedback.
-
|
Basically, Kube-VIP uses ARP to elect a leader node and assign it a virtual IP and all other nodes then use BGP to advertise the virtual IP. This should take care of HA. As far as load balancing is concerned, there still needs to be a router that does it based on the cost-equivalent routes published by all the nodes. |
Beta Was this translation helpful? Give feedback.
-
|
This is how i solved the Client IP issue with my single-node k3s cluster(v1.21.5+k3s2), i think it's almost out-of-box since it need only one line patched, woks with the built-in ServiceLB/Traefik2.
curl -sfL https://get.k3s.io | sh -
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable servicelb" sh -
kubectl patch svc traefik --patch '{"spec":{"externalIPs":["<local ip>"]}}' -nkube-system
Test if X-Real-IP works using whoami kubectl create ns dev
kubectl apply -f https://gist.github.com/lazywalker/4d48eaad0ef111f5e370c9c25570cd39
# on other machine
curl -H Host:whoami.k3s.local <k3s master ip>
Hostname: web-8557b59f65-mmmxx
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.9
IP: fe80::74cb:efff:fe15:75ad
RemoteAddr: 10.42.0.1:40918
GET / HTTP/1.1
Host: whoami.k3s.local
User-Agent: curl/7.79.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.16.36.142
X-Forwarded-Host: whoami.k3s.local
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: k3s
X-Real-Ip: 172.16.36.142Clean up kubectl delete ns devThere you go! if you add agent nodes you may still need to set |
Beta Was this translation helpful? Give feedback.
-
|
the |
Beta Was this translation helpful? Give feedback.
-
|
This is not working for me. I do have I no configuration I was able to get the remote host IP on a plain, vanilla k3s installation with default settings. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, the 2 commands |
Beta Was this translation helpful? Give feedback.
-
|
Many thanks @lazywalker. The |
Beta Was this translation helpful? Give feedback.
-
|
For what its worth I have been trying this with a dual stack setup and with Cluster was installed with: Traefik config override: apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
service:
spec:
externalTrafficPolicy: Local
ipFamilyPolicy: RequireDualStack |
Beta Was this translation helpful? Give feedback.
-
|
Update: same setup seems to be working on v1.23.2+k3s1 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. Applying the traefik config override fixed the problem for me. |
Beta Was this translation helpful? Give feedback.
-
|
I‘d call it “works as designed“ as this is documented, but know you locked yourself down to a single-node “cluster“ as traffic can now potentially be spread imbalanced. |
Beta Was this translation helpful? Give feedback.
-
|
Can any of this apply to an UDP service ? I'm running pihole in k3s and only see the internal IP, not the clients. |
Beta Was this translation helpful? Give feedback.
-
|
so, without a load balancer, what are our options to use traefik and still have the hability to get real client ip? apparently if i set it up as a nodeport and externalTrafficPolicy:local i can get the real ip as "X-Real-Ip: 192.168.x.x" and "X-Forwarded-For: 192.168.x.x" headers... |
Beta Was this translation helpful? Give feedback.
-
|
hi all , deploying a mailserver on a deployment on a single node I tried with loadbalancer /nodeport service mode (tried also calico and port forwarding), but unable to have real client ip for SPF mail validation.Which is fastest way to have this work in single node k3s?thanks |
Beta Was this translation helpful? Give feedback.
-
|
I am all kinds of stuck on this. I feel that I tried just about everything - except that I am sticking with a rootless set-up. |
Beta Was this translation helpful? Give feedback.
-
|
H! What I need to do now to achieve this in multi-node cluster? |
Beta Was this translation helpful? Give feedback.
-
|
I am running a multi (3) node cluster as well. Using If however, you directly hit the node that the service is sitting on, it wil show you the real IP. I really hope there is a way to get this working, I would really like to be able to see useful access logs inside Traefik. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I found a simple solution for this problem. Basically the mode by you contact your service change from: to: than you don't contact the cni anymore. once you contact one url that is an ingress in k3s you enter in the node, than in the load balancer, than in traefik and finally on your service. I only have listened allthing writed here and I have setup traefik with hostNetwork: true and the service with externalTrafficPolicy: Local Thank you for this tread and I hope I will be helpfull. my os: virt kernel and hardening. my k3s: my traefik: klipper version: v0.4.4 what I have done for testing: create a namespace called apache-n deploy a base http container for testing: deploy a service: create an ingress: when contacting the example.example.com I have a response from the container: as you can see X-Forwarded-For: is right and X-Real-Ip: also. |
Beta Was this translation helpful? Give feedback.
-
|
thanks for your post. I was looking for a way to confirm what headers are being set by the proxy and your deployment does this. |
Beta Was this translation helpful? Give feedback.
-
|
I have same issue, Info"controller": {
"kind": "DaemonSet",
"allowSnippetAnnotations": True,
"service": {
"externalTrafficPolicy": "Local",
},
"config": {
"enable-real-ip": True,
"use-forwarded-headers": True,
"compute-full-forwarded-for": True,
"use-proxy-protocol": True,
"proxy-add-original-uri-header": True,
"forwarded-for-header": "proxy_protocol",
"real-ip-header": "proxy_protocol",
},
},InfoInfoInfoInfoI also tried default configuration that is |
Beta Was this translation helpful? Give feedback.
-
|
is there any standard setting that would allow you to send a real IP through a traefik proxy? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, use of See here for an example of hostNetwork #2997 (comment) And yes, hostNetwork means you lose IP isolation, so you need to assign unique ports from what I remember. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, (Note the different than other user reports specification of after applying and trying with |
Beta Was this translation helpful? Give feedback.
-
|
Preserving client source IPs in general is not a problem, but making it work with a scaled ingress controllers or cluster with more than one node is hard. Setting |
Beta Was this translation helpful? Give feedback.
-
|
The problem here was getting the client IP in the context of k3s, your other words are coming directly from the page you're linking and are well known and proven by others that have written about using metallb in front of k3s |
Beta Was this translation helpful? Give feedback.
-
|
I have run into this issue too - although, it's a little weird. I have the policy set to When a device other than the node does a request, I can see the correct IP in the Traefik log: But, when I do the exact same on the node itself: This is very confusing... The reported IP is in fact the And, here is the route config: When I do And, IPv4 forwarding is enabled: Alright... So, as far as I understand, cURL should send the packet as But why is it still not reporting the right IP? Why can other devices on the network be identified properly, but the node itself can not? I am trying to set up IP whitelisting for services, and this is definitively breaking my kneecap... |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, using For reference, here is my configuration: Traefik configapiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
logs:
level: INFO
access:
enabled: true
addInternals: true
fields:
general:
defaultmode: keep
headers:
defaultmode: keep
deployment:
enabled: true
replicas: 1
initContainers:
- name: volume-permissions
image: busybox:latest
command: ["sh", "-c", "touch /data/acme.json; chmod -v 600 /data/acme.json"]
securityContext:
runAsNonRoot: true
runAsGroup: 65532
runAsUser: 65532
volumeMounts:
- name: data
mountPath: /data
ports:
web:
redirectTo:
port: 443 # FIXME. sometimes redirects to :8443...
forwardedHeaders:
trustedIPs: [ "100.64.0.0/24" ]
proxyProtocol: { trustedIPs: [ "100.64.0.0/24" ] }
websecure:
forwardedHeaders:
trustedIPs: [ "100.64.0.0/24" ]
proxyProtocol: { trustedIPs: [ "100.64.0.0/24" ] }
tls:
certResolver: "letsEncrypt"
persistence:
enabled: true
existingClaim: traefik
accessMode: ReadWriteOnce
size: 256Mi
path: /data
providers:
kubernetesCRD:
enabled: true
allowExternalNameServices: true
namespaces: []
kubernetesIngress:
enabled: true
allowExternalNameServices: true
namespaces: []
publishedService:
enabled: true
rbac:
enabled: true
hostNetwork: true
service:
enabled: true
type: LoadBalancer
spec:
externalTrafficPolicy: Local
ingressClass:
enabled: true
isDefaultClass: true
certResolvers:
letsEncrypt:
dnsChallenge:
provider: cloudflare
delayBeforeCheck: 30
resolvers:
- 1.1.1.1
- 8.8.8.8
- 9.9.9.9
storage: /data/acme.json
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
env:
# CF_API_EMAIL, CF_API_KEY || CF_DNS_API_TOKEN[, CF_ZONE_API_TOKEN]
- name: CF_DNS_API_TOKEN
value: <snip>
additionalArguments:
- "--entryPoints.web.http.redirections.entryPoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https" |
Beta Was this translation helpful? Give feedback.
-
|
Also, because I ended up with my brain completely noodled from gawking at iptables for too long, I ended up posting here too: flannel-io/flannel#1967 |
Beta Was this translation helpful? Give feedback.
-
|
@skea999 Setting the deployment kind is apparently unsupported: https://github.com/traefik/traefik-helm-chart/blob/master/traefik/templates/deployment.yaml#L25 |
Beta Was this translation helpful? Give feedback.
-
|
I think that you are right because adding it to my configuration changed nothing, but there is an error, look at here |
Beta Was this translation helpful? Give feedback.
-
|
Honestly, I am not sure. o.o |
Beta Was this translation helpful? Give feedback.
-
|
Solution for me was to ditch servicelb in favor of MetalLB... 1 - Ditch servicelb 2 - Install Metallb
Or use ArgoCD / FluxCD even better. 3 - Configure Metallb IP address pool for LBs. Change depending on your local network, mine is 192.168.2.0/24 and i want traefik LB to be in 192.168.2.150. Configure L2Advertisment so that metallb provisions your LBs using pool. 4 - Extra stuff At this point you should be set up. You might encounter issues though. If you were accessing some services via tunnel (e.g. Zerotier One), this will stop working, since metallb does not listen directly on the node like servicelb does. |
Beta Was this translation helpful? Give feedback.
-
|
I followed the method you provided, but unfortunately, I still cannot obtain the client's real IP in the service. From the MetalLB documentation I learned that only setting externalTrafficPolicy to Local allows for the acquisition of the client's true IP, though the trade-off is that it leads to unbalanced traffic. |
Beta Was this translation helpful? Give feedback.
-
|
After some time I came back to this issue since I managed to fix that Just set this on nginx ingress helm chart config Remeber to Uninstall and then install again nginx ingress, a simple helm upgrade wont help |
Beta Was this translation helpful? Give feedback.
and others
-
Is your feature request related to a problem? Please describe.
I am unable to obtain Real Client IP when using k3s and Traefik v2.2. I always get the cluster IP.
Describe the solution you'd like
I would like to obtain the client IP.
Describe alternatives you've considered
I already set
externalTrafficPolicy: Localin Traefik's Service.
Additional context
Issue can be reproduced by deploying containous/whoami image in cluster
Expected Response
Current Response
Service LoadBalancer Logs
Related
#955
Related Discussion
#679 (comment)
@erikwilson @btashton
Beta Was this translation helpful? Give feedback.
All reactions