Telepresence breaks ec2 machine's internet #2389
Comments
|
First off, you should never run telepresence with The logs aren't that helpful unless you run I also suggest you try the |
|
To answer the question regarding isolating telepresence. Yes, it can be done. You can run telepresence in a docker container. That will only modify the docker containers network and depending on how that network is setup, it can be either completely isolated or confined to a network only reachable from other docker containers. Doing so will of course mean that you won't be able to access the cluster using cluster DNS names from outside of those containers so a process acting as an interceptor must also run in a docker container. |
|
Thinking more about this, the logs from the traffic-manager pod would also be very interesting to see at the point when the ec2 machine's internet breaks. |
|
Another thing of interest is what agent that is installed on the pod that you're intercepting. It's easy to determine by looking at the very start of the log for that pod's |
|
@bhavitsharma can you please try this with the latest 2.5.1 version? I was able to reproduce your problem on an ec2 instance and 2.5.1 should resolve it. It would be great to get that verified. |
|
Thanks for reaching out. I'll test this and get back to you ASAP. |
|
Hi @thallgren , I am still running into same issue. Here's what I did (v 2.5.1).
The thing that stands out is I have to open a tty shell via ec2 serial console to kill the telepresence agent (because SSH stops working). Even after I kill it, I can SSH in my ec2 but the internet is still disconnected. Here's the output of Here are the daemon logs |
|
@bhavitsharma what does your |
|
It looks like this: |
|
I think you have a |
|
My which seems to indicate that the Try adding This will tell the TUN-device to exclude that address. |
|
A qualified guess as to what's happening on your ec2-instance.
If this guess is correct, then the |
|
@bhavitsharma three outputs from your ec2 instance that would be of interest : |
|
Thanks @thallgren, you were right. The EKS cluster to which I was able to connect didn't have its subnet's address prefix collision with my ec2 IP. Connecting to the cluster which does breaks my internet has its subnet colliding with my EC2. I manually verified from our subnets configuration (and terraform configs). I tried |
|
|
|
So finally I was able to connect via SSH also! I added a default route for our VPN gateway the same way your I hope that we can get these features inside telepresence! |
|
@thallgren My only question: Is there a way to specify some default-routes/never-proxy options in telepresence's config rather than kubeconfig? |
|
@bhavitsharma glad to hear that you were able to get things working! Telepresence will currently only support TCP and UDP. To answer your question. No, the kubeconfig is the only place where this can be added. |
|
I'm closing this ticket in favor of #2429 which is more focused on the underlying problem. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Hi, I am using telepresence to connect to my staging and CI EKS clusters via EC2 machines. Connecting to CI clusters works amazingly well, but connecting to staging clusters don't. Specifically, when I connect to telepresence via
sudo -E telepresence connect, the root/user daemon starts but then it somehow interferes with my EC2's internet. Worst case is I am not able to SSH in my machine, the best case - my EC2 machine can't even ping google.com unless I dotelepresence quit.Orthogonal question: Is there a way to run telepresence in a more isolated way such that it doesn't interfere with my machine's internet?
To Reproduce
Steps to reproduce the behavior:
sudo -E telepresence connectExpected behavior
Telepresence to not mess with my machine's internet.
Versions
Additional context
Due to PII, I can't share all the logs gathered from
telepresence gather-logsbut if there's any specific log which is required then I can share after redacting.Seeing logs like:
Connector
Daemon
The text was updated successfully, but these errors were encountered: