Dynamic port forwarding using tsh ssh -D cripples nmap functionality

[AlexTudurean]

Hi there!
I am having some issues when trying to use a teleport node as a sort of proxy to scan resources inside networks behind that said proxy node.

My setup looks like this:
I have a node inside my Teleport cluster that i would like to use to proxy nmap scans to resources behind this node (the resources are disconnected from the internet, the teleport node acts as the only link to the outside, to access the resources you need to first ssh into "proxy node" and then you gain access to the resources inside the said network).

I want to be able to do this remotely from another teleport agent so I basically start a dynamic port forward:
tsh ssh -D 1080 <teleport_node>

Then I use this new socks5 proxy in conjuction with proxychains to scan a resource in the network i mentioned before:
proxychains4 nmap -sT -Pn -sV -n <resource_ip>

The result is that somehow... all ports are open and tcpwrapped. The same command but using scanme.nmap.org as the resource never actually ends (or i never waited enough for it to end, but i don't think it actually ever ends).

Removing the -sV flag from nmap does make it "work" but renders the output absolutely useless because all ports are seen as open.

Doing the same exact thing as above but instead using standard ssh (OpenSSH) to dynamically port forward traffic results in the expected outcome:
ssh -D 127.0.0.1:1080 <teleport_node>

Does anybody have any idea if tsh somehow changes the way that a normal dynamic port forward works as opposed to using ssh?

[webvictim]

This is interesting. tsh dynamic port forwarding is indeed implemented differently to OpenSSH, but the fact that it works when using ssh suggests that this isn't a server-side issue and could instead be a fixable bug with tsh.

[AlexTudurean]

The nmap scan traffic seems to be going through the internet instead of directly through the local networks (our infrastructure is provided by a cloud vendor) in between the hosts that i mentioned in my original question so I will first try to make it so the traffic goes through the desired/expected way and update you if the issue still comes up then.

[webvictim]

Please do, thanks.

[AlexTudurean]

Hi there! It's been a while, I've been busy with other things.
I have created (finally after a really long time as you can see) a more isolated infrastructure where all the traffic goes straight through the machines in question.... aaaandd.. mmye, still nothing. Interestingly, every time nmap hits a closed port, teleport logs it as a forwarding error:

ERRO             Error handling forwarding request for address "192.168.5.16:62". error:[
ERROR REPORT:
Original Error: *net.OpError dial tcp 192.168.5.16:62: connect: connection refused
Stack Trace:
	github.com/gravitational/teleport/lib/srv/reexec.go:628 github.com/gravitational/teleport/lib/srv.handleLocalPortForward
	github.com/gravitational/teleport/lib/srv/reexec.go:780 github.com/gravitational/teleport/lib/srv.runForward.func2
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: dial tcp 192.168.5.16:62: connect: connection refused] srv/reexec.go:781

This somehow seems to mess with the way that nmap determines if a port is open or not.
Sooo ye, any ideas? :))

[webvictim]

I tried to reproduce this and can confirm that I get the same result.

It seems that Teleport implements its SOCKS proxy by opening direct-tcpip channels from the target machine within the SSH session, and the output of these is not directly passed back to the caller which causes nmap to report false results.

I'm not sure if this is strictly a "bug", but it certainly looks like one. I'll investigate a little further and see whether we've ever thought about this sort of use case. It seems to me that we ought to at least propagate the connection failure through to the client - we see connection refused in the Teleport logs, but the client doesn't get that message.

Test outputs

Direct from the host

Here's the test direct from the proxy machine - port 22 open and 23 closed is exactly the output I expect:

ubuntu@ip-172-31-8-63:~$ nmap -A -T4 -Pn -sV -p 22-23 -n 172.31.33.224
Starting Nmap 7.80 ( https://nmap.org ) at 2024-06-05 18:30 UTC
Nmap scan report for 172.31.33.224
Host is up (0.0013s latency).

PORT   STATE  SERVICE VERSION
22/tcp open   ssh     OpenSSH 8.7 (protocol 2.0)
23/tcp closed telnet

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 0.56 seconds

sshd

Using ssh -D -> sshd on the host - results are correct, nmap realises port 23 is closed and reports correctly:

gus@apollo:~ % proxychains4 nmap -A -T4 -Pn -sV -p22-23 -n 172.31.33.224
[proxychains] config file found: /opt/homebrew/etc/proxychains.conf
[proxychains] preloading /opt/homebrew/Cellar/proxychains-ng/4.17/lib/libproxychains4.dylib
[proxychains] DLL init: proxychains-ng 4.17
Starting Nmap 7.95 ( https://nmap.org ) at 2024-06-05 17:30 ADT
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:23 <--socket error or timeout!
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18083  ...  172.31.33.224:22  ...  OK
Nmap scan report for 172.31.33.224
Host is up (0.038s latency).

PORT   STATE  SERVICE VERSION
22/tcp open   ssh     OpenSSH 8.7 (protocol 2.0)
| ssh-hostkey:
|   256 9e:ba:1b:de:63:b5:fb:78:92:07:41:5e:e9:35:b1:b6 (ECDSA)
|_  256 df:3b:5e:bb:68:3c:c7:b2:62:20:ef:9a:d8:0c:da:b7 (ED25519)
23/tcp closed telnet

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 5.42 seconds

Here's the same test using sshd as the server (via Teleport's agentless sshd integration) but with tsh ssh -D as the client - 23 is incorrectly reported as open when it's closed:

gus@apollo:~ % proxychains4 nmap -A -T4 -Pn -sV -p22-23 -n 172.31.33.224
[proxychains] config file found: /opt/homebrew/etc/proxychains.conf
[proxychains] preloading /opt/homebrew/Cellar/proxychains-ng/4.17/lib/libproxychains4.dylib
[proxychains] DLL init: proxychains-ng 4.17
Starting Nmap 7.95 ( https://nmap.org ) at 2024-06-05 15:30 ADT
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
116
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18082  ...  172.31.33.224:23  ...  OK
Nmap scan report for 172.31.33.224
Host is up (0.00060s latency).

PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 8.7 (protocol 2.0)
| ssh-hostkey:
|   256 9e:ba:1b:de:63:b5:fb:78:92:07:41:5e:e9:35:b1:b6 (ECDSA)
|_  256 df:3b:5e:bb:68:3c:c7:b2:62:20:ef:9a:d8:0c:da:b7 (ED25519)
23/tcp open  telnet?

teleport

Here's the same test via the teleport binary - 22 is open, but 23 is incorrectly reported as open too when it's actually closed:

gus@apollo:~ % proxychains4 nmap -A -T4 -Pn -sV -p22-23 -n 172.31.33.224
[proxychains] config file found: /opt/homebrew/etc/proxychains.conf
[proxychains] preloading /opt/homebrew/Cellar/proxychains-ng/4.17/lib/libproxychains4.dylib
[proxychains] DLL init: proxychains-ng 4.17
Starting Nmap 7.95 ( https://nmap.org ) at 2024-06-05 17:23 ADT
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:23  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:22  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:18080  ...  172.31.33.224:23  ...  OK
Nmap scan report for 172.31.33.224
Host is up (0.00066s latency).

PORT   STATE SERVICE    VERSION
22/tcp open  ssh        OpenSSH 8.7 (protocol 2.0)
| ssh-hostkey:
|   256 8e:a4:35:3f:76:23:4d:8f:0b:32:f5:0d:2d:16:5d:05 (ECDSA)
|_  256 85:72:f3:1d:db:fd:cd:a0:78:58:c4:9d:98:6f:66:f6 (ED25519)
23/tcp open  tcpwrapped

[AlexTudurean]

I see, ok, ok, so my assumption that the failure was not passed back to the client was correct. I got the same exact results as you did during testing so at least that adds up. Hmmm, nice :)) If I can in any way help implement such a thing or create a "fix" for such a thing I am more than happy to, but I'll need a tiny bit of guidance :)) Thanks!