k3s flannel TCP fails if custom interface be used with --flannel-iface and TSO is off

[shakibamoshiri]

I am trying to secure connection between two nodes (master and worker) with a layer 2 VPN.
The VPN server can connect to a bridge on each machine and k3s flannel with --flannel-iface vpn_interface can communicate
But the VPN server throughput is low if TSO (TCP Segmentation Offload) is on
So I tried to disable it

ethtool -K vpn_interface tso off

then while layer 2 and 3 are okay , I can ping other pods on worker node from master , but TCP faces timeout
Further checking with tcpdump showed that DF flag is on (Do not fragment)

layer 3 (ping okay) from master to worker

tcpdump -i any  -vvv -n dst host 10.42.0.10
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:19:33.561469 flannel.1 Out IP (tos 0x0, ttl 64, id 15331, offset 0, flags [DF], proto ICMP (1), length 84)
    10.42.1.0 > 10.42.0.10: ICMP echo request, id 9397, seq 1, length 64
07:19:34.562794 flannel.1 Out IP (tos 0x0, ttl 64, id 15576, offset 0, flags [DF], proto ICMP (1), length 84)
    10.42.1.0 > 10.42.0.10: ICMP echo request, id 9397, seq 2, length 64
07:19:35.564544 flannel.1 Out IP (tos 0x0, ttl 64, id 15804, offset 0, flags [DF], proto ICMP (1), length 84)
    10.42.1.0 > 10.42.0.10: ICMP echo request, id 9397, seq 3, length 64
07:19:36.565485 flannel.1 Out IP (tos 0x0, ttl 64, id 15904, offset 0, flags [DF], proto ICMP (1), length 84)
    10.42.1.0 > 10.42.0.10: ICMP echo request, id 9397, seq 4, length 64

layer 4 or 7 (netcat or curl failed) from master to worker

07:19:51.276136 flannel.1 Out IP (tos 0x0, ttl 64, id 57369, offset 0, flags [DF], proto TCP (6), length 60)
    10.42.1.0.56974 > 10.42.0.10.80: Flags [SEW], cksum 0x158c (incorrect -> 0xb59b), seq 1756966590, win 65535, options [mss 1410,sackOK,TS val 1064027828 ecr 0,nop,wscale 10], length 0
07:19:52.286759 flannel.1 Out IP (tos 0x0, ttl 64, id 57370, offset 0, flags [DF], proto TCP (6), length 60)
    10.42.1.0.56974 > 10.42.0.10.80: Flags [S], cksum 0x158c (incorrect -> 0xb268), seq 1756966590, win 65535, options [mss 1410,sackOK,TS val 1064028839 ecr 0,nop,wscale 10], length 0
07:19:54.302759 flannel.1 Out IP (tos 0x0, ttl 64, id 57371, offset 0, flags [DF], proto TCP (6), length 60)
    10.42.1.0.56974 > 10.42.0.10.80: Flags [S], cksum 0x158c (incorrect -> 0xaa88), seq 1756966590, win 65535, options [mss 1410,sackOK,TS val 1064030855 ecr 0,nop,wscale 10], length 0

Thus turning TSO off (=disable) on the VPN bridge -- flannel cannot or disallows TCP !?
I should mention that I always use this VPN with a bridge and TSO is off and had no issue. which mean if there are a web server on worker node netcat or curl can talk with

Any idea what is the root cause of this issue ?
Regards

[shakibamoshiri]

For the first test I used veth for flannel and failed (mentioned above)
today did the second test and used a bridge for flannel and a veth connected to this bridge what TSO was off
Is it working now and I am doing more tests , it failed , I will update this discussion

[shakibamoshiri]

One more time testing again with veth and flannel does not send ? TCP (L4) but allows icmp (L3)
but using a bridge it is okay
The strange part is that the VPN server has no issue with veth setup !