Communication between modems is broken

[Szczempus]

Hello again. In my previous question, the drone problem was related to SBC and it working properly. Now I have a problem with another drone, that was functioning well until yesterday when wifi connection broke permanently. When we get a closer look at the logs, gs detects a new session and tries to decrypt incoming packages, but fails.

GS log:

Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Log opened.
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] # iw reg set BO
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] # ifconfig rtl8812au down
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] # iw dev rtl8812au set monitor otherbss
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] # ifconfig rtl8812au up
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] # iw dev rtl8812au set channel 165 HT20
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Connect telem stream 2(RX), 1(TX) to 127.0.0.1:14550
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] UDPProxyProtocol starting on 37153
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] UDPProxyProtocol starting on 14600
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] UDPProxyProtocol starting on 50033
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Telem RX: /usr/bin/wfb_rx -p 2 -u 14600 -K /etc/gs.key -k 1 -n 2 rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Telem TX: /usr/bin/wfb_tx -p 1 -u 14601 -K /etc/gs.key -B 20 -G long -S 1 -L 0 -M 1 -k 1 -n 2 -T 0 rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] AntennaFactory starting on 8001
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Started telem rx
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Started telem tx
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Send video stream 3 to 127.0.0.1:10000
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] AntennaFactory starting on 8002
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Started video rx
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Video: /usr/bin/wfb_rx -p 3 -c 127.0.0.1 -u 10000 -K /etc/gs.key -k 8 -n 12 rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] UDPProxyProtocol starting on 14800
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] UDPProxyProtocol starting on 51134
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Tunnel RX: /usr/bin/wfb_rx -p 5 -u 14800 -K /etc/gs.key -k 1 -n 2 rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Tunnel TX: /usr/bin/wfb_tx -p 4 -u 14801 -K /etc/gs.key -B 20 -G long -S 1 -L 0 -M 1 -k 1 -n 2 -T 0 rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] AntennaFactory starting on 8003
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Started tunnel rx
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] Started tunnel tx
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] telem tx: Listen on 14601 for rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] telem rx: rtl8812au has DLT_IEEE802_11_RADIO Encap
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] tunnel tx: Listen on 14801 for rtl8812au
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] video rx: rtl8812au has DLT_IEEE802_11_RADIO Encap
Jul 04 07:58:45 gs wfb-server[19507]: 2024-07-04 07:58:45+0000 [-] tunnel rx: rtl8812au has DLT_IEEE802_11_RADIO Encap
Jul 04 07:59:29 gs wfb-server[19507]: 2024-07-04 07:59:29+0000 [-] video rx: New session detected
Jul 04 08:01:20 gs wfb-server[19507]: 2024-07-04 08:01:20+0000 [-] video rx: New session detected
Jul 04 08:01:20 gs wfb-server[19507]: 2024-07-04 08:01:20+0000 [-] video rx: unable to decrypt packet #0x1

On drone side, packets are correctly injected which was checked with tcpdump and netcat, but again, no connection was established with GS.

Drone log

Jul 04 08:01:20 uav wfb-server[2150]: 2024-07-04 08:01:20+0000 [-] Log opened.
Jul 04 08:01:20 uav wfb-server[2150]: 2024-07-04 08:01:20+0000 [-] # iw reg set BO
Jul 04 08:01:20 uav wfb-server[2150]: 2024-07-04 08:01:20+0000 [-] # ifconfig rtl8812au down
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] # iw dev rtl8812au set monitor otherbss
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] # ifconfig rtl8812au up
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] # iw dev rtl8812au set channel 165 HT20
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Listen for telem stream 1(RX), 2(TX) on 0.0.0.0:14550
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] UDPProxyProtocol starting on 14550
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] UDPProxyProtocol starting on 14700
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] UDPProxyProtocol starting on 56961
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Telem RX: /usr/bin/wfb_rx -p 1 -u 14700 -K /etc/drone.key -k 1 -n 2 rtl881
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Telem TX: /usr/bin/wfb_tx -p 2 -u 14701 -K /etc/drone.key -B 20 -G long -S
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Started telem rx
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Started telem tx
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Listen for video stream 3 on 0.0.0.0:10000
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Started video tx
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Video: /usr/bin/wfb_tx -p 3 -u 10000 -K /etc/drone.key -B 20 -G long -S 1 
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] UDPProxyProtocol starting on 14900
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] UDPProxyProtocol starting on 52691
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Tunnel RX: /usr/bin/wfb_rx -p 4 -u 14900 -K /etc/drone.key -k 1 -n 2 rtl88
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Tunnel TX: /usr/bin/wfb_tx -p 5 -u 14901 -K /etc/drone.key -B 20 -G long -
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Started tunnel rx
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] Started tunnel tx
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] telem tx: Listen on 14701 for rtl8812au
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] video tx: Listen on 10000 for rtl8812au
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] telem rx: rtl8812au has DLT_IEEE802_11_RADIO Encap
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] tunnel tx: Listen on 14901 for rtl8812au
Jul 04 08:01:21 uav wfb-server[2150]: 2024-07-04 08:01:21+0000 [-] tunnel rx: rtl8812au has DLT_IEEE802_11_RADIO Encap

Drone config

[common]
mavlink_agg_timeout=0.05
wifi_channel=169
wifi_region='BO'

[drone_mavlink]
%23%20peer='connect://127.0.0.1:14550'  # outgoing connection
peer='listen://0.0.0.0:14550'   # incoming connection

[drone_video]
peer='listen://0.0.0.0:10000'  # listen for video stream (gstreamer on drone) up

GS config

[common]
mavlink_agg_timeout=0.05
wifi_channel=169
wifi_region='BO'

[gs_mavlink]
%23%20peer='listen://0.0.0.0:14550'   # incoming connection
peer='connect://127.0.0.1:14550'  # outgoing connection

[gs_video]
peer='connect://127.0.0.1:10000'  # outgoing connection for

Compiled driver version: v5.6.4.2_35491.20191025
Wifibroadcast app version: 21.8.27.56115-1

What I tried was changing the modems, generating a new key pair for drone and gs, removing and reinserting driver again, and checking the power wiring and antenna connections. All that with no success. Do you have any guess what may be a problem?

[svpcom]

1. what is %23%20peer ? Change it to # peer if you want to comment line.
2. Ensure that modem(s) was not died (ac180 can die if work several seconds without antenna or with unmatched antenna)
3. You wfb-ng version is very old. It is not an issue itself but I'll recommend to upgrade. At least upgrade 8812au driver from https://github.com/svpcom/rtl8812au
4. Ensure that you have unmanaged-devices in /etc/NetworkManager/NetworkManager.conf. New wfb-ng doesn't require it (it do automatically during start), but old version requires

[svpcom]

Or something like wpa_supplicant/dhclient/network_manager/etc touch wifi interfaces

[svpcom]

What python version do you have? (see error description in window title)

[svpcom]

It your python is old, just remove strict_map_key=False from wfb_ng/cli.py

[Szczempus]

see dmesg. It seems your card was disconnected

Yep, you were right. I didn't notice that my usb hub was disconnecting the modem

What python version do you have? (see error description in window title)

python2.7. I will recompile the application with a modified argument and see if it works.

[svpcom]

Python2.7 is not supported. Please use python 3.x

[Szczempus]

Well, @svpcom, your answer about ac180 modules, which die if they have disconnected or unmatched antennas was partially the cause of my problem. As it occurred, when I swapped both modems to a version with metal shielding around them, the problem disappeared. Since then I have got near-field probes and checked the EMI of the modems with and without shielding. As a result, I saw beautiful interferences ranging from 200MHz to about 500MHz on the modem without shielding. If you guys would like to I can share with you a spectrogram of three versions of AC180 which I have, to give you an overview to choose the best one to buy ;)

[svpcom]

Yes, please share the measurements