AP_DDS: MicroROS agent doesn't publish topics when GPS automatically gets GPS lock. #25535
Comments
|
If you could add the mavlink log output as well that would be great. From memory, there were no mentions of AP_DDS showing in the log if the AP initialised with a hot GPS lock. I had a look through the code and could not see any conditionals, especially anything that would prevent the output such as "DDS initialised" - could something be crashing? |
|
So I have connected to the cubeorange using the mavlink terminal console. This is the output when the device instantly find geofix. Note that the mavlink never receives anything more than what is in this picture: This is the output when it doesn't. As you can see it calls the AP_DDS this time: |
|
Thanks for the detailed report. I have most of the similar hardware, I will try to reproduce it ASAP. Sorry for the trouble! |
|
Which commit hash are you currently testing? |
|
Hey @Ryanf55, so we are now on trunk. Our configuration is: # project configured on Thu Nov 16 12:52:45 2023 by
# waf 2.0.9 (abi 20, python 30a0cf0 on linux)
# using /home/kyle/dev/test/ardupilot/ardupilot/modules/waf/waf-light configure --board CubeOrange --enable-ddsSo we think we have narrowed the issue down to the AP_DDS_client.cpp file. Specifically the "APP_DDS_Client::init_session function. bool AP_DDS_Client::init_session()
{
// init session
uxr_init_session(&session, comm, key);
// Register topic callbacks
uxr_set_topic_callback(&session, AP_DDS_Client::on_topic_trampoline, this);
// ROS-2 Service : Register service request callbacks
uxr_set_request_callback(&session, AP_DDS_Client::on_request_trampoline, this);
while (!uxr_create_session(&session)) {
GCS_SEND_TEXT(MAV_SEVERITY_INFO, "%s Initialization waiting...", msg_prefix);
hal.scheduler->delay(1000);
}Using mavlink messages we can see that everything upto the "while (!uxr_create_session(&session))" executes, but it appears to get stuck in "uxr_create_session(&session))" which is from "session.c". NOTE: The "initialization waiting..." msg does not execute and nor does anything beyond this point (when GPS has instant geofix, it does when it had no geofix first). |
|
Hi, Sorry it appears that it actually gets stuck earlier in the AP_DDS_client.cpp code but I had it commented out and forgotten about it: // check ping
const uint64_t ping_timeout_ms{1000};
const uint8_t ping_max_attempts{10};
if (!uxr_ping_agent_attempts(comm, ping_timeout_ms, ping_max_attempts)) {
GCS_SEND_TEXT(MAV_SEVERITY_ERROR, "DDS Client: No ping response, exiting");
// return;
}The return is commented out because otherwise the hardware will stop trying to connect to the DDS client after 10 seconds. This has no bearing on the current issue. The code appears to get stuck in "uxr_ping_agent_attempts". We are theorising that it could be something to do with the locking transport stuff within the "uxr_ping_agent_attempts" function. |
Is there any way to reproduce this in SITL? |
|
I dont know exactly. That code should still execute regardless. I'll test a bit later |
It works fine in SITL when configured normally, we are unsure how to simulate a cold GPS fix with SITL. |
My only suspicion is that time is frozen on the FC, and the loops don't advance on device without GPS. I worked with CubePilot last night on setting up the debugger to reproduce locally on my Cube Orange but ran out of time. I've seen it in eProsima/Micro-XRCE-DDS-Client#350 |
Okay we will look into that, thankyou for putting your time into this issue! |
|
I've tried reproducing on 6X, and fail to get serial working. 6X has nothing connected except USB. This looks like a hardware-only regression. Sorry, we don't really have automated testing on hardware yet; and despite the current SITL testing that is done, it doesn't cover any of the ChibiOS stuff. Here's some info on what I've done so far. AND Result in autopilot: And, no apparent data in the micro ros agent: Also tested back on the following hashes and observed failure:
What's left to test. |
|
@Ryanf55 running tests on MatekH743-WING (v1) installed into a tilt-rotor VTOL. Vehicle on bench, has
Config and install: ./waf configure --board MatelH743 --enable-dds
./waf plane --uploadTTL-USB connectionSERIAL2_BAUD 57 # 57600
SERIAL2_PROTOCOL 2.0 # MAVLink2
SERIAL5_BAUD 57 # 57600
SERIAL5_PROTOCOL 45.0./install/micro_ros_agent/lib/micro_ros_agent/micro_ros_agent serial --dev /dev/cu.usbserial-4 --baudrate 57600 --verbose 6 --refs /Users/rhys/Code/ros2/humble/ros2-ardupilot/install/ardupilot_sitl/share/ardupilot_sitl/config/dds_xrce_profile.xmlConnection successful. SiK telem connectionFlip the roles of ./install/micro_ros_agent/lib/micro_ros_agent/micro_ros_agent serial --dev /dev/cu.usbserial-0001 --baudrate 57600 --verbose 6 --refs /Users/rhys/Code/ros2/humble/ros2-ardupilot/install/ardupilot_sitl/share/ardupilot_sitl/config/dds_xrce_profile.xmlcorrection: initial ping is ok, creation requests fail. AP: DDS: Init complete
AP: DDS: Participant session request failure
AP: DDS Client: Creation Requests failedI suspect there may be a timeout in the second case. Display ping outputDo you see the initial ping in a terminal for the 6x? Run the below instead of the micro-ROS agent (the line breaks after each ping request are mine): python -m serial.tools.miniterm /dev/cu.usbserial-0001 57600 --echo --encoding hexlify
--- Miniterm on /dev/cu.usbserial-0001 57600,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12
7E 00 00 10 00 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00 CF 12 Update
Confirmed for SiK radio serial connection: - constexpr uint8_t maxTimeMsPerRequestMs = 250;
+ constexpr uint16_t maxTimeMsPerRequestMs = 500;Solves the issue of the DDS participant creation failing when using a telemetry radio for the connection. |
|
I did some more testing. Despite having SERIAL1_PROTOCOL set to 45, I still see some GCS status message texts coming out the port. Even if DDS_ENABLE to 0, there is still data being shoved out the port.
Mavlink is hogging the port: |
|
Can you repeat the same test as Rhys and I?
Paste the output of 3. If it's what we expect, there should be 10 ping attemps. What I see is Mavlink stomping on the serial traffic and interleaving data which corrupts the stream. |
|
Thanks for all the work on this. Yep, I'll do this and get back to you in 2 hours or so. |
|
@KyleJewiss, if you have a TTL to USB serial adapter another test would be to use a different UART (not one of the ones usually assigned to Telem or GPS) for the DDS data. That may give you a temporary workaround while we determine what is causing the issue Ryan is observing with the USB assigned to SERIAL1. |
|
Tagged this for dev call to ask the team if there is any known reason that SERIAL1_PROTOCOL would be ignored. |
|
Alright, so I rebased to master and I tested both of the serial ports available on the CubeOrange, and when the here3+ gets a hot geofix there is NOTHING at all coming across the DDS Serial port. When it doesn't get hot geofix it does. We have figured out a SUCCESSFUL work around for now. We have added this 5 second delay in This seems to avoid the mavlink/transport conflict and I get data with GPS hotfix. One other thing I wanted to mention in case it is important is that we do not have radio connect to the CubeOrange as we do not need it for what we are doing. I don't think it is contributing to the issue but just in case. |
|
On CubeOrange, serial0 and serial6. The first and last are USB. Check the hwdef.daf (see otg and otg2). 2nd one defaults to SLCAN on Orange, but others default to mavlink. |
Sorry, what are you recommending we change the values to? |
|
Sorry, for more context, that was notes from today's dev call. One reason I couldn't get DDS working over serial was because ardupilot/libraries/AP_HAL_ChibiOS/hwdef/CubeOrange/hwdef.inc Lines 30 to 31 in 69bfe9b The defaults are MavLink on the second USB because it's configured so: ardupilot/libraries/AP_HAL_ChibiOS/hwdef/CubeOrange/defaults.parm Lines 10 to 11 in 69bfe9b Cube Orange+ inherits the hwconfig from CubeOrange: BUT, it then sets SERIAL5_PROTOCOL to mavlink ardupilot/libraries/AP_HAL_ChibiOS/hwdef/CubeOrangePlus/defaults.parm Lines 10 to 11 in 69bfe9b There is no USB serial available for DDS on CubeOrange+ as I was expecting. I was chasing the wrong issue. On Pixhawk6X, I use a different port, but I don't have the connector kit for that board yet (it's shipping since Saturday). I need to make a cable to test your exact setup. Can you upload your params file and I'll copy wiring from that? |
|
Thank you for that. Here is the parameter file. |
|
@KyleJewiss did you solve this? If not then please ping me on discord and we can look at it together |
USB is never assigned to SERIAL1 |
Bug report
AP_DDS: MicroROS agent doesn't publish topics when GPS automatically gets GPS lock.
We have Ardupilot running on a PX4 with a CubeOrange and a Here3+ GPS. We followed the documentation in the AP_DDS library to setup DDS/MicroROS: https://github.com/ArduPilot/ardupilot/tree/master/libraries/AP_DDS
We got it all working, where the data from the Here3+ could be published as topics to a ROS domain, however this is only the case when the Here3+ starts in a flashing blue "Disarmed with no GPS lock" state. When it is outside it automatically achieves GPS lock and flashes green "Disarmed (ready to arm), GPS lock acquired". See documentation here: https://docs.cubepilot.org/user-guides/here-3/here-3-manual
When the GPS starts without GPS lock the console output looks like the following:
When the GPS starts with GPS lock the console output looks stops as this point. It never publishes any of the topics:
Version
4.5 dev version
Platform
[ ] All
[ ] AntennaTracker
[ ] Copter
[ ] Plane
[ X ] Rover
[ ] Submarine
Hardware type
Pixhawk4 with CubeOrange and a Here3+
Logs
The text was updated successfully, but these errors were encountered: