Multiple Issues after Humble Upgrade (teleop, TF tree, cameras, time)

[teufelweich]

Please provide the following information:

• OS: Ubuntu 22.04
• ROS Distro: Humble
• Built from source or installed: Installed following the instructions that were present in the TurtelBot4 - Basic Setup
• Package version: ros-humble-turtlebot4-setup 1.0.2-1jammy.20230302.131753

I've upgraded one of my Turtlebots to Humble and experience the following issues currently.
I'm using the Discovery Setup and Namespaces.

There are several Issues:

1. (always) I am not able to drive the Robot using ros2 run teleop_twist_keyboard teleop_twist_keyboard namespace:=/tb2 , the Robot does not react at all

2. (always) the camera-topic does not provide any messages

3. (sometimes) after launching turtlebot4_viz view_model.launch.py namespace:=/tb2I can see in the GUI that the bumper is floating (sometimes) half a meter in front of the robot

4. (sometimes) after launching turtlebot4_viz view_model.launch.py namespace:=/tb2there are no laserscans visible

5. (sometimes) the tf trees for both wheels are missing (No transform from [left_wheel] to [base_link])

6. (always) the turtlebot4_diagnostics reports following errors/warnings:

   • Full Name: /Turtlebot4/Create3/IMU/turtlebot4_diagnostics: imu topic status
     Component: turtlebot4_diagnostics: imu topic status
     Hardware ID: Turtlebot4
     Level: WARNING
     Message: Frequency too high.

   • Full Name: /Turtlebot4/Camera/turtlebot4_diagnostics: color preview image topic status
     Component: turtlebot4_diagnostics: color preview image topic status
     Hardware ID: Turtlebot4
     Level: ERROR
     Message: No events recorded.

   • Full Name: /Turtlebot4/Camera/turtlebot4_diagnostics: stereo depth topic status
     Component: turtlebot4_diagnostics: stereo depth topic status
     Hardware ID: Turtlebot4
     Level: ERROR
     Message: No events recorded.

7. (always) LIDAR auto standby is not working

8. (always) The Create 3 has the wrong time and needs manual restart of the ntpd from the webinterface.

Service Status

ubuntu@ubuntu:~$ sudo service turtlebot4 status
● turtlebot4.service - "bringup turtlebot4"
     Loaded: loaded (/lib/systemd/system/turtlebot4.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-02 21:42:43 UTC; 31min ago
   Main PID: 750 (turtlebot4-star)
      Tasks: 148 (limit: 4416)
     Memory: 368.7M
        CPU: 29min 48.984s
     CGroup: /system.slice/turtlebot4.service
             ├─ 750 /bin/bash /usr/sbin/turtlebot4-start
             ├─ 992 /usr/bin/python3 /opt/ros/humble/bin/ros2 launch /tmp/turtlebot4.launch.py
             ├─1148 /opt/ros/humble/lib/turtlebot4_node/turtlebot4_node --ros-args -r __ns:=/tb2 --params-file /tmp/tmp0z9qijan --params-file /tmp/launch_params_3vt05gle
             ├─1150 /opt/ros/humble/lib/turtlebot4_base/turtlebot4_base_node --ros-args -r __ns:=/tb2 --params-file /tmp/tmp0z9qijan
             ├─1152 /opt/ros/humble/lib/joy_linux/joy_linux_node --ros-args -r __node:=joy_linux_node -r __ns:=/tb2 -r /diagnostics:=diagnostics
             ├─1154 /opt/ros/humble/lib/teleop_twist_joy/teleop_node --ros-args -r __node:=teleop_twist_joy_node -r __ns:=/tb2 --params-file /tmp/tmpcr84ae75
             ├─1156 /opt/ros/humble/lib/rplidar_ros/rplidar_composition --ros-args -r __node:=rplidar_composition -r __ns:=/tb2 --params-file /tmp/launch_params_ofgwptls
             ├─1158 /opt/ros/humble/lib/rclcpp_components/component_container --ros-args -r __node:=oakd_container -r __ns:=/tb2
             ├─1160 /opt/ros/humble/lib/robot_state_publisher/robot_state_publisher --ros-args -r __node:=robot_state_publisher -r __ns:=/tb2 --params-file /tmp/launch_params_khloukkf --params-file /tmp/launch_p>
             ├─1162 /opt/ros/humble/lib/tf2_ros/static_transform_publisher --x 0.0 --y 0.1165 --z 0.0402 --roll -1.5707 --pitch 0.0 --yaw 0.0 --frame-id base_link --child-frame-id wheel_drop_left --ros-args -r _>
             ├─1164 /opt/ros/humble/lib/tf2_ros/static_transform_publisher --x 0.0 --y -0.1165 --z 0.0402 --roll -1.5707 --pitch 0.0 --yaw 0.0 --frame-id base_link --child-frame-id wheel_drop_right --ros-args -r>
             ├─1166 /opt/ros/humble/lib/diagnostic_aggregator/aggregator_node --ros-args -r __ns:=/tb2 --params-file /tmp/tmpw9kncsk7 -r /diagnostics:=diagnostics -r /diagnostics_agg:=diagnostics_agg -r /diagnos>
             └─1168 /usr/bin/python3 /opt/ros/humble/lib/turtlebot4_diagnostics/diagnostics_updater --ros-args -r __ns:=/tb2 -r /diagnostics:=diagnostics -r /diagnostics_agg:=diagnostics_agg -r /diagnostics_topl>

Mar 02 21:51:39 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793899.572860093] [tb2.turtlebot4_node]:
Mar 02 21:51:42 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793902.432227347] [tb2.turtlebot4_node]: OAKD stopped
Mar 02 21:51:42 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793902.432475327] [tb2.turtlebot4_node]: RPLIDAR stopped
Mar 02 21:51:43 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793903.432626085] [tb2.turtlebot4_node]: oakd/stop_camera service available, sending request
Mar 02 21:51:43 ubuntu turtlebot4-start[992]: [component_container-6] [INFO] [1677793903.433418375] [tb2.oakd]: Stopping camera.
Mar 02 21:51:43 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793903.433610021] [tb2.turtlebot4_node]: stop_motor service available, sending request
Mar 02 21:51:43 ubuntu turtlebot4-start[992]: [rplidar_composition-5] [INFO] [1677793903.621819677] [tb2.rplidar_composition]: Ingnoring stop_motor request because rplidar_node is in 'auto standby' mode
Mar 02 21:51:43 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793903.623292517] [tb2.turtlebot4_node]: stop_motor service completed.
Mar 02 21:51:45 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793905.377228299] [tb2.turtlebot4_node]: oakd/stop_camera service got results: Success
Mar 02 21:51:45 ubuntu turtlebot4-start[992]: [turtlebot4_node-1] [INFO] [1677793905.379607427] [tb2.turtlebot4_node]:

topic list

$ ros2 topic list
/diagnostics
/oakd/rgb/preview/camera_info
/parameter_events
/rosout
/tb2/battery_state
/tb2/behavior_server/transition_event
/tb2/bt_navigator/transition_event
/tb2/clicked_point
/tb2/cmd_audio
/tb2/cmd_lightring
/tb2/cmd_vel
/tb2/cmd_vel_nav
/tb2/color/preview/image
/tb2/controller_server/transition_event
/tb2/cost_cloud
/tb2/diagnostics
/tb2/diagnostics_agg
/tb2/diagnostics_toplevel_state
/tb2/dock_status
/tb2/evaluation
/tb2/function_calls
/tb2/global_costmap/global_costmap/transition_event
/tb2/goal_pose
/tb2/hazard_detection
/tb2/hmi/buttons
/tb2/hmi/display
/tb2/hmi/display/message
/tb2/hmi/led
/tb2/imu
/tb2/initialpose
/tb2/interface_buttons
/tb2/ip
/tb2/ir_intensity
/tb2/ir_opcode
/tb2/joint_states
/tb2/joy
/tb2/joy/set_feedback
/tb2/kidnap_status
/tb2/local_costmap/clearing_endpoints
/tb2/local_costmap/costmap
/tb2/local_costmap/costmap_raw
/tb2/local_costmap/costmap_updates
/tb2/local_costmap/footprint
/tb2/local_costmap/local_costmap/transition_event
/tb2/local_costmap/published_footprint
/tb2/local_costmap/voxel_grid
/tb2/local_plan
/tb2/marker
/tb2/mobility_monitor/transition_event
/tb2/mouse
/tb2/oakd/rgb/preview/camera_info
/tb2/oakd/rgb/preview/image_raw
/tb2/oakd/rgb/preview/image_raw/compressed
/tb2/oakd/rgb/preview/image_raw/compressedDepth
/tb2/oakd/rgb/preview/image_raw/theora
/tb2/odom
/tb2/planner_server/transition_event
/tb2/received_global_plan
/tb2/robot_description
/tb2/robot_state/transition_event
/tb2/scan
/tb2/slip_status
/tb2/smoother_server/transition_event
/tb2/speed_limit
/tb2/static_transform/transition_event
/tb2/stereo/depth
/tb2/stop_status
/tb2/tf
/tb2/tf_static
/tb2/transformed_global_plan
/tb2/velocity_smoother/transition_event
/tb2/waypoint_follower/transition_event
/tb2/wheel_status
/tb2/wheel_ticks
/tb2/wheel_vels

When running $ ros2 run teleop_twist_keyboard teleop_twist_keyboard namespace:=/tb2 I get this warning:
[WARN] [1680685917.521080858] [rcl]: Found remap rule 'namespace:=/tb2'. This syntax is deprecated. Use '--ros-args --remap namespace:=/tb2' instead.
Using the proposed syntax of the warning doesn't work either. Is using --ros-args --remap namespace:=/tb2 the proper way for teleop?

[roni-kreinin]

1. For teleop you will need to call it like this: ros2 run teleop_twist_keyboard teleop_twist_keyboard --ros-args -r __ns:=/tb2.
2. If the robot is docked, the camera will be turned off by default due to the new power saver mode. Check that the camera turns on when the robot undocks.
3. Run sudo apt update && sudo apt upgrade to update your packages. This issue was fixed here.
4. Probably related to 2.
5. Fixed in same PR as 3.
6. Camera issue is same as 2. IMU diagnostics need to be updated to fix this warning, but it can be ignored for now.
7. If diagnostics are enabled, there will always be a subscriber to the lidar, so it will always be on. You can disable diagnostics using the turtlebot4-setup tool.
8. Try running turtlebot4-ntpd-sync from the RPi.

[teufelweich]

Thanks! This solved most of the issues, but some still persist:

2. For the camera: we only get an image using the topic '/tb2/oakd/rgb/preview/image_raw' (for '/tb2/color/preview/image' and '/tb2/stereo/depth' we do not get any signal or error in turtlebot4_viz, but in diagnostics we get for both topics the error: no events recorded). When using ROS2 installed inside a VM we have the option to choose between 3 different topics for the camera: /tb2/oakd/rgb/preview/image_raw, /tb2/color/preview/image and /tb2/stereo/depth. But only image_raw is working. On a Linux Notebook we only have the topic image_raw which works fine.
3. The bumper is still floating in front of the robot sometimes.

For 3. we tried looking into the TF tree with ros2 run tf2_tools view_frames --ros-args -r __ns:=/tb2 but found nothing obvious. (see PDF) frames_2023-04-17_11.50.11.pdf

[teufelweich]

When trying to SLAM we get some weird behavior, depending on where (VM or native Linux) we run the commands:

Please provide the following information:

• OS: Mint 21 / VM with Ubuntu 22.04 under Windows 11
• ROS Distro: both Humble
• Built from source or installed: installed
• Package version: 1.0.1-1jammy.20230327.055048

Expected behaviour
We were trying to SLAM according to this tutorial.
Therefor we first used a VM (Ubuntu 22.04 under Windows) it worked fine. Then we also tried to slam using 2 different Notebooks with native Mint 21, we were not able to slam, using the same robot with the same configs.

Actual behaviour
Using the VM we managed to do this, the tf-Tree had no errors and the map was visible as described in the tutorial.
Using the native Linux-Mint Notebook slam does not work at all: we are unable to generate a map. We always get the error:

student@lenbookTP1:~$ ros2 launch turtlebot4_navigation slam.launch.py sync:=false
[INFO] [launch]: All log files can be found below /local/student/.ros/log/2023-05-02-09-07-03-008899-lenbookTP1-5166
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [async_slam_toolbox_node-1]: process started with pid [5167]
[async_slam_toolbox_node-1] [INFO] [1683011223.130218648] [slam_toolbox]: Node using stack size 40000000
[async_slam_toolbox_node-1] [INFO] [1683011223.148948062] [slam_toolbox]: Using solver plugin solver_plugins::CeresSolver
[async_slam_toolbox_node-1] [INFO] [1683011223.149022371] [slam_toolbox]: CeresSolver: Using SCHUR_JACOBI preconditioner.
[async_slam_toolbox_node-1] [INFO] [1683011226.488932534] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1681817204.986 for reason 'discarding message because the queue is full'
[async_slam_toolbox_node-1] [INFO] [1683011226.635250109] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1681817205.135 for reason 'discarding message because the queue is full'
[async_slam_toolbox_node-1] [INFO] [1683011226.779429496] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1681817205.281 for reason 'discarding message because the queue is full'

And as soon as we start the turtlebot4_viz from the Mint-Notebook we get an error in the tf-tree:

No transform from [right|left_wheel] to [map|baselink|...selectetFrame...]

The VM and native Machine influence each other, depending on where which command is run.
Everything is fine if we start slam from the VM and then open turtlebot4_viz in the VM.
Then we open turtlebot4_viz on the Mint-Notebook and the map is visible on both devices but only on the VM the robot is visible, too.
After we start turtlebot4_viz on the Mint-Notebook both TF-Trees will show the error written above. (before in the VM there was no error)

[roni-kreinin]

Thanks! This solved most of the issues, but some still persist:

2. For the camera: we only get an image using the topic '/tb2/oakd/rgb/preview/image_raw' (for '/tb2/color/preview/image' and '/tb2/stereo/depth' we do not get any signal or error in turtlebot4_viz, but in diagnostics we get for both topics the error: no events recorded). When using ROS2 installed inside a VM we have the option to choose between 3 different topics for the camera: /tb2/oakd/rgb/preview/image_raw, /tb2/color/preview/image and /tb2/stereo/depth. But only image_raw is working. On a Linux Notebook we only have the topic image_raw which works fine.
3. The bumper is still floating in front of the robot sometimes.

For 3. we tried looking into the TF tree with ros2 run tf2_tools view_frames --ros-args -r __ns:=/tb2 but found nothing obvious. (see PDF) frames_2023-04-17_11.50.11.pdf

The offset bumper issue is fixed in the source package and should be released soon.

As for the issue with the Mint machines, can you check that ros2 topic list is displaying all of the same topics as it does on the VM? It sounds like you may not be seeing the Create 3 data from the Mint machines.

[Fran-FC]

Thanks! This solved most of the issues, but some still persist:

2. For the camera: we only get an image using the topic '/tb2/oakd/rgb/preview/image_raw' (for '/tb2/color/preview/image' and '/tb2/stereo/depth' we do not get any signal or error in turtlebot4_viz, but in diagnostics we get for both topics the error: no events recorded). When using ROS2 installed inside a VM we have the option to choose between 3 different topics for the camera: /tb2/oakd/rgb/preview/image_raw, /tb2/color/preview/image and /tb2/stereo/depth. But only image_raw is working. On a Linux Notebook we only have the topic image_raw which works fine.

3. The bumper is still floating in front of the robot sometimes.

For 3. we tried looking into the TF tree with ros2 run tf2_tools view_frames --ros-args -r __ns:=/tb2 but found nothing obvious. (see PDF) frames_2023-04-17_11.50.11.pdf

The point 2. (stereo camera) happens to me also, I stereo depth camera is unusable and I cannot read anything except the raw image.

[teufelweich]

Thanks! This solved most of the issues, but some still persist:

2. For the camera: we only get an image using the topic '/tb2/oakd/rgb/preview/image_raw' (for '/tb2/color/preview/image' and '/tb2/stereo/depth' we do not get any signal or error in turtlebot4_viz, but in diagnostics we get for both topics the error: no events recorded). When using ROS2 installed inside a VM we have the option to choose between 3 different topics for the camera: /tb2/oakd/rgb/preview/image_raw, /tb2/color/preview/image and /tb2/stereo/depth. But only image_raw is working. On a Linux Notebook we only have the topic image_raw which works fine.
3. The bumper is still floating in front of the robot sometimes.

For 3. we tried looking into the TF tree with ros2 run tf2_tools view_frames --ros-args -r __ns:=/tb2 but found nothing obvious. (see PDF) frames_2023-04-17_11.50.11.pdf

The offset bumper issue is fixed in the source package and should be released soon.

Great News!

As for the issue with the Mint machines, can you check that ros2 topic list is displaying all of the same topics as it does on the VM? It sounds like you may not be seeing the Create 3 data from the Mint machines.

The topic lists are identical.

~$ ros2 topic list
/battery_state
/cmd_audio
/cmd_lightring
/cmd_vel
/diagnostics
/dock_status
/function_calls
/hazard_detection
/hmi/buttons
/hmi/display
/hmi/display/message
/hmi/led
/imu
/interface_buttons
/ip
/ir_intensity
/ir_opcode
/joint_states
/joy
/joy/set_feedback
/kidnap_status
/mobility_monitor/transition_event
/mouse
/oakd/rgb/preview/camera_info
/oakd/rgb/preview/image_raw
/oakd/rgb/preview/image_raw/compressed
/oakd/rgb/preview/image_raw/compressedDepth
/oakd/rgb/preview/image_raw/theora
/odom
/parameter_events
/robot_description
/robot_state/transition_event
/rosout
/scan
/slip_status
/static_transform/transition_event
/stop_status
/tf
/tf_static
/wheel_status
/wheel_ticks
/wheel_vels

inside the VM we can SLAM with these topics, on the native machine SLAM doesn't work

[teufelweich]

Reading several other issues like #181 and #188 we think we have kind of the same problem.
Inside of the VM (where SLAM works) we get messages in the /odom topic and on the native machines not.
When receiving messages they have the correct timestamps, but SLAM did also work when the timestamps where incorrect.

That's really frustrating.

[Fingerling42]

Hi @teufelweich.

Did you manage to solve the problem with the error no events recorded in /color/preview/image and /stereo/depth? I ran into exactly the same problem, only my robot already has Humble pre-installed right away.

[teufelweich]

SLAM is still not working for Native MINT and still working in VM (so nothing changed).
We also tried MINT as VM which worked fine and we tried Ubuntu native which did not work -> it seems to work independently of OS but only in a VM. Natively it does NOT work.
As already mentioned ros2 topic list and ros2 service list are identical on Mint, VM and Pi.
BUT ros2 node list is different:

Mint:

student@lenbookTP1:~$ ros2 node list
/joint_state_publisher
/joy_linux_node
/launch_ros_1102
/oakd
/oakd_container
/robot_state_publisher
/rplidar_composition
/rqt_gui_py_node_21442
/rqt_gui_py_node_3643
/teleop_twist_joy_node
/turtlebot4_base_node
/turtlebot4_node

VM (Ubuntu) and Pi:

ubuntu@ubuntu:~$ ros2 node list
/_internal/composite_hazard
/_internal/kinematics_engine
/_internal/mobility
/_internal/stasis
/joint_state_publisher
/joy_linux_node
/launch_ros_1102
/mobility_monitor
/motion_control
/oakd
/oakd_container
/robot_state
/robot_state_publisher
/rplidar_composition
/rqt_gui_py_node_21442
/rqt_gui_py_node_3643
/static_transform
/system_health
/teleop_twist_joy_node
/turtlebot4_base_node
/turtlebot4_node
/ui_mgr

missing:

/_internal/composite_hazard
/_internal/kinematics_engine
/_internal/mobility
/_internal/stasis
/mobility_monitor
/motion_control
/robot_state
/static_transform
/system_health
/ui_mgr

according to this lesson there should be even more internal topics visible on the pie:

/_internal/battery
/_internal/floor_type
/_internal/imu
/_internal/ir_codes
/_internal/ir_intensity
/_internal/mouse

Also the rqt_graph shows a lot less in Mint, probably because many nodes are missing...

Because of missing node /robot_state we're not getting any data from topics:

• /battery_state
• /tf
• /wheel_status
• /wheel_ticks
• /wheel_vels

That could be a reason why we can't use SLAM

That's what we get in MINT:

That's how it looks from Pi:

Info about /robot_state node (from Pi):

@Fingerling42: we did not work on this issue, as we are still unable to SLAM.

[hilary-luo]

@teufelweich I have been going through and checking on old tickets. I see that there has been recent activity here. Are you looking for support on this ticket still? - If so do you mind giving me a summary of where you are at now and what the outstanding issues are?

@Fingerling42 Do you also have ongoing issues?

[Fingerling42]

@hilary-luo
Hi. Yes, the problem is with stereo camera. In theory, it should publish the image to the /color/preview/image and /stereo/depth topics (at least that’s what it says in the manual). But it only publishes a low-quality image to /oakd/rgb/preview/image_raw (meaning the camera is working).

I tried to run the robot diagnostics, and for both topics it gives the problem Level: ERROR, Message: No events recorded

[hilary-luo]

@Fingerling42 are you running galactic? Can you confirm the package version that you are running for the turtlebot4-robot package?

[Fingerling42]

@hilary-luo It is Humble, it was pre-installed.

[hilary-luo]

@Fingerling42 can you please run sudo apt update and sudo apt upgrade on your robot to update all of the packages to the latest and then reboot and then let me know if the issues still persists.

[teufelweich]

@Fingerling42 can you please run sudo apt update and sudo apt upgrade on your robot to update all of the packages to the latest and then reboot and then let me know if the issues still persists.

We updated the packages on the PI and on the native machine and our issues still persist. We don't see the node /robot_state and thus can not SLAM or Navigate

[hilary-luo]

@teufelweich are you working with @Fingerling42 on the same robot? if not, my last question to you was the following:

@teufelweich I have been going through and checking on old tickets. I see that there has been recent activity here. Are you looking for support on this ticket still? - If so do you mind giving me a summary of where you are at now and what the outstanding issues are?

The quoted text that you responded to was for @Fingerling42 for the issue he is having with his camera. If you (@teufelweich) are interested in resuming troubleshooting on this ticket then please provide a summary of where you are at now and what the outstanding core issues are. I ask this particularly because I was not part of the original troubleshooting. Am I correct in saying that you have a robot on Humble running the latest create3 firmware and primarily you are trying unsuccessfully to run SLAM / Navigation?

[hilary-luo]

This issue is being closed due to inactivity. If you are still experiencing the issue, feel free to reopen this ticket when you are ready to continue the troubleshooting process.