Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

REQUIRED process [ouster/os_nodelet_mgr-2] has died! #365

Closed
retepik opened this issue Sep 10, 2024 · 14 comments
Closed

REQUIRED process [ouster/os_nodelet_mgr-2] has died! #365

retepik opened this issue Sep 10, 2024 · 14 comments
Assignees
@Samahu
Labels
bug Something isn't working

Comments

@retepik
Copy link

retepik commented Sep 10, 2024

Hi,
my configuration is :
Ubuntu 20.04
Ros : Noetic
Ouster OS-1 128 Rev C.
firmware: 2.5.3

after update to latest ouster-ros driver I get error

REQUIRED process [ouster/os_nodelet_mgr-2] has died! process has died [pid 43842, exit code -11, cmd /opt/ros/noetic/lib/nodelet/nodelet manager __name:=os_nodelet_mgr __log:=/home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2.log]. log file: /home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2*.log Initiating shutdown!

Strange think is that I get error only on Release build
catkin_make --cmake-args -DCMAKE_BUILD_TYPE=Release
when I use Debug, or RelWithDebInfo build type, all work.
I try it with fresh default installation of ROS and ouster-ros driver, but with same behavior

my command is :
roslaunch --master-logger-level='debug' -v /home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/sensor.launch sensor_hostname:=169.254.192.176 udp_dest:=192.168.113.195 lidar_port:=7502 imu_port:=7503 replay:=false lidar_mode:=512x10 metadata:=/home/mapper/metadata.json viz:=true image:=true udp_profile_lidar:=LEGACY

output :
`... logging to /home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/roslaunch-EOS-PC-01-43815.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

... loading XML file [/opt/ros/noetic/etc/ros/roscore.xml]
... executing command param [rosversion roslaunch]
Added parameter [/rosversion]
... executing command param [rosversion -d]
Added parameter [/rosdistro]
Added core node of type [rosout/rosout] in namespace [/]
... loading XML file [/home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/sensor.launch]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
Added parameter [/ouster/os_node/sensor_hostname]
Added parameter [/ouster/os_node/udp_dest]
Added parameter [/ouster/os_node/lidar_port]
Added parameter [/ouster/os_node/imu_port]
Added parameter [/ouster/os_node/udp_profile_lidar]
Added parameter [/ouster/os_node/lidar_mode]
Added parameter [/ouster/os_node/timestamp_mode]
Added parameter [/ouster/os_node/metadata]
Added parameter [/ouster/os_node/azimuth_window_start]
Added parameter [/ouster/os_node/azimuth_window_end]
Added parameter [/ouster/os_node/persist_config]
Added parameter [/ouster/os_node/attempt_reconnect]
Added parameter [/ouster/os_node/dormant_period_between_reconnects]
Added parameter [/ouster/os_node/max_failed_reconnect_attempts]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
... loading XML file [/home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/common.launch]
Added parameter [/ouster/os_cloud_node/tf_prefix]
Added parameter [/ouster/os_cloud_node/sensor_frame]
Added parameter [/ouster/os_cloud_node/lidar_frame]
Added parameter [/ouster/os_cloud_node/imu_frame]
Added parameter [/ouster/os_cloud_node/point_cloud_frame]
Added parameter [/ouster/os_cloud_node/timestamp_mode]
Added parameter [/ouster/os_cloud_node/dynamic_transforms_broadcast]
Added parameter [/ouster/os_cloud_node/dynamic_transforms_broadcast_rate]
Added parameter [/ouster/os_cloud_node/proc_mask]
Added parameter [/ouster/os_cloud_node/scan_ring]
Added parameter [/ouster/os_cloud_node/ptp_utc_tai_offset]
Added parameter [/ouster/os_cloud_node/point_type]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
Added parameter [/ouster/img_node/proc_mask]
Added node of type [nodelet/nodelet] in namespace [/ouster/]
Added node of type [rviz/rviz] in namespace [/]
... done importing include file [/home/mapper/Programs/ouster/ouster-ros/src/ouster-ros/launch/common.launch]
started roslaunch server http://EOS-PC-01:35957/

SUMMARY

PARAMETERS

  • /ouster/img_node/proc_mask: IMG|PCL|IMU|SCAN
  • /ouster/os_cloud_node/dynamic_transforms_broadcast: False
  • /ouster/os_cloud_node/dynamic_transforms_broadcast_rate: 1.0
  • /ouster/os_cloud_node/imu_frame: os_imu
  • /ouster/os_cloud_node/lidar_frame: os_lidar
  • /ouster/os_cloud_node/point_cloud_frame:
  • /ouster/os_cloud_node/point_type: original
  • /ouster/os_cloud_node/proc_mask: IMG|PCL|IMU|SCAN
  • /ouster/os_cloud_node/ptp_utc_tai_offset: -37.0
  • /ouster/os_cloud_node/scan_ring: 0
  • /ouster/os_cloud_node/sensor_frame: os_sensor
  • /ouster/os_cloud_node/tf_prefix:
  • /ouster/os_cloud_node/timestamp_mode:
  • /ouster/os_node/attempt_reconnect: False
  • /ouster/os_node/azimuth_window_end: 360000
  • /ouster/os_node/azimuth_window_start: 0
  • /ouster/os_node/dormant_period_between_reconnects: 1.0
  • /ouster/os_node/imu_port: 7503
  • /ouster/os_node/lidar_mode: 512x10
  • /ouster/os_node/lidar_port: 7502
  • /ouster/os_node/max_failed_reconnect_attempts: 2147483647
  • /ouster/os_node/metadata: /home/mapper/meta...
  • /ouster/os_node/persist_config: False
  • /ouster/os_node/sensor_hostname: 169.254.192.176
  • /ouster/os_node/timestamp_mode:
  • /ouster/os_node/udp_dest: 192.168.113.195
  • /ouster/os_node/udp_profile_lidar: LEGACY
  • /rosdistro: noetic
  • /rosversion: 1.16.0

NODES
/
rviz (rviz/rviz)
/ouster/
img_node (nodelet/nodelet)
os_cloud_node (nodelet/nodelet)
os_node (nodelet/nodelet)
os_nodelet_mgr (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [43825]
ROS_MASTER_URI=http://localhost:11311

setting run_id to 3eee6cfc-6f8b-11ef-b8ff-4de79e617463
process[rosout-1]: started with pid [43835]
started core service [/rosout]
process[ouster/os_nodelet_mgr-2]: started with pid [43842]
process[ouster/os_node-3]: started with pid [43843]
process[ouster/os_cloud_node-4]: started with pid [43844]
process[ouster/img_node-5]: started with pid [43845]
process[rviz-6]: started with pid [43847]
[ INFO] [1725982932.289950072]: Loading nodelet /ouster/os_node of type ouster_ros/OusterSensor to manager os_nodelet_mgr with the following remappings:
[ INFO] [1725982932.293050262]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] has not been advertised, waiting...
[ INFO] [1725982932.294509733]: Loading nodelet /ouster/os_cloud_node of type ouster_ros/OusterCloud to manager os_nodelet_mgr with the following remappings:
[ INFO] [1725982932.298256232]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] has not been advertised, waiting...
[ INFO] [1725982932.299489196]: Loading nodelet /ouster/img_node of type ouster_ros/OusterImage to manager os_nodelet_mgr with the following remappings:
[ INFO] [1725982932.299622864]: Initializing nodelet with 8 worker threads.
[ INFO] [1725982932.314283501]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] is now available.
[ INFO] [1725982932.317125156]: OusterImage: node initialized!
[ INFO] [1725982932.319181422]: waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] is now available.
[ INFO] [1725982932.323722392]: reset service created
[ INFO] [1725982932.323961602]: get_metadata service created
[ INFO] [1725982932.324198765]: get_config service created
[ INFO] [1725982932.324426216]: set_config service created
[ INFO] [1725982932.326390974]: Will send UDP data to 192.168.113.195
[ INFO] [1725982932.326405420]: Contacting sensor 169.254.192.176 ...
[ INFO] [1725982932.434273591]: Sensor 169.254.192.176 was configured successfully
[ INFO] [1725982932.434348984]: Starting sensor 169.254.192.176 initialization... Using ports: 7502/7503
[2024-09-10 17:42:12.434] [ouster::sensor] [info] initializing sensor client: 169.254.192.176 expecting lidar port/imu port: 7502/7503 (0 means a random port will be chosen)
[ INFO] [1725982937.387807405]: rviz version 1.14.25
[ INFO] [1725982937.387842245]: compiled against Qt version 5.12.8
[ INFO] [1725982937.387848643]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1725982937.394058730]: Forcing OpenGl version 0.
[ INFO] [1725982937.504260036]: Stereo is NOT SUPPORTED
[ INFO] [1725982937.504309830]: OpenGL device: llvmpipe (LLVM 12.0.0, 256 bits)
[ INFO] [1725982937.504320998]: OpenGl version: 4.5 (GLSL 4.5) limited to GLSL 1.4 on Mesa system.
[2024-09-10 17:42:29.069] [ouster::sensor] [info] parsing non-legacy metadata format
[ INFO] [1725982949.071479386]: OusterImage: retrieved new sensor metadata!
[2024-09-10 17:42:29.072] [ouster::sensor] [info] parsing non-legacy metadata format
[ INFO] [1725982949.084645981]: Wrote sensor metadata to /home/mapper/metadata.json
[ INFO] [1725982949.084790939]: ouster client version: 0.10.0+2898060-release
product: OS-1-128, sn: 992038000083, firmware rev: v2.5.3
lidar mode: 512x10, lidar udp profile: LEGACY
[ INFO] [1725982949.128753576]: OusterCloud: nodelet created!
[ INFO] [1725982949.128811387]: OusterCloud: retrieved new sensor metadata!
[2024-09-10 17:42:29.129] [ouster::sensor] [info] parsing non-legacy metadata format
[ WARN] [1725982949.135778085]: point_cloud_frame is set to sensor_frame. If you need to reproject or raytrace the points then use the lidar_frame instead. Refer to: #33 for more details.
[ INFO] [1725982949.139339167]: OusterCloud: using static transforms broadcast
REQUIRED process [ouster/os_nodelet_mgr-2] has died!
process has died [pid 43842, exit code -11, cmd /opt/ros/noetic/lib/nodelet/nodelet manager __name:=os_nodelet_mgr __log:=/home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2.log].
log file: /home/mapper/.ros/log/3eee6cfc-6f8b-11ef-b8ff-4de79e617463/ouster-os_nodelet_mgr-2*.log
Initiating shutdown!

[rviz-6] killing on exit
[ouster/img_node-5] killing on exit
[ouster/os_node-3] killing on exit
[ouster/os_cloud_node-4] killing on exit
[ouster/os_nodelet_mgr-2] killing on exit
[ INFO] [1725982949.473568623]: Unloading nodelet /ouster/img_node from manager os_nodelet_mgr
[ INFO] [1725982949.473692499]: Unloading nodelet /ouster/os_node from manager os_nodelet_mgr
[ INFO] [1725982949.473811439]: Unloading nodelet /ouster/os_cloud_node from manager os_nodelet_mgr
[ INFO] [1725982949.476741537]: waitForService: Service [/ouster/os_nodelet_mgr/unload_nodelet] could not connect to host [EOS-PC-01:47311], waiting...
[ WARN] [1725982949.476782891]: Couldn't find service os_nodelet_mgr/unload_nodelet, perhaps the manager is already shut down
[ INFO] [1725982949.477072606]: waitForService: Service [/ouster/os_nodelet_mgr/unload_nodelet] could not connect to host [EOS-PC-01:47311], waiting...
[ WARN] [1725982949.477121257]: Couldn't find service os_nodelet_mgr/unload_nodelet, perhaps the manager is already shut down
[ INFO] [1725982949.477417235]: waitForService: Service [/ouster/os_nodelet_mgr/unload_nodelet] could not connect to host [EOS-PC-01:47311], waiting...
[ WARN] [1725982949.477476105]: Couldn't find service os_nodelet_mgr/unload_nodelet, perhaps the manager is already shut down
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done`

Could you help me with this bug?
Thanks
retepik
@retepik retepik added the bug Something isn't working label Sep 10, 2024
@Samahu
Copy link
Contributor

Samahu commented Sep 10, 2024 via email

@Samahu
Copy link
Contributor

Samahu commented Sep 10, 2024 via email

@retepik
Copy link
Author

retepik commented Sep 11, 2024

Hi Samahu,
I didn't make changes in original code, I used cloned code from git repositary

@yulan0215
Copy link

Hi Samahu, I didn't make changes in original code, I used cloned code from git repositary

Hi, did you have any solution for this? I have the same sensor and same problem, thx.

@yulan0215
Copy link

yulan0215 commented Oct 17, 2024 •
edited
Loading

I also used the very recently version ROS driver and the firmware is:
image
Can you please check if the problem was from firmware or IP?
The error is shown here:
image

image
Thanks for your help!

@Samahu
Copy link
Contributor

Samahu commented Oct 17, 2024

Here are few things to try that can help us narrow down the problem (the log isn't giving sufficient output unfortunately), try each step one by one:

  • First update to the latest version 0.13.1
  • Switch to driver.launch instead of the sensor.launch (it offers the same parameters).
  • Try leaving lidar_port and imu_port values set to zero so the driver would pick random free ports
  • Set proc_mask="IMG" alone or proc_mask="IMU" alone or even leave empty proc_mask="", this should help us find if the bug is a result of one processors.

Finally, you may need to enable GDB (but let's try this after the steps suggested above).

@Samahu
Copy link
Contributor

Samahu commented Oct 17, 2024 •
edited
Loading

Can you please check if the problem was from firmware or IP?
The error is shown here:

@yulan0215 your problem could be different, it is not clear how do you initialized the driver, a text log would be much clearer than an image.

@yulan0215
Copy link

Can you please check if the problem was from firmware or IP?

The error is shown here:

@yulan0215 your problem could be different, it is not clear how do you initialized the driver, a text log works would be much clearer than an image.

Many thanks for your quick reply. Although right now I could not change the Roslauch and I want to mention I also loss the data even there was no error or warning from terminal. I have to mention my ouster LiDAR directly connected to Jetson Xavier NX with Ethernet cable, do u have any idea for that? Many thx.

@Samahu
Copy link
Contributor

Samahu commented Oct 17, 2024

I want to mention I also loss the data even there was no error or warning from terminal. I have to mention my ouster LiDAR directly connected to Jetson Xavier NX with Ethernet cable, do u have any idea for that? Many thx.

This might be a network related issue, try applying the recommend network settings described in https://github.com/ouster-lidar/ouster-ros/blob/master/util/network-configure.bash

@yulan0215

This comment was marked as off-topic.

Sign in to view
@Samahu Samahu self-assigned this Oct 21, 2024
@Samahu Samahu closed this as completed Oct 21, 2024
@retepik
Copy link
Author

retepik commented Nov 6, 2024

I have still issue with this bug. I tested it on another PC with Ubuntu 20, ros noetic, with last git clone, with recommend network setting (link), with driver.launch, without setting imu and lidar port, but I have still same issue. I found that I have got issue on Release and RelWithDebInfo when proc_mask contains PCL, another combination proc_mask without PCL work. In Debug work all combination proc_mask.
I found also change ouput in console :
in Release : waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] has not been advertised, waiting...
in Debug: : waitForService: Service [/ouster/os_nodelet_mgr/load_nodelet] could not connect to host

Do you have any idea, how can I fix it? It's frustrating.
Thanks

@Samahu
Copy link
Contributor

Samahu commented Nov 6, 2024

I never tried tested the driver performance on NVIDIA NX, so it could be below the threshold that the driver with the default PointCloud size could run on, try using a lower lidar mode such as the 128x512.

@retepik
Copy link
Author

retepik commented Nov 7, 2024

I do not use NVIDIA NX, I have standard desktop with AMD Ryzen 9 processor. I tested all lidar mode, but still same result.

@Samahu
Copy link
Contributor

Samahu commented Nov 7, 2024

Please contact Ouster Support Staff so they can look closely at the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Samahu Samahu

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Samahu @retepik @yulan0215