Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HolisticLandmark clashes #60

Open
GeorgeS2019 opened this issue Dec 10, 2024 · 12 comments
Open

HolisticLandmark clashes #60

GeorgeS2019 opened this issue Dec 10, 2024 · 12 comments
Labels
bug Something isn't working platform:windows

Comments

@GeorgeS2019
Copy link

If draw_landmarks(landmarks) commented, it runs continuously.

func _result_callback(stream_name: String, packet: MediaPipePacket) -> void:
	print(stream_name)
	if stream_name == "image_in":
		var image_frame = MediaPipeImage.create_from_packet(packet)
		image = image_frame.get_image()
	elif stream_name in ["pose_landmarks", "face_landmarks", "left_hand_landmarks", "right_hand_landmarks"]:
		var proto := packet.get_proto("")
		var landmarks = proto.get("landmark")
		# draw_landmarks(landmarks)
	elif stream_name == "face_blendshapes":
		var proto = packet.get_proto("")
		var classifications = proto.get("classification")
		show_blendshapes(classifications)
E 0:00:54:0888   HolisticLandmarker.gd:81 @ show_blendshapes(): This function in this node (/root/VisionTask/VBoxContainer/Image/Blendshapes) can only be accessed from either the main thread or a thread group. Use call_deferred() instead.
  <C++ Error>    Condition "!is_readable_from_caller_thread()" is true. Returning: (false)
  <C++ Source>   scene/main/node.cpp:1278 @ can_auto_translate()
  <Stack Trace>  HolisticLandmarker.gd:81 @ show_blendshapes()
                 HolisticLandmarker.gd:22 @ _result_callback()

E 0:00:54:0889   HolisticLandmarker.gd:81 @ show_blendshapes(): Caller thread can't call this function in this node (/root/VisionTask/VBoxContainer/Image/Blendshapes). Use call_deferred() or call_thread_group() instead.
  <C++ Error>    Condition "!is_accessible_from_caller_thread()" is true.
  <C++ Source>   scene/main/canvas_item.cpp:423 @ queue_redraw()
  <Stack Trace>  HolisticLandmarker.gd:81 @ show_blendshapes()
                 HolisticLandmarker.gd:22 @ _result_callback()
				 
E 0:00:54:0890   HolisticLandmarker.gd:81 @ show_blendshapes(): This function in this node (/root/VisionTask/VBoxContainer/Image/Blendshapes) can only be accessed from the main thread. Use call_deferred() instead.
  <C++ Error>    Condition "is_inside_tree() && !is_current_thread_safe_for_nodes()" is true.
  <C++ Source>   scene/gui/control.cpp:1613 @ update_minimum_size()
  <Stack Trace>  HolisticLandmarker.gd:81 @ show_blendshapes()
                 HolisticLandmarker.gd:22 @ _result_callback()

E 0:00:54:0892   HolisticLandmarker.gd:81 @ show_blendshapes(): Caller thread can't call this function in this node (/root/VisionTask/VBoxContainer/Image/Blendshapes). Use call_deferred() or call_thread_group() instead.
  <C++ Error>    Condition "!is_accessible_from_caller_thread()" is true.
  <C++ Source>   scene/main/node.cpp:3311 @ update_configuration_warnings()
  <Stack Trace>  HolisticLandmarker.gd:81 @ show_blendshapes()
                 HolisticLandmarker.gd:22 @ _result_callback()
@j20001970
Copy link
Owner

j20001970 commented Dec 10, 2024 •
edited
Loading

Packet callbacks invoked by MediaPipeGraph are not executed during idle time in Godot, thus the error.

Demo project has updated to GDMP 28aaccc to make use of MediaPipeTaskRunner for holistic landmark task, that should fix the issue.

@GeorgeS2019
Copy link
Author 

CrashHandlerException: Program crashed with signal 11
Engine version: Godot Engine v4.3.stable.official (77dcf97d82cbfe4e4615475fa52ca03da645dbd8)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] error(-1): no debug info in PE/COFF executable
[2] error(-1): no debug info in PE/COFF executable
[3] error(-1): no debug info in PE/COFF executable
[4] error(-1): no debug info in PE/COFF executable
[5] error(-1): no debug info in PE/COFF executable
[6] error(-1): no debug info in PE/COFF executable
[7] error(-1): no debug info in PE/COFF executable
[8] error(-1): no debug info in PE/COFF executable
[9] error(-1): no debug info in PE/COFF executable
[10] error(-1): no debug info in PE/COFF executable
[11] error(-1): no debug info in PE/COFF executable
[12] error(-1): no debug info in PE/COFF executable
[13] error(-1): no debug info in PE/COFF executable
[14] error(-1): no debug info in PE/COFF executable
[15] error(-1): no debug info in PE/COFF executable
[16] error(-1): no debug info in PE/COFF executable
[17] error(-1): no debug info in PE/COFF executable
[18] error(-1): no debug info in PE/COFF executable
[19] error(-1): no debug info in PE/COFF executable
[20] error(-1): no debug info in PE/COFF executable
[21] error(-1): no debug info in PE/COFF executable
[22] error(-1): no debug info in PE/COFF executable
[23] error(-1): no debug info in PE/COFF executable
[24] error(-1): no debug info in PE/COFF executable
[25] error(-1): no debug info in PE/COFF executable
[26] error(-1): no debug info in PE/COFF executable
[27] error(-1): no debug info in PE/COFF executable
[28] error(-1): no debug info in PE/COFF executable
[29] error(-1): no debug info in PE/COFF executable
[30] error(-1): no debug info in PE/COFF executable
-- END OF BACKTRACE --
================================================================

@j20001970 j20001970 added bug Something isn't working platform:windows labels Dec 12, 2024
@j20001970
Copy link
Owner

Could you try narrowing down to which line in GDScript lead to the crash? No platform-specific code is used in MediaPipeTaskRunner from GDMP side, chances are that there are issues in mediapipe when running on Windows.

CrashHandlerException: Program crashed with signal 11
Engine version: Godot Engine v4.3.stable.official (77dcf97d82cbfe4e4615475fa52ca03da645dbd8)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] error(-1): no debug info in PE/COFF executable
[2] error(-1): no debug info in PE/COFF executable
[3] error(-1): no debug info in PE/COFF executable
[4] error(-1): no debug info in PE/COFF executable
[5] error(-1): no debug info in PE/COFF executable
[6] error(-1): no debug info in PE/COFF executable
[7] error(-1): no debug info in PE/COFF executable
[8] error(-1): no debug info in PE/COFF executable
[9] error(-1): no debug info in PE/COFF executable
[10] error(-1): no debug info in PE/COFF executable
[11] error(-1): no debug info in PE/COFF executable
[12] error(-1): no debug info in PE/COFF executable
[13] error(-1): no debug info in PE/COFF executable
[14] error(-1): no debug info in PE/COFF executable
[15] error(-1): no debug info in PE/COFF executable
[16] error(-1): no debug info in PE/COFF executable
[17] error(-1): no debug info in PE/COFF executable
[18] error(-1): no debug info in PE/COFF executable
[19] error(-1): no debug info in PE/COFF executable
[20] error(-1): no debug info in PE/COFF executable
[21] error(-1): no debug info in PE/COFF executable
[22] error(-1): no debug info in PE/COFF executable
[23] error(-1): no debug info in PE/COFF executable
[24] error(-1): no debug info in PE/COFF executable
[25] error(-1): no debug info in PE/COFF executable
[26] error(-1): no debug info in PE/COFF executable
[27] error(-1): no debug info in PE/COFF executable
[28] error(-1): no debug info in PE/COFF executable
[29] error(-1): no debug info in PE/COFF executable
[30] error(-1): no debug info in PE/COFF executable
-- END OF BACKTRACE --
================================================================

@GeorgeS2019
Copy link
Author

I think UI threads conflict. The App displays complete landmarks for 0.5 sec and clashes

@j20001970
Copy link
Owner

I think UI threads conflict. The App displays complete landmarks for 0.5 sec and clashes

Does performing single image inference multiple times also lead to the crash?

@GeorgeS2019
Copy link
Author

When I suppressed draw landmarks but only displaying in text the blendshapes, the app run continuously.

Single image inference multiple times before callback is independent of UI.
However, the sending of next frame image for GDectension multiple time inference must wait until all drawing of landmarks are completed . This is my guest.

@j20001970
Copy link
Owner

Does updating to mediapipe v0.10.20 (j20001970/GDMP-demo@d483c7a) solves your problem?

@GeorgeS2019
Copy link
Author

GeorgeS2019 commented Jan 1, 2025 •
edited
Loading

It works with image and loaded video.

Live Camera

@j20001970
crash again. Same error message as here
#60 (comment)

Not sure why.

I see a completely render face landmarks and post landmarks, however, after I frame, the app crash.

Using loaded video, you could see missing face landmarks in same frames. miss alignment of post landmarks.

Godot_v4 3-stable_win64_LDFpkCGnfn

@GeorgeS2019
Copy link
Author

This project may need to have way to check the stability of landmarks. Pose landmarks using dots may not be sufficient.

e.g use lines to link up these dots for both pose and hand landmarks

image

@GeorgeS2019
Copy link
Author

@srcnalt
srcnalt/rpm-godot-sdk#3
Godot Mediapipe holistic landmark is almost there

Perhaps time to look into it, especially Facebook Meta is putting strong interest to Godot. See Godot Cons 2024 Nov

@GeorgeS2019
Copy link
Author

GeorgeS2019 commented Jan 1, 2025 •
edited
Loading

@Malcolmnixon
GodotReadyPlayerMeAvatar

The hollisticlandmark is here
The motion matching is now GDextension
@GuilhermeGSousa
https://github.com/GuilhermeGSousa/godot-motion-matching-demo

@GeorgeS2019
Copy link
Author

google-ai-edge/mediapipe#3421 (comment)

@GeorgeS2019 GeorgeS2019 mentioned this issue Jan 5, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working platform:windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GeorgeS2019 @j20001970