Cannot run examples on macOS 10.15 "Illegal instruction: 4"

[lejar]

I tried running the examples (using the pip release of cefpanda) on macOS 10.15 and I am getting the error "Illegal instruction: 4". I made a minimal script with just cefpython and panda3d and found the following weirdness:

If you initialize cefpython before panda3d, then panda3d only gets mouse move events when the mouse buttons are held down. If you initialize cefpython after panda3d, then you get the illegal instruction error when running cefpanda.MessageLoopWork().

Here's the minimal example with cefpython being initialized before panda3d:

from cefpython3 import cefpython

from direct.showbase.ShowBase import ShowBase
from direct.task import Task


class MyApp(ShowBase):
    def __init__(self):
        super().__init__()
        self.disableMouse()

        self.updateTask = self.taskMgr.add(self.updateTask, 'updateTask')

    def updateTask(self, task):
        if self.mouseWatcherNode.hasMouse():
            x = self.mouseWatcherNode.getMouseX()
            y = self.mouseWatcherNode.getMouseY()
            print(self.mouseWatcherNode, x, y)
        cefpython.MessageLoopWork()
        return Task.cont


def main():
    settings = {
        "windowless_rendering_enabled": True,
    }

    switches = {
        "disable-gpu": "",
        "disable-gpu-compositing": "",
        "enable-begin-frame-scheduling": "",
    }

    cefpython.Initialize(settings=settings, switches=switches)
    app = MyApp()
    app.run()
    cefpython.Shutdown()


if __name__ == '__main__':
    main()

[el-dee]

"Illegal instruction: 4" also happens on Mojave (10.14.6), also I had to comment out

"resources_dir_path": cefpython.GetModuleDirectory()

Otherwise I got the following errors:

[1106/103607.503450:ERROR:main_delegate.cc(717)] Could not load cef.pak
[1106/103607.503801:ERROR:main_delegate.cc(734)] Could not load cef_100_percent.pak
[1106/103607.503842:ERROR:main_delegate.cc(743)] Could not load cef_200_percent.pak
[1106/103607.503869:ERROR:main_delegate.cc(753)] Could not load cef_extensions.pak
[1106/103607.505199:ERROR:content_client.cc(272)] No data resource available for id 191

So I think the root cause is that CEF Python loads extensions from another installation (probably system wide as I have Chrome installed) and can't load them due to incompatible SDK version used (that's the actual meaning of Illegal Instruction: 4)

[el-dee]

Ok, I could make it work with Panda with the following modifications :

in app_settings:

 "resources_dir_path": "",

Otherwise cef can't load the .pak files.

Split up CEFPanda.init into two parts :

first part is called before Panda3D init and ends with cefpython.Initialize(app_settings, command_line_settings)

Second part is called after ShowBase.init(self) and contains the remaining code of the former init()

Third, line :

self._set_browser_size()

is replaced by :

self._set_browser_size(base.win)

otherwise the window is not painted until one moves the window.

[Moguri]

Thanks for looking into this!

I am still confused on why parts of CEF need to be initialized before Panda3D. Is this a CEF bug? Something strange that Panda is doing? I would prefer to not split up the init if possible since it creates a slightly more complicated API (especially if there are restrictions on when the inits can be called).

For setting resources_dir_path, does "" work for all platforms, or do we need to do some branching? It looks like the PySDL2 example doesn't even bother setting any of the _path options, so maybe they can just be removed?

For the self._set_browser_size(), yes, that should be changed. That call doesn't even do anything as it is currently written, so I would call it a bug.

[el-dee]

I am still confused on why parts of CEF need to be initialized before Panda3D. Is this a CEF bug? Something strange that Panda is doing? I would prefer to not split up the init if possible since it creates a slightly more complicated API (especially if there are restrictions on when the inits can be called).

I don't know yet, I only discovered that by trial and error :). Though, looking at the cefpython examples, many call cef.Initialize() before initializing the graphic engine (at least for gtk-x and wxWidgets).

What you could do is create the texture and card only in a lazy initializer called by the load_xxx() method to keep the differed initialization hidden.

For setting resources_dir_path, does "" work for all platforms, or do we need to do some branching? It looks like the PySDL2 example doesn't even bother setting any of the _path options, so maybe they can just be removed?

So far all the examples I have seen do not configure any of the resources path, on the other hand I haven't tested yet to bundle cef in a distributed panda app, maybe that's necessary when the resources are bundled in an app

[el-dee]

Small follow up, I was wondering why in my app I did not have the crash on Mac with the default cefpanda; actually my app is using TK for the clipboard and TK is initialized before CEF. Strangely, it seems to prevent the 'Illegal instruction: 4' crash.

So there is some state of the system or the graphical environment that CEF relies upon that panda changes when initializing which then confuses CEF.

I tried to initialize panda without creating a window and then creating it manually afterward but CEF still crashes.

[lejar]

Alright I've been doing a lot of research the past couple days and I think I've figured it out.
cefpython and panda3d both create their own NSApplication, which is the main event handler on MacOS. The one in panda3d is a subclass of NSApplication, and it does some custom event handling. I think if you make the panda3d instance first, the cefpython event handling logic will get confused and you get the illegal instruction error.

On the other hand, creating the cefpython NSApplication first will make your panda3d window not accept any input except for drag events. The easy solution for this is adding this to your python script after initializing cefpython:

import pycocoa
pycocoa.NSApplication.sharedApplication().setActivationPolicy_(pycocoa.NSApplicationActivationPolicyRegular)

This can actually be done without the need for pycocoa by making a small change to panda3d in some objective-c code, but I'll be putting that in a separate pull request on panda3d's repo.

[el-dee]

Ah, that rings something, maybe we should try with the option external-message-pump to true so cef doesn't create it's own event logic.

Actually according to the documentation it's mandatory on macos

[lejar]

I tried using external-message-pump already but I didn't notice any difference with regards to the event handling. Do you understand what it is supposed to be doing?

[el-dee]

Looking at external-message-pump more closely, its purpose is to solve another problem when the event loop in cef hangs not crash at initialization, see
cztomczak/cefpython#442

[el-dee]

I had time to look again at the problem, and this time with a debugger :)

The stack trace when the app crashes is as follow :

Crashing on exception: _createMenuRef called with existing principal MenuRef already associated with menu

Application Specific Backtrace 1:
0   CoreFoundation                      0x00007fff53a0dacd __exceptionPreprocess + 256
1   libobjc.A.dylib                     0x00007fff7e0eea17 objc_exception_throw + 48
2   CoreFoundation                      0x00007fff53a0d8ff +[NSException raise:format:] + 201
3   AppKit                              0x00007fff50f6bae4 -[NSCarbonMenuImpl _createMenuRef] + 62
4   AppKit                              0x00007fff50f6b420 -[NSCarbonMenuImpl _instantiateCarbonMenu] + 140
5   AppKit                              0x00007fff50f459d9 -[NSApplication finishLaunching] + 691
6   AppKit                              0x00007fff50f453c7 -[NSApplication run] + 250
7   Chromium Embedded Framework         0x000000012117318c cef_time_delta + 2449884
8   Chromium Embedded Framework         0x0000000121171e6e cef_time_delta + 2444990
9   Chromium Embedded Framework         0x0000000121191919 cef_time_delta + 2574697
10  Chromium Embedded Framework         0x0000000120e85baf ChromeAppModeStart_v4 + 2234751
11  cefpython_py37.so                   0x000000011f145524 _ZL43__pyx_pw_14cefpython_py37_35MessageLoopWorkP7_objectS0_ + 116
12  Python                              0x000000010ab5150b _PyMethodDef_RawFastCallKeywords + 583
13  Python                              0x000000010ab50a38 _PyCFunction_FastCallKeywords + 41
14  Python                              0x000000010abe5368 call_function + 628
15  Python                              0x000000010abde343 _PyEval_EvalFrameDefault + 6767
16  Python                              0x000000010ab50e0c function_code_fastcall + 106
17  Python                              0x000000010ab5177f _PyObject_Call_Prepend + 131
18  Python                              0x000000010ab50b45 PyObject_Call + 136
19  core.cpython-37m-darwin.so          0x000000010bbe63b2 _ZN12PythonThread16call_python_funcEP7_objectS1_ + 178
20  core.cpython-37m-darwin.so          0x000000010bbe9f19 _ZN10PythonTask14do_python_taskEv + 185
21  core.cpython-37m-darwin.so          0x000000010bbe9e4a _ZN10PythonTask7do_taskEv + 26
22  libpanda.1.10.dylib                 0x000000010c7836ef _ZN9AsyncTask18unlock_and_do_taskEv + 319
23  libpanda.1.10.dylib                 0x000000010c787f1c _ZN14AsyncTaskChain16service_one_taskEPNS_20AsyncTaskChainThreadE + 652
24  libpanda.1.10.dylib                 0x000000010c786d72 _ZN14AsyncTaskChain7do_pollEv + 418
25  libpanda.1.10.dylib                 0x000000010c78e3bc _ZN16AsyncTaskManager4pollEv + 76
26  core.cpython-37m-darwin.so          0x000000010b8b2a64 _ZL31Dtool_AsyncTaskManager_poll_138P7_objectS0_ + 52
27  Python                              0x000000010ab5150b _PyMethodDef_RawFastCallKeywords + 583
28  Python                              0x000000010ab55cdd _PyMethodDescr_FastCallKeywords + 81
29  Python                              0x000000010abe5402 call_function + 782
30  Python                              0x000000010abde32a _PyEval_EvalFrameDefault + 6742
31  Python                              0x000000010ab50e0c function_code_fastcall + 106
32  Python                              0x000000010abe53d5 call_function + 737
33  Python                              0x000000010abde32a _PyEval_EvalFrameDefault + 6742
34  Python                              0x000000010abe5bc9 _PyEval_EvalCodeWithName + 1698
35  Python                              0x000000010ab50a00 _PyFunction_FastCallKeywords + 212
36  Python                              0x000000010abe53d5 call_function + 737
37  Python                              0x000000010abde32a _PyEval_EvalFrameDefault + 6742
38  Python                              0x000000010ab50e0c function_code_fastcall + 106
39  Python                              0x000000010abe53d5 call_function + 737
40  Python                              0x000000010abde32a _PyEval_EvalFrameDefault + 6742
41  Python                              0x000000010abe5bc9 _PyEval_EvalCodeWithName + 1698
42  Python                              0x000000010abdc831 PyEval_EvalCode + 51
43  Python                              0x000000010ac0ac3c run_mod + 54
44  Python                              0x000000010ac09c6f PyRun_FileExFlags + 160
45  Python                              0x000000010ac09326 PyRun_SimpleFileExFlags + 270
46  Python                              0x000000010ac21b2e pymain_main + 5445
47  Python                              0x000000010ac2219c _Py_UnixMain + 56
48  libdyld.dylib                       0x00007fff7f8bd3d5 start + 1

This exception is raised when one calls a second time [NSApplication finishLaunching]. CocoaGraphicsWindow:CocoaGraphicsWindow() is doing it in panda and CEF does it again in MessageLoopWork

Using "external_message_pump": True in app_settings force CEF to not initialise the NSApp and message loop and rely on the external application to do so.

With that parameter I don't get the crash anymore on Mojave nor Catalina.

Note that CocoaGraphicsWindow is already setting the activation policy as suggested above.

    [NSApp setActivationPolicy:NSApplicationActivationPolicyRegular];

So I would suggest to set ``"external_message_pump": True``` by default for macOS.

[Moguri]

Can this issue be closed with PR #10 being merged, or are there still some remaining issues?

[el-dee]

I think there are no more issues, at least related to a crash :-) Though if @lejar you could test with the newly released Panda to be sure ?