Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some error I don'T understand #2

Open
beniroquai opened this issue Nov 5, 2024 · 5 comments
Open

Some error I don'T understand #2

beniroquai opened this issue Nov 5, 2024 · 5 comments

Comments

@beniroquai
Copy link
Contributor

beniroquai commented Nov 5, 2024

Moin @jhnnsrs! So it seems we are almost ready. Login is working, functions are exposed, but when I start a workflow from the orkestrator I get this error (with the current implementation of the interface: https://github.com/openUC2/imswitch-arkitekt-next/blob/master/src/imswitch_arkitekt_next/imswitch_arkitekt_next_controller.py#L26):

ERROR:imswitch:[HistoScanController] Could not connect to remote histo manager: 'VirtualMicroscopeManager' object has no attribute '_imswitch_client'
2024-11-05 20:38:02 DEBUG [ImConMainController] Creating controller for widget Hypha
DEBUG:imswitch:[ImConMainController] Creating controller for widget Hypha
2024-11-05 20:38:02 DEBUG [ImConMainController] Creating controller for widget Histogramm
DEBUG:imswitch:[ImConMainController] Creating controller for widget Histogramm
2024-11-05 20:38:02 DEBUG [ImConMainController] Creating controller for widget imswitch_arkitekt_next
DEBUG:imswitch:[ImConMainController] Creating controller for widget imswitch_arkitekt_next
2024-11-05 20:38:04 DEBUG [imswitch_arkitekt_next_controller] Initializing imswitch arkitekt_next controller
DEBUG:imswitch:[imswitch_arkitekt_next_controller] Initializing imswitch arkitekt_next controller
Failed to call init_extensions for installed package humanfriendly: humanfriendly.__spec__ is None
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/arkitekt_next/service_registry.py", line 100, in check_and_import_services
    module_spec = importlib.util.find_spec(module_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/importlib/util.py", line 115, in find_spec
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: humanfriendly.__spec__ is None
/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
Failed to call init_extensions for installed package humanfriendly: humanfriendly.__spec__ is None
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/structures/default.py", line 83, in check_and_import_structures
    module_spec = importlib.util.find_spec(module_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/importlib/util.py", line 115, in find_spec
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: humanfriendly.__spec__ is None
Failed to call init_extensions for installed package humanfriendly: humanfriendly.__spec__ is None
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/__arkitekt__.py", line 80, in check_and_import_extensions
    module_spec = importlib.util.find_spec(module_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/importlib/util.py", line 115, in find_spec
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: humanfriendly.__spec__ is None
Waiting for future
2024-11-05 20:38:06 DEBUG [imswitch_arkitekt_next_controller] Start Arkitekt Runtime
DEBUG:imswitch:[imswitch_arkitekt_next_controller] Start Arkitekt Runtime
2024-11-05 20:38:06 DEBUG [ImConMainController] Creating controller for widget Objective
DEBUG:imswitch:[ImConMainController] Creating controller for widget Objective
Backend macosx is interactive backend. Turning interactive mode on.
╭───────────────────────────── Device Code Grant ──────────────────────────────╮
│                                                                              │
│     Please visit the following URL:                                          │
│     http://localhost/lok/f/configure/?device_code=f71e83f9&grant=device_code │
│     or go to this URL:                                                       │
│     http://localhost/lok/f/device                                            │
│     and enter the code:                                                      │
│     f71e83f9                                                                 │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
2024-11-05 20:38:14 DEBUG [ImConMainController] Start ImSwitch Server
DEBUG:imswitch:[ImConMainController] Start ImSwitch Server
2024-11-05 20:38:14 INFO [main] initialize module imcontrol
INFO:imswitch:[main] initialize module imcontrol
2024-11-05 20:38:14 INFO [main] init done
INFO:imswitch:[main] init done
2024-11-05 20:38:15 DEBUG [ImSwitchServer] Started server with URI -> PYRO:ImSwitchServer@0.0.0.0:54333
DEBUG:imswitch:[ImSwitchServer] Started server with URI -> PYRO:ImSwitchServer@0.0.0.0:54333
INFO:     Started server process [24303]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on https://0.0.0.0:8001 (Press CTRL+C to quit)
We are in luck. We found a local template
Actor entered
{}
source='1' handle='return_0' type=<EventType.NEXT: 'NEXT'> value=() exception=None caused_by=(0,)
source='1' handle='return_0' type=<EventType.COMPLETE: 'COMPLETE'> value=None exception=None caused_by=(0,)
Starting to iterate?
CRITICAL:root:Assignation Error id='5ffc461a-b0ec-41b7-8b7a-80e2d5b23dcc' type=<MessageType.ASSIGN: 'ASSIGN'> mother=None parent=5 assignation=5 reference='ark-3d060041-71a4-4a44-a0b9-9cc199947fe4_0' provision=None reservation=None args={'n': None, 'timeout': '5'} message=None user=None cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 276, in on_assign
    async for returns in iterate_spawned(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/koil/helpers.py", line 433, in iterate_spawned
    f = loop.run_in_executor(
        ^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2747, in uvloop.loop.Loop.run_in_executor
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
ERROR:rekuest_next.postmans.utils:Error in Assignation
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/postmans/utils.py", line 207, in aiterate_raw
    raise AssignException(f"Critical error: {ass.message}")
rekuest_next.postmans.errors.AssignException: Critical error: cannot schedule new futures after interpreter shutdown
ERROR:reaktion_next.atoms.generic:ark-3d060041-71a4-4a44-a0b9-9cc199947fe4 map failed
ERROR:rekuest_next.actors.base:Assign task <Task finished name='Task-241' coro=<ThreadedGenActor.on_assign() done, defined at /Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py:253> exception=RuntimeError('cannot schedule new futures after interpreter shutdown')> failed with exception cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/base.py", line 160, in assign_task_done
    raise task.exception()
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 318, in on_assign
    raise e
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 276, in on_assign
    async for returns in iterate_spawned(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/koil/helpers.py", line 433, in iterate_spawned
    f = loop.run_in_executor(
        ^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2747, in uvloop.loop.Loop.run_in_executor
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
source='ark-3d060041-71a4-4a44-a0b9-9cc199947fe4' handle='return_0' type=<EventType.ERROR: 'ERROR'> value=None exception=AssignException('Critical error: cannot schedule new futures after interpreter shutdown') caused_by=(0,)
source='ark-3d060041-71a4-4a44-a0b9-9cc199947fe4' handle='return_0' type=<EventType.ERROR: 'ERROR'> value=None exception=AssignException('Critical error: cannot schedule new futures after interpreter shutdown') caused_by=(0,)
CRITICAL:root:Assignation id='b3db3515-0ff8-4335-99b0-378645d52935' type=<MessageType.ASSIGN: 'ASSIGN'> mother=None parent=None assignation=5 reference=None provision=None reservation=None args={} message=None user=None failed
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/reaktion_next/actor.py", line 333, in on_assign
    raise spawned_event.exception
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/reaktion_next/atoms/generic.py", line 159, in run
    async for result in self.merge_map(event):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/reaktion_next/atoms/arkitekt.py", line 59, in merge_map
    async for r in self.contract.aiterate_raw(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/postmans/utils.py", line 236, in aiterate_raw
    raise e
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/postmans/utils.py", line 207, in aiterate_raw
    raise AssignException(f"Critical error: {ass.message}")
rekuest_next.postmans.errors.AssignException: Critical error: cannot schedule new futures after interpreter shutdown
@jhnnsrs
Copy link

jhnnsrs commented Nov 6, 2024

hm that might be a bug with uvloop which is used both by uvicorn and with koil and could cause some issues

could you try and see if

self.app = easy(...)
self.app.koil.uvify = False

#... later
self.app.enter()

fixes that issue?

@beniroquai
Copy link
Contributor Author

beniroquai commented Nov 7, 2024

Did not change anything yet.

Current implementation:

class imswitch_arkitekt_next_controller(ImConWidgetController):
    """Linked to CameraPluginWidget."""
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.__logger = initLogger(self)
        self.__logger.debug("Initializing imswitch arkitekt_next controller")
        
        # initalize arkitekt connection                
        self.app = easy("TEST5", url="localhost")
        # bind functions to the app
        self.app.register(self.generate_n_string)
        self.app.register(self.upload_image)
        self.app.register(self.print_string)
        self.app.koil.uvify = False
        self.app.enter()
        self.app.run_detached()
        self.__logger.debug("Start Arkitekt Runtime")

current error:

2024-11-07 21:52:26 DEBUG [imswitch_arkitekt_next_controller] Initializing imswitch arkitekt_next controller
DEBUG:imswitch:[imswitch_arkitekt_next_controller] Initializing imswitch arkitekt_next controller
Failed to call init_extensions for installed package humanfriendly: humanfriendly.__spec__ is None
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/arkitekt_next/service_registry.py", line 100, in check_and_import_services
    module_spec = importlib.util.find_spec(module_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/importlib/util.py", line 115, in find_spec
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: humanfriendly.__spec__ is None
/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
Failed to call init_extensions for installed package humanfriendly: humanfriendly.__spec__ is None
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/structures/default.py", line 83, in check_and_import_structures
    module_spec = importlib.util.find_spec(module_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/importlib/util.py", line 115, in find_spec
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: humanfriendly.__spec__ is None
Failed to call init_extensions for installed package humanfriendly: humanfriendly.__spec__ is None
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/__arkitekt__.py", line 80, in check_and_import_extensions
    module_spec = importlib.util.find_spec(module_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/importlib/util.py", line 115, in find_spec
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: humanfriendly.__spec__ is None
Backend macosx is interactive backend. Turning interactive mode on.
Waiting for future
2024-11-07 22:01:14 DEBUG [imswitch_arkitekt_next_controller] Start Arkitekt Runtime
DEBUG:imswitch:[imswitch_arkitekt_next_controller] Start Arkitekt Runtime
2024-11-07 22:01:14 DEBUG [ImConMainController] Creating controller for widget Objective
DEBUG:imswitch:[ImConMainController] Creating controller for widget Objective
2024-11-07 22:01:14 DEBUG [ImConMainController] Start ImSwitch Server
DEBUG:imswitch:[ImConMainController] Start ImSwitch Server
2024-11-07 22:01:14 INFO [main] initialize module imcontrol
INFO:imswitch:[main] initialize module imcontrol
2024-11-07 22:01:14 INFO [main] init done
INFO:imswitch:[main] init done
2024-11-07 22:01:14 DEBUG [ImSwitchServer] Started server with URI -> PYRO:ImSwitchServer@0.0.0.0:54333
DEBUG:imswitch:[ImSwitchServer] Started server with URI -> PYRO:ImSwitchServer@0.0.0.0:54333
INFO:     Started server process [41849]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on https://0.0.0.0:8001 (Press CTRL+C to quit)
╭───────────────────────────── Device Code Grant ──────────────────────────────╮
│                                                                              │
│     Please visit the following URL:                                          │
│     http://localhost/lok/f/configure/?device_code=4aad2023&grant=device_code │
│     or go to this URL:                                                       │
│     http://localhost/lok/f/device                                            │
│     and enter the code:                                                      │
│     4aad2023                                                                 │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
CRITICAL:root:Assignation Error id='8aec0ce5-fec9-4243-9df4-f04947ce02a6' type=<MessageType.ASSIGN: 'ASSIGN'> mother=None parent=None assignation=8 reference=None provision=None reservation=None args={'n': None, 'timeout': None} message=None user=None cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 276, in on_assign
    async for returns in iterate_spawned(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/koil/helpers.py", line 433, in iterate_spawned
    f = loop.run_in_executor(
        ^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2747, in uvloop.loop.Loop.run_in_executor
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
ERROR:rekuest_next.actors.base:Assign task <Task finished name='Task-130' coro=<ThreadedGenActor.on_assign() done, defined at /Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py:253> exception=RuntimeError('cannot schedule new futures after interpreter shutdown')> failed with exception cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/base.py", line 160, in assign_task_done
    raise task.exception()
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 318, in on_assign
    raise e
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 276, in on_assign
    async for returns in iterate_spawned(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/koil/helpers.py", line 433, in iterate_spawned
    f = loop.run_in_executor(
        ^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2747, in uvloop.loop.Loop.run_in_executor
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
CRITICAL:root:Assignation Error id='bfa6e4a0-9da6-4afb-98db-b50e865f75f7' type=<MessageType.ASSIGN: 'ASSIGN'> mother=None parent=None assignation=9 reference=None provision=None reservation=None args={'n': 2, 'timeout': 1} message=None user=None cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 276, in on_assign
    async for returns in iterate_spawned(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/koil/helpers.py", line 433, in iterate_spawned
    f = loop.run_in_executor(
        ^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2747, in uvloop.loop.Loop.run_in_executor
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
ERROR:rekuest_next.actors.base:Assign task <Task finished name='Task-135' coro=<ThreadedGenActor.on_assign() done, defined at /Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py:253> exception=RuntimeError('cannot schedule new futures after interpreter shutdown')> failed with exception cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/base.py", line 160, in assign_task_done
    raise task.exception()
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 318, in on_assign
    raise e
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/rekuest_next/actors/functional.py", line 276, in on_assign
    async for returns in iterate_spawned(
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/site-packages/koil/helpers.py", line 433, in iterate_spawned
    f = loop.run_in_executor(
        ^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2747, in uvloop.loop.Loop.run_in_executor
  File "/Users/bene/mambaforge/envs/imswitch311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

@jhnnsrs
Copy link

jhnnsrs commented Nov 8, 2024

Hmm could be that because the handle got garbage collected that it won't schedule again. Can you try assigning the run detached function to a handle like:

self.handle = self.app.run_detached()

If that doesn't work my remote debugging skills are no longer useful and I will dig in once back.

@beniroquai
Copy link
Contributor Author

beniroquai commented Nov 8, 2024

image CC: @jacopoabramo

Indeed, the way ImSwitch was kept alive was .. well .. not defined in the no-qt mode. Now it's ugly, but at least defined. @jacopoabramo, maybe you have an idea how to beautify this here:

https://github.com/openUC2/ImSwitch/blob/96e57184803d19822a34afc9592584497f303fd1/imswitch/imcommon/applaunch.py#L58-L63

@jhnnsrs
Copy link

jhnnsrs commented Nov 8, 2024

Woop Woop. I am wondering how this worked with the fastapi now if that was also running in a thread :D magic of threads. Just a general word of caution: qt uses an event loop - pysignals doesn't and will call callbacks immediately (afaik). So be wary of very weird race conditions :P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants