Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out - (orchestrator:keynesian_beauty_contest) #215

Open
enricorotundo opened this issue Feb 17, 2025 · 1 comment
Open

Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out - (orchestrator:keynesian_beauty_contest) #215

enricorotundo opened this issue Feb 17, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@enricorotundo
Copy link
Collaborator

enricorotundo commented Feb 17, 2025 •
edited
Loading

SDK logs:

naptha-sdk-py3.12➜  naptha-sdk git:(main) time naptha run orchestrator:keynesian_beauty_contest -p "num_agents=10" --agent_nodes "localhost,localhost" 
2025-02-17 13:14:18,205 - naptha_sdk.client.node - INFO - Node URL: http://localhost:7001
2025-02-17 13:14:18,205 - naptha_sdk.inference - INFO - Node URL: http://localhost:7001
2025-02-17 13:14:18,205 - naptha_sdk.client.hub - INFO - Hub URL: ws://localhost:3001/rpc
Parsed parameters: {'num_agents': '10'}
2025-02-17 13:14:18,225 - naptha_sdk.client.hub - INFO - Attempting authentication for user: enricorotundo1
2025-02-17 13:14:18,254 - naptha_sdk.client.hub - INFO - Successfully authenticated user: enricorotundo1
...
pending orchestrator keynesian_beauty_contest
running orchestrator keynesian_beauty_contest
error orchestrator keynesian_beauty_contest
Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
naptha run orchestrator:keynesian_beauty_contest -p "num_agents=10"    0.54s user 0.13s system 4% cpu 13.605 total

Node logs:

INFO:     127.0.0.1:56457 - "POST /orchestrator/run HTTP/1.1" 200 OK
2025-02-17 13:14:25 - node.server.http_server - INFO - Checking orchestrator run
2025-02-17 13:14:25 - node.server.http_server - INFO - Found orchestrator status: pending
2025-02-17 13:14:25 - node.server.http_server - INFO - Response: None
INFO:     127.0.0.1:56472 - "POST /orchestrator/check HTTP/1.1" 200 OK
2025-02-17 13:14:28 - node.server.http_server - INFO - Checking orchestrator run
2025-02-17 13:14:28 - node.server.http_server - INFO - Found orchestrator status: running
2025-02-17 13:14:28 - node.server.http_server - INFO - Response: None
INFO:     127.0.0.1:56522 - "POST /orchestrator/check HTTP/1.1" 200 OK
2025-02-17 13:14:31 - node.server.http_server - INFO - Checking orchestrator run
2025-02-17 13:14:31 - node.server.http_server - INFO - Found orchestrator status: error
2025-02-17 13:14:31 - node.server.http_server - INFO - Orchestrator run failed. Returning error: Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
2025-02-17 13:14:31 - node.server.http_server - INFO - Response: Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
INFO:     127.0.0.1:56528 - "POST /orchestrator/check HTTP/1.1" 200 OK

Celery logs:

[2025-02-17 16:38:06,153: WARNING/ForkPoolWorker-64] 2025-02-17 16:38:06,153 - naptha_sdk.client.node - ERROR - Error running agent: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,153: ERROR/ForkPoolWorker-64] Error running agent: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,155: WARNING/ForkPoolWorker-64] 2025-02-17 16:38:06,154 - naptha_sdk.client.node - ERROR - Full traceback: NoneType: None
[2025-02-17 16:38:06,154: ERROR/ForkPoolWorker-64] Full traceback: NoneType: None

[2025-02-17 16:38:06,156: ERROR/ForkPoolWorker-64] Module import paths: ['/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest', '/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.venv/lib/python3.12/site-packages']
[2025-02-17 16:38:06,156: ERROR/ForkPoolWorker-64] Current working directory: /Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest
[2025-02-17 16:38:06,157: ERROR/ForkPoolWorker-64] Module directory contents: [PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/pyproject.toml'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/tests'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/README.md'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.gitignore'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.venv'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.gitattributes'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/poetry.lock'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.git')]
[2025-02-17 16:38:06,158: ERROR/ForkPoolWorker-64] Error running orchestrator: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,160: ERROR/ForkPoolWorker-64] Traceback: Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 258, in load_and_run
    result = await run_func(module_run=module_run_dict)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 65, in run
    results = await beauty_contest.run_beauty_contest(module_run, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 49, in run_beauty_contest
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/modules/agent.py", line 17, in run
    agent_run = await node.run_module(module_type="agent", run_input=module_run_input)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 98, in run_module
    return await self.run_module_ws(module_type, run_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 123, in run_module_ws
    raise Exception(response['message'])
Exception: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 329, in start_run
    response = await loader.load_and_run(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 268, in load_and_run
    raise RuntimeError(f"Module execution failed: {str(e)}") from e
RuntimeError: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

[2025-02-17 16:38:06,160: ERROR/ForkPoolWorker-64] Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,162: ERROR/ForkPoolWorker-64] Traceback: Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 258, in load_and_run
    result = await run_func(module_run=module_run_dict)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 65, in run
    results = await beauty_contest.run_beauty_contest(module_run, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 49, in run_beauty_contest
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/modules/agent.py", line 17, in run
    agent_run = await node.run_module(module_type="agent", run_input=module_run_input)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 98, in run_module
    return await self.run_module_ws(module_type, run_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 123, in run_module_ws
    raise Exception(response['message'])
Exception: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 128, in _run_module_async
    await module_run_engine.start_run()
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 329, in start_run
    response = await loader.load_and_run(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 268, in load_and_run
    raise RuntimeError(f"Module execution failed: {str(e)}") from e
RuntimeError: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
@enricorotundo enricorotundo added the bug Something isn't working label Feb 17, 2025
@enricorotundo
Copy link
Collaborator Author

Got the error above with another orchestrator as well:

naptha run orchestrator:multiagent_chat -p "prompt='i would like to count up to ten, one number at a time. ill start. one.'" --agent_nodes "localhost,localhost" --kb_nodes "localhost"

@enricorotundo enricorotundo mentioned this issue Feb 17, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Naptha Infrastructure Contributors
Status: No status
Milestone
No milestone
Development

No branches or pull requests
1 participant
@enricorotundo