Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> #383

Open
sheng-di opened this issue Jan 28, 2024 · 2 comments
Open

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> #383

sheng-di opened this issue Jan 28, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@sheng-di
Copy link

Issue Description / 问题描述

Please provide a detailed description of the error or issue you encountered. / 请详细描述您遇到的错误或问题。

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

Steps to Reproduce / 复现步骤

Please provide the specific steps to reproduce the error. / 请提供复现错误的具体步骤。

docker compose up

Expected Behavior / 预期行为

Describe the behavior you expected to see. / 请描述您期望的正确行为。
No Error.

Environment / 环境信息

  • Operating System / 操作系统：Windows
  • Python Version / Python 版本：3.10
  • Other Relevant Information / 其他相关信息：In WSL2 with docker-desktop.

Error Screenshots or Logs / 错误截图或日志

If possible, please provide relevant screenshots or logs of the error. / 如果可能，请提供相关的错误截图或日志文件。

Additional Notes / 其他备注

If you have any additional information or notes, please add them here. / 如果有其他补充信息，请在此处添加。

➜  XAgent git:(main) ✗ docker compose up
[+] Running 5/0
 ⠿ Container xagent-ToolServerNode-1  Created                                                                                                                                  0.0s
 ⠿ Container xagent-xagent-mysql-1    Running                                                                                                                                  0.0s
 ⠿ Container xagent-db-1              Running                                                                                                                                  0.0s
 ⠿ Container xagent-xagent-redis-1    Running                                                                                                                                  0.0s
 ⠿ Container XAgent-Server            Running                                                                                                                                  0.0s
Attaching to XAgent-Server, xagent-ToolServerManager-1, xagent-ToolServerNode-1, xagent-db-1, xagent-xagent-mysql-1, xagent-xagent-redis-1
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Starting gunicorn 21.2.0
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [7] [INFO] Booting worker with pid: 7
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [8] [INFO] Booting worker with pid: 8
xagent-ToolServerNode-1     | Traceback (most recent call last):
xagent-ToolServerNode-1     |   File "/usr/local/bin/uvicorn", line 8, in <module>
xagent-ToolServerNode-1     |     sys.exit(main())
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
xagent-ToolServerNode-1     |     return self.main(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
xagent-ToolServerNode-1     |     rv = self.invoke(ctx)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
xagent-ToolServerNode-1     |     return ctx.invoke(self.callback, **ctx.params)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
xagent-ToolServerNode-1     |     return __callback(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
xagent-ToolServerNode-1     |     run(
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
xagent-ToolServerNode-1     |     server.run()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
xagent-ToolServerNode-1     |     return asyncio.run(self.serve(sockets=sockets))
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
xagent-ToolServerNode-1     |     return loop.run_until_complete(main)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
xagent-ToolServerNode-1     |     return future.result()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
xagent-ToolServerNode-1     |     config.load()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
xagent-ToolServerNode-1     |     self.loaded_app = import_from_string(self.app)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
xagent-ToolServerNode-1     |     module = importlib.import_module(module_str)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerNode-1     |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerNode-1     |   File "/app/main.py", line 13, in <module>
xagent-ToolServerNode-1     |     from config import CONFIG,logger
xagent-ToolServerNode-1     |   File "/app/config.py", line 73, in <module>
xagent-ToolServerNode-1     |     CONFIG = NodeConfig()
xagent-ToolServerNode-1     |   File "/app/config.py", line 31, in __init__
xagent-ToolServerNode-1     |     self.cfg:Dict = yaml.load(open(config_file_path, "r", encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerNode-1     | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/node.yml'
xagent-ToolServerNode-1 exited with code 1
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [7] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [8] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [8] [INFO] Worker exiting (pid: 8)
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [7] [INFO] Worker exiting (pid: 7)
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [1] [ERROR] Worker (pid:7) exited with code 3
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [1] [ERROR] Worker (pid:8) exited with code 3
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 209, in run
xagent-ToolServerManager-1  |     self.sleep()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 360, in sleep
xagent-ToolServerManager-1  |     ready = select.select([self.PIPE[0]], [], [], 1.0)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
xagent-ToolServerManager-1  |     self.reap_workers()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
xagent-ToolServerManager-1  |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
xagent-ToolServerManager-1  | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | During handling of the above exception, another exception occurred:
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 662, in kill_worker
xagent-ToolServerManager-1  |     os.kill(pid, sig)
xagent-ToolServerManager-1  | ProcessLookupError: [Errno 3] No such process
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | During handling of the above exception, another exception occurred:
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/bin/gunicorn", line 8, in <module>
xagent-ToolServerManager-1  |     sys.exit(run())
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
xagent-ToolServerManager-1  |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 236, in run
xagent-ToolServerManager-1  |     super().run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run
xagent-ToolServerManager-1  |     Arbiter(self).run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 229, in run
xagent-ToolServerManager-1  |     self.halt(reason=inst.reason, exit_status=inst.exit_status)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 342, in halt
xagent-ToolServerManager-1  |     self.stop()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 393, in stop
xagent-ToolServerManager-1  |     self.kill_workers(sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 652, in kill_workers
xagent-ToolServerManager-1  |     self.kill_worker(pid, sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 662, in kill_worker
xagent-ToolServerManager-1  |     os.kill(pid, sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
xagent-ToolServerManager-1  |     self.reap_workers()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
xagent-ToolServerManager-1  |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
xagent-ToolServerManager-1  | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
xagent-ToolServerManager-1 exited with code 1
@sheng-di sheng-di added the bug Something isn't working label Jan 28, 2024
@sheng-di
Copy link
Author 

docker compose up
[+] Running 6/0
 ⠿ Container xagent-db-1                 Created                                                                             0.0s
 ⠿ Container xagent-xagent-redis-1       Created                                                                             0.0s
 ⠿ Container xagent-ToolServerManager-1  Created                                                                             0.0s
 ⠿ Container xagent-xagent-mysql-1       Created                                                                             0.0s
 ⠿ Container xagent-ToolServerNode-1     Created                                                                             0.0s
 ⠿ Container XAgent-Server               Created                                                                             0.0s
Attaching to XAgent-Server, xagent-ToolServerManager-1, xagent-ToolServerNode-1, xagent-db-1, xagent-xagent-mysql-1, xagent-xagent-redis-1
^Bxagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.542+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.544+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":21},"outgoing":{"minWireVersion":6,"maxWireVersion":21},"isInternalClient":true}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.546+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrationDonors"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMigrationRecipients"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"TENANT_M", "id":7091600, "ctx":"main","msg":"Starting TenantMigrationAccessBlockerRegistry"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"daf36b8df185"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"7.0.5","gitVersion":"7809d71e84e314b497f282ea8aa06d7ded3eb205","openSSLVersion":"OpenSSL 3.0.2 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2204","distarch":"x86_64","target_arch":"x86_64"}}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"security":{"authorization":"enabled"}}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.557+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.557+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.557+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3299M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.710 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.712 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.712 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.712 * Configuration loaded
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.712 * monotonic clock: POSIX clock_gettime
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.714 * Running mode=standalone, port=6379.
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.716 * Server initialized
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.716 * Ready to accept connections tcp
xagent-xagent-mysql-1       | 2024-01-28 10:18:17+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
xagent-xagent-mysql-1       | 2024-01-28 10:18:18+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
xagent-xagent-mysql-1       | 2024-01-28 10:18:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
xagent-xagent-mysql-1       | '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [1] [INFO] Starting gunicorn 21.2.0
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [7] [INFO] Booting worker with pid: 7
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [8] [INFO] Booting worker with pid: 8
xagent-ToolServerNode-1     | Traceback (most recent call last):
xagent-ToolServerNode-1     |   File "/usr/local/bin/uvicorn", line 8, in <module>
xagent-ToolServerNode-1     |     sys.exit(main())
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
xagent-ToolServerNode-1     |     return self.main(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
xagent-ToolServerNode-1     |     rv = self.invoke(ctx)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
xagent-ToolServerNode-1     |     return ctx.invoke(self.callback, **ctx.params)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
xagent-ToolServerNode-1     |     return __callback(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
xagent-ToolServerNode-1     |     run(
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
xagent-ToolServerNode-1     |     server.run()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
xagent-ToolServerNode-1     |     return asyncio.run(self.serve(sockets=sockets))
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
xagent-ToolServerNode-1     |     return loop.run_until_complete(main)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
xagent-ToolServerNode-1     |     return future.result()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
xagent-ToolServerNode-1     |     config.load()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
xagent-ToolServerNode-1     |     self.loaded_app = import_from_string(self.app)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
xagent-ToolServerNode-1     |     module = importlib.import_module(module_str)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerNode-1     |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerNode-1     |   File "/app/main.py", line 13, in <module>
xagent-ToolServerNode-1     |     from config import CONFIG,logger
xagent-ToolServerNode-1     |   File "/app/config.py", line 73, in <module>
xagent-ToolServerNode-1     |     CONFIG = NodeConfig()
xagent-ToolServerNode-1     |   File "/app/config.py", line 31, in __init__
xagent-ToolServerNode-1     |     self.cfg:Dict = yaml.load(open(config_file_path, "r", encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerNode-1     | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/node.yml'
xagent-xagent-mysql-1       | 2024-01-28T10:18:18.253736Z 0 [System] [MY-015015] [Server] MySQL Server - start.
xagent-xagent-mysql-1       | 2024-01-28T10:18:18.873413Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1
xagent-xagent-mysql-1       | 2024-01-28T10:18:18.903021Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.408+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":1850}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.408+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.412621Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [8] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.422+00:00"},"s":"W",  "c":"CONTROL",  "id":22178,   "ctx":"initandlisten","msg":"/sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'","tags":["startupWarnings"]}
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.423+00:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [8] [INFO] Worker exiting (pid: 8)
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.429+00:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":21},"outgoing":{"minWireVersion":6,"maxWireVersion":21},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":21,"maxWireVersion":21},"outgoing":{"minWireVersion":21,"maxWireVersion":21},"isInternalClient":true}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.429+00:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"7.0","context":"startup"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.430+00:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.432+00:00"},"s":"I",  "c":"CONTROL",  "id":6608200, "ctx":"initandlisten","msg":"Initializing cluster server parameters from disk"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.432+00:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.439+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [7] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [7] [INFO] Worker exiting (pid: 7)
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.452+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.452+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"CONTROL",  "id":8423403, "ctx":"initandlisten","msg":"mongod startup complete","attr":{"Summary of time elapsed":{"Startup from clean shutdown?":true,"Statistics":{"Transport layer setup":"0 ms","Run initial syncer crash recovery":"0 ms","Create storage engine lock file in the data directory":"0 ms","Get metadata describing storage engine":"0 ms","Validate options in metadata against current startup options":"0 ms","Create storage engine":"1853 ms","Write current PID to file":"10 ms","Initialize FCV before rebuilding indexes":"6 ms","Drop abandoned idents and get back indexes that need to be rebuilt or builds that need to be restarted":"1 ms","Rebuild indexes for collections":"0 ms","Load cluster parameters from disk for a standalone":"1 ms","Build user and roles graph":"0 ms","Verify indexes for admin.system.users collection":"0 ms","Set up the background thread pool responsible for waiting for opTimes to be majority committed":"1 ms","Initialize information needed to make a mongod instance shard aware":"0 ms","Start up the replication coordinator":"2 ms","Start transport layer":"1 ms","_initAndListen total elapsed time":"1902 ms"}}}}
xagent-ToolServerNode-1 exited with code 1
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Worker (pid:8) exited with code 3
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Worker (pid:7) was sent SIGTERM!
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.616525Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.620672Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.657981Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.658054Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.661305Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Shutting down: Master
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Reason: Worker failed to boot.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.690017Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.690152Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.3.0'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
xagent-ToolServerManager-1 exited with code 3

@sailaoda
Copy link
Collaborator

sailaoda commented Mar 5, 2024

hello， please check your file "./assets/config/node.yml and manager.yml" like this
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sheng-di @sailaoda