After I close the session,there are many warings-> RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses

[neblen]

1.I dont't know the reason,which causes the warning.
as following:

2021-06-08 02:13:25,663 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:25,737 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:30,669 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:30,745 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:35,676 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:35,751 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:40,678 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:40,757 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:45,684 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:45,764 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 02:13:50,688 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'

2.Then I create a new session.
It happens->

/opt/conda/lib/python3.8/site-packages/graphscope/framework/errors.py in check_grpc_response(response)
    169     error_type = _gs_error_types.get(status.code)
    170     if error_type:
--> 171         raise error_type(status.error_msg, detail)
    172     else:
    173         raise RuntimeError(

ConnectionError: 'Cannot setup more than one connection at the same time.'

[lidongze0629]

Hi, @neblen, It seems the client is disconnected from the server ,and could you report more details here? Or how to reproduce the behavior.

[neblen]

Hi，If I connect graphscope in notebook，it sometimes happen.
The erreo info is as following.

2021-06-08 10:34:43,310 [INFO][coordinator:513]: build frontend 192.168.0.55:31489 for graph 5501896057650258
2021-06-08 06:07:54,378 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: Socket closed'
2021-06-08 06:07:54,379 [WARNING][rpc:71]: Grpc call '_fetch_logs_impl' failed: StatusCode.UNAVAILABLE: Socket closed
2021-06-08 06:07:59,432 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:04,439 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:09,446 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:14,452 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:19,459 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:24,465 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:29,472 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: failed to connect to all addresses'
2021-06-08 06:08:34,474 [WARNING][session:573]: 'RPC failed, the engine might have crashed: rpc send_heartbeat: failed with error code StatusCode.UNAVAILABLE, details: fai

[neblen]

I deploy graphscope on k8s cluster by using https://github.com/alibaba/GraphScope/tree/main/charts/graphscope
I don't know clearly these charts.What should I pay attention to in this helm chart?

[neblen]

I still have some questions.When I connect graphscope in Pycharm,I forget to close session,but it is still normal。
Some logs are as following.

021-06-08 14:35:32,210 [INFO][coordinator:513]: build frontend192.168.0.55:32440 for graph 5543730941740292
2021-06-08 14:35:32,974 [INFO][coordinator:513]: build frontend 192.168.0.55:32300 for graph 5543732276494872
2021-06-08 14:35:33,576 [INFO][coordinator:555]: Coordinator close interactive instance with url[http://192.168.0.55:8080/instance/close?graphName=5543730941740292&podNameList=gs-engine-graphscope-gd74f,gs-engine-graphscope-tcjw4&containerName=engine&waitingForDelete=False]
2021-06-08 14:35:35,077 [INFO][coordinator:555]: Coordinator close interactive instance with url[http://192.168.0.55:8080/instance/close?graphName=5543732276494872&podNameList=gs-engine-graphscope-gd74f,gs-engine-graphscope-tcjw4&containerName=engine&waitingForDelete=False]

[lidongze0629]

I still have some questions.When I connect graphscope in Pycharm,I forget to close session,but it is still normal。
Some logs are as following.

021-06-08 14:35:32,210 [INFO][coordinator:513]: build frontend192.168.0.55:32440 for graph 5543730941740292
2021-06-08 14:35:32,974 [INFO][coordinator:513]: build frontend 192.168.0.55:32300 for graph 5543732276494872
2021-06-08 14:35:33,576 [INFO][coordinator:555]: Coordinator close interactive instance with url[http://192.168.0.55:8080/instance/close?graphName=5543730941740292&podNameList=gs-engine-graphscope-gd74f,gs-engine-graphscope-tcjw4&containerName=engine&waitingForDelete=False]
2021-06-08 14:35:35,077 [INFO][coordinator:555]: Coordinator close interactive instance with url[http://192.168.0.55:8080/instance/close?graphName=5543732276494872&podNameList=gs-engine-graphscope-gd74f,gs-engine-graphscope-tcjw4&containerName=engine&waitingForDelete=False]

It is normal. Params initializing_interactive_engine defaults to True and it will create gremlin server pod asynchronously when a new graph be created.

# you can disable it by set initializing_interactive_engine to False
graphscope.set_option(initializing_interactive_engine=False)

[lidongze0629]

I still have some questions.When I connect graphscope in Pycharm,I forget to close session,but it is still normal。
Some logs are as following.

021-06-08 14:35:32,210 [INFO][coordinator:513]: build frontend192.168.0.55:32440 for graph 5543730941740292
2021-06-08 14:35:32,974 [INFO][coordinator:513]: build frontend 192.168.0.55:32300 for graph 5543732276494872
2021-06-08 14:35:33,576 [INFO][coordinator:555]: Coordinator close interactive instance with url[http://192.168.0.55:8080/instance/close?graphName=5543730941740292&podNameList=gs-engine-graphscope-gd74f,gs-engine-graphscope-tcjw4&containerName=engine&waitingForDelete=False]
2021-06-08 14:35:35,077 [INFO][coordinator:555]: Coordinator close interactive instance with url[http://192.168.0.55:8080/instance/close?graphName=5543732276494872&podNameList=gs-engine-graphscope-gd74f,gs-engine-graphscope-tcjw4&containerName=engine&waitingForDelete=False]

It seems the GraphScope service is not available, maybe the engine backend is crashed.

Firstly, you can check pod status by kubectl get pod, if all pods is running, you can logs the coordinator pod by kubectl logs -f <coordinator_pod_name>