[Bug]: The status of collections show “0% is loaded".

[allonx]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.3.15
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): ubuntu22.04
- CPU/Memory: 384G
- GPU: 
- Others:

Current Behavior

Miluvus 2.3.15使用docker compose 部署。数据量2000万条，机房断电后ups击穿，milvus重启后，collection状态显示0% is loaded.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

error_logs.txt

Anything else?

No response

[xiaofan-luan]

@allonx
did you try to restart the node?

There seems to be no hard data damage
restart docker and check if it works.
If no we'd like to glad more help. connect me at [email protected]

[yanliang567]

@allonx quick questions, did etcd service go back to running healthy after power back? could you please share the etcd logs for us?

[yanliang567]

/assign @allonx

[allonx]

@allonx quick questions, did etcd service go back to running healthy after power back? could you please share the etcd logs for us?

etcd_logs.txt

[allonx]

@allonx did you try to restart the node?

There seems to be no hard data damage restart docker and check if it works. If no we'd like to glad more help. connect me at [email protected]

I had restart docker-compose again. It doesn't work!

[allonx]

logs.tar.gz

@allonx did you try to restart the node?
There seems to be no hard data damage restart docker and check if it works. If no we'd like to glad more help. connect me at [email protected]

I had restart docker-compose again. It doesn't work!

logs.tar.gz

[xiaofan-luan]

@allonx
What I need is all log from querycoord, datacoord and querynode
you can collect with https://github.com/milvus-io/milvus/tree/master/deployments/export-log
Right now it seems that there are many small segments in the cluster.

maybe some file brokes on object storage or rocksdb.

Since you are use standalone mode, there is only one copy of all the data.
but we need error log to find which data actaully broke

[allonx]

For Milvus installed with docker-compose, you can use docker compose logs > milvus.log to export the logs.

The file (logs.tar.gz) I uploaded is a zip file containing all the logs, which was exported according to the command you provided above.

[xiaofan-luan]

@allonx

this a bug already fixed for rocksmq

#36618

try to upgrade to 2.4 latest see if it fixed.

[xiaofan-luan]

rocksmq is slow on recovery.
Or you can try to remove the rocksdb directory to fast recovery(could lose some near line data)

[allonx]

@allonx

this a bug already fixed for rocksmq

#36618

try to upgrade to 2.4 latest see if it fixed.

yeah, I upgraded milvus to version 2.4.15 and it seems to be working, but the recovery is slow. How can I remove the rocksdb diretory? Where is it located?