Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: In stability testing, the latency of query/search is unstable and has increased by many times at certain moments. #37527

Open
1 task done
zhuwenxing opened this issue Nov 8, 2024 · 6 comments
Open
1 task done

[Bug]: In stability testing, the latency of query/search is unstable and has increased by many times at certain moments. #37527

zhuwenxing opened this issue Nov 8, 2024 · 6 comments
Assignees
@XuanYang-cn @liliu-z
Labels
kind/bug Issues or changes related a bug severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20241107-f813fb45-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

image

When the delay in query/search increased, a lot of rate limits on insert/delete operations appeared.

image
At the point of increased latency, the CPU and memory usage of the query node are relatively stable.

Expected Behavior

The latency of search/query can be relatively stable.

Steps To Reproduce

No response

Milvus Log

cluster: 4am
ns: chaos-testing
pod info

❯ k get pod|grep fts-stable-test-11                                  
fts-stable-test-11-etcd-0                                         0/1     Pending                  0                 9h
fts-stable-test-11-etcd-1                                         1/1     Running                  0                 15h
fts-stable-test-11-etcd-2                                         1/1     Running                  0                 15h
fts-stable-test-11-kafka-0                                        0/2     Pending                  0                 9h
fts-stable-test-11-kafka-1                                        2/2     Running                  1 (15h ago)       15h
fts-stable-test-11-kafka-2                                        2/2     Running                  0                 15h
fts-stable-test-11-kafka-exporter-5d659487fb-v4fsz                1/1     Running                  4 (15h ago)       15h
fts-stable-test-11-milvus-datanode-6bd55bd8bd-2nggb               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-datanode-6bd55bd8bd-5d2t6               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-indexnode-964df8d55-4fmfv               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-indexnode-964df8d55-kqgmm               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-mixcoord-68b694d6df-xp6cl               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-proxy-8458cfb765-5qbfv                  1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-querynode-56c6787d78-czx9t              1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-querynode-56c6787d78-d9zwd              1/1     Running                  3 (15h ago)       15h
fts-stable-test-11-milvus-querynode-56c6787d78-wl5jx              1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-minio-0                                        1/1     Running                  0                 15h
fts-stable-test-11-minio-1                                        1/1     Running                  0                 15h
fts-stable-test-11-minio-2                                        0/1     Pending                  0                 9h
fts-stable-test-11-minio-3                                        1/1     Running                  0                 15h
fts-stable-test-11-zookeeper-0                                    1/1     Running                  0                 15h
fts-stable-test-11-zookeeper-1                                    1/1     Running                  0                 15h
fts-stable-test-11-zookeeper-2                                    0/1     Pending                  0                 9h

grafana link

Anything else?

Will the rate limit of insert/delete operations affect the latency of search/query? If so, it would contradict the design principle of Milvus's read-write separation.
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 8, 2024
@zhuwenxing zhuwenxing added this to the 2.5.0 milestone Nov 8, 2024
@yanliang567
Copy link
Contributor

/assign @liliu-z
/unassign

@sre-ci-robot sre-ci-robot assigned liliu-z and unassigned yanliang567 Nov 8, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 8, 2024
@zhuwenxing zhuwenxing added the severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. label Nov 11, 2024
@liliu-z
Copy link
Member

liliu-z commented Nov 11, 2024 •
edited
Loading

Screen Shot 2024-11-11 at 4 01 37 PM

Noticed the host node CPU rising.

Screen Shot 2024-11-11 at 4 02 02 PM

Pod CPU was reaching to its limit 4CPU

Some comments:

  1. Plz paste the node number, since checking other pods' affect is the most common thing and we have to go through all nodes in 4am 1 by 1 to find out the match node.
  2. Plz don't do any performance related test on 4am, especially long-period test.

@liliu-z liliu-z removed their assignment Nov 12, 2024
@liliu-z
Copy link
Member

liliu-z commented Nov 12, 2024

/assign @zhuwenxing

@zhuwenxing
Copy link
Contributor Author

/unassign

/assign @liliu-z

There's a strange point here: the time point at which the search/query delay increases coincides with the emergence of the rate limit on insert/delete operations. It's necessary to confirm whether these two events are related or independent.

detailed pod info including node name

[2024-11-07T11:08:50.570Z] + kubectl get pods -o wide

[2024-11-07T11:08:50.571Z] + grep fts-stable-test-11

[2024-11-07T11:08:51.131Z] fts-stable-test-11-etcd-0                                         1/1     Running            0                 2m58s   10.104.16.155   4am-node21   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-etcd-1                                         1/1     Running            0                 2m58s   10.104.30.158   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-etcd-2                                         1/1     Running            0                 2m58s   10.104.19.147   4am-node28   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-0                                        2/2     Running            0                 2m58s   10.104.16.157   4am-node21   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-1                                        2/2     Running            1 (2m26s ago)     2m58s   10.104.19.145   4am-node28   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-2                                        2/2     Running            0                 2m58s   10.104.30.159   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-exporter-5d659487fb-v4fsz                1/1     Running            4 (2m12s ago)     2m58s   10.104.30.150   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-datanode-6bd55bd8bd-2nggb               1/1     Running            2 (2m36s ago)     2m58s   10.104.18.224   4am-node25   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-datanode-6bd55bd8bd-5d2t6               1/1     Running            2 (2m35s ago)     2m58s   10.104.15.123   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-indexnode-964df8d55-4fmfv               1/1     Running            2 (2m36s ago)     2m58s   10.104.15.122   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-indexnode-964df8d55-kqgmm               1/1     Running            2 (2m36s ago)     2m58s   10.104.24.37    4am-node29   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-mixcoord-68b694d6df-xp6cl               1/1     Running            2 (2m36s ago)     2m58s   10.104.15.121   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-proxy-8458cfb765-5qbfv                  1/1     Running            2 (2m36s ago)     2m58s   10.104.30.149   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-querynode-56c6787d78-czx9t              1/1     Running            2 (2m35s ago)     2m58s   10.104.18.227   4am-node25   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-querynode-56c6787d78-d9zwd              1/1     Running            3 (2m10s ago)     2m58s   10.104.5.252    4am-node12   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-querynode-56c6787d78-wl5jx              1/1     Running            2 (2m35s ago)     2m58s   10.104.15.124   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-0                                        1/1     Running            0                 2m58s   10.104.30.156   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-1                                        1/1     Running            0                 2m58s   10.104.20.216   4am-node22   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-2                                        1/1     Running            0                 2m58s   10.104.16.156   4am-node21   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-3                                        1/1     Running            0                 2m58s   10.104.24.40    4am-node29   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-zookeeper-0                                    1/1     Running            0                 2m58s   10.104.20.215   4am-node22   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-zookeeper-1                                    1/1     Running            0                 2m58s   10.104.30.157   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-zookeeper-2                                    1/1     Running            0                 2m58s   10.104.16.159   4am-node21   <none>           <none>

@sre-ci-robot sre-ci-robot assigned liliu-z and unassigned zhuwenxing Nov 12, 2024
@liliu-z
Copy link
Member

liliu-z commented Nov 12, 2024

Looks like the increased deletion volume caused the rate limit. But it is weird to make search that slow.
/assign @XuanYang-cn
Plz help take a look

@ThreadDao
Copy link
Contributor

maybe related with #37413

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@XuanYang-cn XuanYang-cn

@liliu-z liliu-z

Labels
kind/bug Issues or changes related a bug severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
5 participants
@zhuwenxing @ThreadDao @XuanYang-cn @yanliang567 @liliu-z