forked from pingcap/docs
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Troubleshoot Read and Write latency issue document (pingcap#3615)
* Add Troubleshoot Read and Write latency issue document * Update troubleshoot-cpu-issues.md * Update troubleshoot-cpu-issues.md * apply troubleshooting map translation Signed-off-by: Ran <[email protected]> * Apply suggestions from code review Co-authored-by: Ran <[email protected]> * Update troubleshoot-cpu-issues.md Co-authored-by: Lilian Lee <[email protected]> Co-authored-by: Ran <[email protected]>
- Loading branch information
3 people
authored
Oct 12, 2020
1 parent
e514af5
commit 7fb0721
Showing
2 changed files
with
140 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| --- | ||
| title: Troubleshoot Increased Read and Write Latency | ||
| summary: Learn how to troubleshoot the issue of increased read and write latency. | ||
| --- | ||
|
|
||
| # Troubleshoot Increased Read and Write Latency | ||
|
|
||
| This document introduces the possible causes of read and write latency and jitters, and how to troubleshoot these issues. | ||
|
|
||
| ## Common causes | ||
|
|
||
| ### Incorrect TiDB execution plan | ||
|
|
||
| The execution plan of queries is unstable and might select the incorrect index, which causes higher latency. | ||
|
|
||
| #### Phenomenon | ||
|
|
||
| * If the query execution plan is output in the slow log, you can directly view the plan. Execute the `select tidb_decode_plan('xxx...')` statement to parse the detailed execution plan. | ||
| * The number of scanned keys in the monitor abnormally increases; in the slow log, the number of `Scan Keys` are large. | ||
| * The SQL execution duration in TiDB is greatly different than that in other databases such as MySQL. You can compare the execution plan of other databases (for example, whether `Join Order` is different). | ||
|
|
||
| #### Possible reason | ||
|
|
||
| The statistics is inaccurate. | ||
|
|
||
| #### Troubleshooting methods | ||
|
|
||
| * Update the statistical information | ||
| * Execute `analyze table` manually and execute `analyze` periodically with the `crontab` command to keep the statistics accurate. | ||
| * Execute `auto analyze` automatically. Lower the threshold value of `analyze ratio`, increase the frequency of information collection, and set the start and end time of the execution. See the following examples: | ||
| * `set global tidb_auto_analyze_ratio=0.2;` | ||
| * `set global tidb_auto_analyze_start_time='00:00 +0800';` | ||
| * `set global tidb_auto_analyze_end_time='06:00 +0800';` | ||
| * Bind the execution plan | ||
| * Modify the application SQL statements and execute `use index` to consistently use the index of the column. | ||
| * In 3.0 versions, you do not need to modify the application SQL statements. Use `create global binding` to create the binding SQL statement of `force index`. | ||
| * In 4.0 versions, [SQL Plan Management](/sql-plan-management.md) is supported, which avoids the performance decrease caused by unstable execution plans. | ||
|
|
||
| ### PD anomalies | ||
|
|
||
| #### Phenomenon | ||
|
|
||
| There is an abnormal increase of the `wait duration` metric for the PD TSO. This metric represents the duration of waiting for PD to return requests. | ||
|
|
||
| #### Possible reasons | ||
|
|
||
| * Disk issue. The disk where the PD node is located has full I/O load. Investigate whether PD is deployed with other components with high I/O demand and the health of the disk. You can verify the cause by viewing the monitor metrics in **Grafana** -> **disk performance** -> **latency**/**load**. You can also use the FIO tool to run a check on the disk if necessary. | ||
|
|
||
| * Network issues between PD peers. The PD log shows `lost the TCP streaming connection`. You need to check whether there is a problem with the network between PD nodes and verify the cause by viewing `round trip` in the monitor **Grafana** -> **PD** -> **etcd**. | ||
|
|
||
| * High server load. The log shows `server is likely overloaded`. | ||
|
|
||
| * PD cannot elect a Leader: The PD log shows `lease is not expired`. [This issue](https://github.com/etcd-io/etcd/issues/10355) has been fixed in v3.0.x and v2.1.19. | ||
|
|
||
| * The leader election is slow. The Region loading duration is long. You can check this issue by running `grep "regions cost"` in the PD log. If the result is in seconds, such as `load 460927 regions cost 11.77099s`, it means the Region loading is slow. You can enable the `region storage` feature in v3.0 by setting `use-region-storage` to `true`, which significantly reduce the Region loading duration. | ||
|
|
||
| * The network issue between TiDB and PD. Check whether the network from TiDB to PD Leader is running normally by accessing the monitor **Grafana** -> **blackbox_exporter** -> **ping latency**. | ||
|
|
||
| * PD reports the `FATAL` error, and the log shows `range failed to find revision pair`. This issue has been fixed in v3.0.8 ([#2040](https://github.com/pingcap/pd/pull/2040)). | ||
|
|
||
| * When the `/api/v1/regions` interface is used, too many Regions might cause PD OOM. This issue has been fixed in v3.0.8 ([#1986](https://github.com/pingcap/pd/pull/1986)). | ||
|
|
||
| * PD OOM during the rolling upgrade. The size of gRPC messages is not limited, and the monitor shows that `TCP InSegs` is relatively large. This issue has been fixed in v3.0.6 ([#1952](https://github.com/pingcap/pd/pull/1952)). | ||
|
|
||
| * PD panics. [Report a bug](https://github.com/tikv/pd/issues/new?labels=kind/bug&template=bug-report.md). | ||
|
|
||
| * Other causes. Get goroutine by running `curl http://127.0.0.1:2379/debug/pprof/goroutine?debug=2` and [report a bug](https://github.com/pingcap/pd/issues/new?labels=kind%2Fbug&template=bug-report.md). | ||
|
|
||
| ### TiKV anomalies | ||
|
|
||
| #### Phenomenon | ||
|
|
||
| The `KV Cmd Duration` metric in the monitor increases abnormally. This metric represents the duration between the time that TiDB sends a request to TiKV and the time that TiDB receives the response. | ||
|
|
||
| #### Possible reasons | ||
|
|
||
| * Check the `gRPC duration` metric. This metric represents the total duration of a gRPC request in TiKV. You can find out the potential network issue by comparing `gRPC duration` of TiKV and `KV duration` of TiDB. For example, the gRPC duration is short but the KV duration of TiDB is long, which indicates that the network latency between TiDB and TiKV might be high, or that the NIC bandwidth between TiDB and TiKV is fully occupied. | ||
|
|
||
| * Re-election because TiKV is restarted. | ||
| * After TiKV panics, it is pulled up by `systemd` and runs normally. You can check whether panic has occurred by viewing the TiKV log. Because this issue is unexpected, [report a bug](https://github.com/tikv/tikv/issues/new?template=bug-report.md) if it happens. | ||
| * TiKV is stopped or killed by a third party and then pulled up by `systemd`. Check the cause by viewing `dmesg` and the TiKV log. | ||
| * TiKV is OOM, which causes restart. | ||
| * TiKV is hung because of dynamically adjusting `THP` (Transparent Hugepage). | ||
|
|
||
| * Check monitor: TiKV RocksDB encounters write stall and thus results in re-election. You can check if the monitor **Grafana** -> **TiKV-details** -> **errors** shows `server is busy`. | ||
|
|
||
| * Re-election because of network isolation. | ||
|
|
||
| * If the `block-cache` configuration is too large, it might cause TiKV OOM. To verify the cause of the problem, check the `block cache size` of RocksDB by selecting the corresponding instance in the monitor **Grafana** -> **TiKV-details**. Meanwhile, check whether the `[storage.block-cache] capacity = # "1GB"`parameter is set properly. By default, TiKV's `block-cache` is set to `45%` of the total memory of the machine. You need to explicitly specify this parameter when you deploy TiKV in the container, because TiKV obtains the memory of the physical machine, which might exceed the memory limit of the container. | ||
|
|
||
| * Coprocessor receives many large queries and returns a large volume of data. gRPC fails to send data as quickly as the coprocessor returns data, which results in OOM. To verify the cause, you can check whether `response size` exceeds the `network outbound` traffic by viewing the monitor **Grafana** -> **TiKV-details** -> **coprocessor overview**. | ||
|
|
||
| ### Bottleneck of a single TiKV thread | ||
|
|
||
| There are some single threads in TiKV that might become the bottleneck. | ||
|
|
||
| * Too many Regions in a TiKV instance causes a single gRPC thread to be the bottleneck (Check the **Grafana** -> **TiKV-details** -> **Thread CPU/gRPC CPU Per Thread** metric). In v3.x or later versions, you can enable `Hibernate Region` to resolve the issue. | ||
| * For versions earlier than v3.0, when the raftstore thread or the apply thread becomes the bottleneck (**Grafana** -> **TiKV-details** -> **Thread CPU/raft store CPU** and **Async apply CPU** metrics exceed `80%`), you can scale out TiKV (v2.x) instances or upgrade to v3.x with multi-threading. | ||
|
|
||
| ### CPU load increases | ||
|
|
||
| #### Phenomenon | ||
|
|
||
| The usage of CPU resources becomes the bottleneck. | ||
|
|
||
| #### Possible reasons | ||
|
|
||
| * Hotspot issue | ||
| * High overall load. Check the slow queries and expensive queries of TiDB. Optimize the executing queries by adding indexes or executing queries in batches. Another solution is to scale out the cluster. | ||
|
|
||
| ## Other causes | ||
|
|
||
| ### Cluster maintenance | ||
|
|
||
| Most of each online cluster has three or five nodes. If the machine to be maintained has the PD component, you need to determine whether the node is the leader or the follower. Disabling a follower has no impact on the cluster operation. Before disabling a leader, you need to switch the leadership. During the leadership change, performance jitter of about 3 seconds will occur. | ||
|
|
||
| ### Minority of replicas are offline | ||
|
|
||
| By default, each TiDB cluster has three replicas, so each Region has three replicas in the cluster. These Regions elect the leader and replicate data through the Raft protocol. The Raft protocol ensures that TiDB can still provide services without data loss even when the nodes (that are fewer than half of replicas) fail or are isolated. For the cluster with three replicas, the failure of one node might cause performance jitter but the usability and correctness in theory are not affected. | ||
|
|
||
| ### New indexes | ||
|
|
||
| Creating indexes consumes a huge amount of resources when TiDB scans tables and backfills indexes. Index creation might even conflict with the frequently updated fields, which affects the application. Creating indexes on a large table often takes a long time, so you must try to balance the index creation time and the cluster performance (for example, creating indexes at the off-peak time). | ||
|
|
||
| **Parameter adjustment:** | ||
|
|
||
| Currently, you can use `tidb_ddl_reorg_worker_cnt` and `tidb_ddl_reorg_batch_size` to dynamically adjust the speed of index creation. Usually, the smaller the values, the smaller the impact on the system, with longer execution time though. | ||
|
|
||
| In general cases, you can first keep their default values (`4` and `256`), observe the resource usage and response speed of the cluster, and then increase the value of `tidb_ddl_reorg_worker_cnt` to increase the concurrency. If no obvious jitter is observed in the monitor, increase the value of `tidb_ddl_reorg_batch_size`. If the columns involved in the index creation are frequently updated, the many resulting conflicts will cause the index creation to fail and be retried. | ||
|
|
||
| In addition, you can also set the value of `tidb_ddl_reorg_priority` to `PRIORITY_HIGH` to prioritize the index creation and speed up the process. But in the general OLTP system, it is recommended to keep its default value. | ||
|
|
||
| ### High GC pressure | ||
|
|
||
| The transaction of TiDB adopts the Multi-Version Concurrency Control (MVCC) mechanism. When the newly written data overwrites the old data, the old data is not replaced, and both versions of data are stored. Timestamps are used to mark different versions. The task of GC is to clear the obsolete data. | ||
|
|
||
| * In the phase of Resolve Locks, a large amount of `scan_lock` requests are created in TiKV, which can be observed in the gRPC-related metrics. These `scan_lock` requests call all Regions. | ||
| * In the phase of Delete Ranges, a few (or no) `unsafe_destroy_range` requests are sent to TiKV, which can be observed in the gRPC-related metrics and the **GC tasks** panel. | ||
| * In the phase of Do GC, each TiKV by default scans the leader Regions on the machine and performs GC to each leader, which can be observed in the **GC tasks** panel. |