Skip to content

Commit

Permalink
Preview PR pingcap/docs#19899 and this preview is triggered from commit
Browse files Browse the repository at this point in the history
  • Loading branch information
Docsite Preview Bot committed Jan 6, 2025
1 parent dba8f8b commit 1b7d27a
Show file tree
Hide file tree
Showing 3 changed files with 20 additions and 32 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -83,17 +83,17 @@ TiDB v8.5.0 introduces multiple enhancements to mitigate the impact of cloud dis
### Test results
The failover time of the IO latency jitter is 30% shorter, and P99/999 latency is reduced by 70% or more.
Failovers are now available in multiple IO delay scenarios, and P99/999 latency during impacts is reduced by up to 98%.
Tabel Preview:
The following test results show the performance with and without IO latency jitter improvement:
<table>
<thead>
<tr>
<th rowspan="2">Workload description</th>
<th colspan="2">Failover time</th>
<th colspan="2">Maximum latency during impact (P999)</th>
<th colspan="2">Maximum latency during impact (P99)</th>
<th colspan="2">Maximum latency during impacts (P999)</th>
<th colspan="2">Maximum latency during impacts (P99)</th>
</tr>
<tr>
<th>Current</th>
Expand All @@ -116,7 +116,7 @@ Tabel Preview:
</tr>
<tr>
<td>IO delay of 5 ms lasts for 10 mins</td>
<td>2mins</td>
<td>2 mins</td>
<td>Failover not available</td>
<td>37.9 ms</td>
<td>462 ms</td>
Expand All @@ -125,7 +125,7 @@ Tabel Preview:
</tr>
<tr>
<td>IO delay of 10 ms lasts for 10 mins</td>
<td>3mins</td>
<td>3 mins</td>
<td>Failover not available</td>
<td>69 ms</td>
<td>3 s</td>
Expand All @@ -134,7 +134,7 @@ Tabel Preview:
</tr>
<tr>
<td>IO delay of 50 ms lasts for 10 mins</td>
<td>3mins</td>
<td>3 mins</td>
<td>Failover not available</td>
<td>1.36 s</td>
<td>13.2 s</td>
Expand All @@ -143,7 +143,7 @@ Tabel Preview:
</tr>
<tr>
<td>IO delay of 100 ms lasts for 10 mins</td>
<td>3mins</td>
<td>3 mins</td>
<td>Failover not available</td>
<td>7.53 s</td>
<td>32 s</td>
Expand All @@ -153,29 +153,17 @@ Tabel Preview:
</tbody>
</table>
- Test results without IO latency jitter improvement
| Workload | Failover time | QPS drop rate | Maximum latency (P999) during failover | Maximum latency (P99) during failover |
| --- | --- | --- | --- | --- |
| IO delay of 1 s lasts for 10 mins | 4 mins | 99% | 1 min | 56 s |
| IO delay of 500 ms lasts for 10 mins | 4 mins | 99% | 54 s | 7.8 s |
| IO delay of 100 ms lasts for 10 mins | Failover not achieved | 99% | 32 s | 26 s |
| IO delay of 50 ms lasts for 10 mins | Failover not achieved | 97% | 13.2 s | 6.7 s |
| IO delay of 10 ms lasts for 10 mins | Failover not achieved | 94% | 3 s | 1.45 s |
| IO delay of 5 ms lasts for 10 mins | Failover not achieved | 81% | 462 ms | 246 ms |
| IO delay of 2 ms lasts for 10 mins | Failover not achieved | 38% | 232 ms | 22.9 ms |
- Test results with IO latency jitter improvement
| Workload | Failover time | QPS drop rate | Maximum latency (P999) during failover | Maximum latency (P99) during failover |
| --- | --- | --- | --- | --- |
| IO delay of 1 s lasts for 10 mins | 3 mins | 93% | 4.66 s | 929 ms |
| IO delay of 500 ms lasts for 10 mins | 2 mins | 92% | 7.22 s | 894 ms |
| IO delay of 100 ms lasts for 10 mins | 3 mins | 80% | 7.53 s | 1.7 s |
| IO delay of 50 ms lasts for 10 mins | 3 mins | 53% | 1.36 s | 238 ms |
| IO delay of 10 ms lasts for 10 mins | 3 mins | 18% | 69 ms | 25 ms |
| IO delay of 5 ms lasts for 10 mins | 2 mins | 29% | 37.9 ms | 10 ms |
| IO delay of 2 ms lasts for 10 mins | Almost no effect | 1% | 14 ms | 7.9 ms |
<img src="/media/tidb-cloud/v8.5-maximum-latency-during-impacts-p999.png" width="700" alt="Maximum latency during impacts (P999)" />
<img src="/media/tidb-cloud/v8.5-maximum-latency-during-impacts-p99.png" width="700" alt="Maximum latency during impacts (P99)" />
### Further improvements
Due to the inherent risk of physical disk damage, the cloud disk jitter issue is unavoidable. To mitigate its impact, TiKV introduces a [slow node detection mechanism](https://docs.pingcap.com/tidb/v8.5/pd-scheduling-best-practices#troubleshoot-tikv-node). This mechanism uses [evict-slow-store-scheduler](https://docs.pingcap.com/tidb/v8.5/pd-control#scheduler-show--add--remove--pause--resume--config--describe) to detect and manage slow nodes, reducing the effects of cloud disk jitter.
The severity of disk jitter might also be highly related to users' workload profiles. In latency-sensitive scenarios, designing applications in conjunction with TiDB features can further minimize the impact of IO jitter on applications. For example, in read-heavy and latency-sensitive environments, adjusting the [tikv_client_read_timeout](/system-variables.md#tikv_client_read_timeout-new-in-v740) system variable according to latency requirements and using stale reads or follower reads can enable faster failover retries to other replica peers for KV requests sent from TiDB. This reduces the impact of IO jitter on a single TiKV node and helps improve query latency. Note that the effectiveness of this feature depends on the workload profile, which should be evaluated before implementation.
Additionally, cloud users can reduce the probability of jitter by choosing cloud disks with higher performance.
## Batch processing
Expand Down Expand Up @@ -262,4 +250,4 @@ TiKV scaling performance improves by over 40%, and TiKV node scaling out duratio
In addition to the preceding test data, you can refer to the following benchmark results for v8.5.0 performance:
- [TPC-C performance test report](/tidb-cloud/v8.5-performance-benchmarking-with-tpcc.md)
- [Sysbench performance test report](/tidb-cloud/v8.5-performance-benchmarking-with-sysbench.md)
- [Sysbench performance test report](/tidb-cloud/v8.5-performance-benchmarking-with-sysbench.md)

0 comments on commit 1b7d27a

Please sign in to comment.