br: add metrics description for snapshot restore and PITR #18516
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -484,6 +484,45 @@ This section provides a detailed description of these key metrics on the **TiKV- | |
| - Initial Scanning Trigger Reason: The reason for triggering incremental scanning | ||
| - Region Checkpoint Key Putting: The number of checkpoint operations logged to the PD | ||
|
|
||
| ### Snapshot restore | ||
D3Hunter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - Import CPU Utilization: CPU utilization aggregated by sst importer. | ||
| - Import Thread Count: number of threads used by sst importer. | ||
| - Import Errors: error counts during sst import. | ||
| - Import RPC Duration: time spent for various RPC calls in sst importer. | ||
| - Import RPC Ops: number of total RPC calls in sst importer. | ||
| - Import RPC Count: number of inflight RPC calls in sst importer. | ||
| - Import Write/Download RPC Duration: RPC time for write/download in sst importer. | ||
| - Import Wait Duration: time spent on downloading task waiting in queue for execution. | ||
| - Import Read SST Duration: time spent on reading from external storage of a file and download it to TiKV. | ||
| - Import Rewrite SST Duration: time spent on rewriting SST based on rewrite rules. | ||
| - Import Ingest RPC Duration: time spent on handling ingest RPC request on TiKV. | ||
| - Import Ingest SST Duration: time spent on ingesting SST into RocksDB. | ||
| - Import Ingest SST Bytes: number of bytes ingested. | ||
| - Import Download SST Throughput: SST download throughput in bytes per second. | ||
| - TTL Expired: number of expired items after TTL in backup files. | ||
|
There was a problem hiding this comment. we can also skip this item since it's raw_kv related, and it also needs to be filtered out from snapshot restore panel. |
||
| - cloud request: number of request to cloud providers. | ||
|
|
||
| ### Point-in-Time Restore | ||
Oreoxmt marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - CPU Usage: CPU utilization by PITR. | ||
| - P99 RPC Duration: 99 percentile of the RPC requests time. | ||
| - Import RPC Ops: number of total RPC calls in sst importer. | ||
| - Import RPC Count: number of inflight RPC calls in sst importer. | ||
D3Hunter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - Cache Events: number of events on file cache during sst import. | ||
| - Overall RPC Duration: time spent on RPC calls. | ||
| - Read File into Memory Duration: time spent on downloading files from external storage and loaded in to memory. | ||
| - Queuing Time: time spent on waiting to get scheduled on a thread. | ||
| - Apply Request Throughput: Apply request rate in bytes. | ||
| - Downloaded File Size: downloaded file size in bytes. | ||
| - Apply Batch Size: number of bytes for applying to Raft store in one batch. | ||
| - Blocked by Concurrency Time: time spent on waiting to get executed due to concurrency constraint. | ||
| - Apply Request Speed: speed of applying request to Raft store. | ||
| - Cached File in Memory: files cached by the applying requests of importer. | ||
| - Engine Requests Unfinished: number of pending requests to Raft store. | ||
| - Apply Time: time spent on writing data to the Raft store. | ||
| - Raft Store Memory Usage: memory usage for Raft store. | ||
Oreoxmt marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| > **Note:** | ||
|
There was a problem hiding this comment. @Tristan1900 @BornChanger Could you please help confirm whether the following metrics belong to Point-in-Time Restore or Log Backup? docs/grafana-tikv-dashboard.md Lines 525 to 534 in 2528780 There was a problem hiding this comment. ah good catch, it should be log backup, I thought it was a note for the entire dashboard. |
||
| > | ||
| > The following monitoring metrics all use TiDB nodes as their data source, but they have some impact on the log backup process. Therefore, they are placed in the **TiKV Details** dashboard for ease of reference. TiKV actively pushes progress most of the time, but it is normal for some of the following monitoring metrics to occasionally not have sampled data. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
most are for import, why call it
Snapshot restoreThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The purpose of this PR is to add metrics description for restore, as a new member in the team I find it myself confused when looking at sst importer where it's actually used by restore. I feel like customer might have the same doubt when looking the metrics, so changing the name to be snapshot restore.
I have brought up to the team that we probably should have an abstraction of the restore on the TiKV path, where internally it still calls sst importer. It can be more explicit and meaningful to users this way, and inside of it we can rename or create new metrics to be decoupled with sst importer.
let me know if that makes sense!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm