Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grafana: supply the explanation on the metrics of Flow-Control in TiKV. #15720

Merged
merged 8 commits into from
Dec 18, 2023
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions grafana-tikv-dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,21 @@ This section provides a detailed description of these key metrics on the **TiKV-

![TiKV Dashboard - Storage metrics](/media/tikv-dashboard-storage.png)

### Flow Control

- Scheduler flow: The scheduler traffic on each TiKV instance in real time.
- Scheduler discard ratio: The rejection ratio of scheduler requests on each TiKV instance. If this ratio is not 0, it indicates that flow control exists. When `Compaction pending bytes` exceed its threshold, TiKV linearly increases the `Scheduler Discard Ratio` based on the exceeded portion. The client will retry rejected requests automatically.
qiancai marked this conversation as resolved.
Show resolved Hide resolved
- Throttle duration: The blocked duration for the execution of the scheduler requests when flow control is triggered due to too many L0 files. If this metric has values, it indicates that flow control exists.
- Scheduler throttled CF: The CF that triggers RocksDB throttling when the flow control threshold is reached
- Flow controller actions: The actions that trigger RocksDB throttling when the flow control threshold is reached
- Flush/L0 flow: The traffic of flush and L0 compaction for different CFs of RocksDB on each TiKV instance
qiancai marked this conversation as resolved.
Show resolved Hide resolved
- Flow control factors: The factors related to triggering RocksDB throttling.
- Compaction pending bytes: The size of the RocksDB data awaiting compaction in real time on each TiKV instance.
- Txn command throttled duration: The blocked duration for commands related to transactions due to throttling. Under normal circumstances, this metric is 0.
- Non-txn command throttled duration: The blocked duration for other commands due to throttling. Under normal circumstances, this metric is 0.
Comment on lines +198 to +199
Copy link
Collaborator

@hfxsd hfxsd Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

最后这两个,Txn command throttled duration、Non-txn command throttled duration 截图中没有

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


![TiKV Dashboard - Flow Control metrics](/media/tikv-dashboard-flow-control.png)

### Scheduler

- Scheduler stage total: The number of commands at each stage per second. There should not be a lot of errors in a short time.
Expand Down
Binary file added media/tikv-dashboard-flow-control.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading