Add new Index advisor user doc #19867

qw4990 · 2025-01-02T08:11:48Z

First-time contributors' checklist

I've signed Contributor License Agreement that's required for repo owners to accept my contribution.

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions.

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

Oreoxmt · 2025-01-02T08:18:22Z

@qw4990 Could you please invite a tech reviewer? Thanks!

index-advisor.md

ti-chi-bot · 2025-01-02T08:38:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from oreoxmt, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-01-05T09:52:10Z

@qw4990: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-verify	`e9126e0`	link	true	`/test pull-verify`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

songrijie · 2025-01-07T07:15:58Z

index-advisor.md

+
+# Recommend Index command
+
+The SQL syntax for the new SQL command `Recommend Index` is shown below. The command can be used for a single SQL query (`For` option) or for a workload (`Run` option). The comamnd also supports setting options for subsequent runs of the command.


Suggested change

The SQL syntax for the new SQL command `Recommend Index` is shown below. The command can be used for a single SQL query (`For` option) or for a workload (`Run` option). The comamnd also supports setting options for subsequent runs of the command.

SQL command `RECOMMEND INDEX` is introduced for index advisor tasks. Sub command `RUN` explores historical workloads and saves the recommendations in system tables. With option `FOR`, the command targets particular SQL statement even if it was not executed in the past. The command also accepts extra options for advance control.

songrijie · 2025-01-07T07:22:27Z

index-advisor.md

+The SQL syntax for the new SQL command `Recommend Index` is shown below. The command can be used for a single SQL query (`For` option) or for a workload (`Run` option). The comamnd also supports setting options for subsequent runs of the command.
+
+```sql
+Recommend Index [Run | For <SQL> | <Options>]


Suggested change

Recommend Index [Run | For <SQL> | <Options>]

Recommend Index Run [ For <SQL> ] [<Options>]

songrijie · 2025-01-07T07:28:09Z

index-advisor.md

+
+```sql
+mysql> CREATE TABLE t(a int, b int, c int);
+mysql> RECOMMEND INDEX RUN for "select a, b from t where a=1 and b=1";


Does it command accept "\G" as delimiter? column-style printing improves readability for single output.

songrijie · 2025-01-07T07:40:22Z

index-advisor.md

+mysql> CREATE TABLE t(a int, b int, c int);
+mysql> RECOMMEND INDEX RUN for "select a, b from t where a=1 and b=1";
+----------+-------+------------+---------------+------------+----------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+---------------------------------+
+| database | table | index_name | index_columns | index_size | reason                                                                                                                           | top_impacted_query                                                                            | create_index_statement          |


If "index_size" is not exact value, better to rename it with "est_index_size".

It's recommended to append meaningful string as suffix of index name, so that we are able to understand it's from index advisor. Name conflict can be avoid as well. For example "idx_..._<trailing 8 chars in plan digest>

index_size is the maximum number of columns an index can have. May be we can rename to reflect that, say "max_index_columns".

songrijie · 2025-01-07T07:44:42Z

index-advisor.md

+----------+-------+------------+---------------+------------+----------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+---------------------------------+
+| database | table | index_name | index_columns | index_size | reason                                                                                                                           | top_impacted_query                                                                            | create_index_statement          |
+----------+-------+------------+---------------+------------+----------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+---------------------------------+
+| test     | t     | idx_a_b    | a,b           | 19872      | Column [a b] appear in Equal or Range Predicate clause(s) in query: select `a` , `b` from `test` . `t` where `a` = ? and `b` = ? | [{"Query":"SELECT `a`,`b` FROM `test`.`t` WHERE `a` = 1 AND `b` = 1","Improvement":0.999994}] | CREATE INDEX idx_a_b ON t(a,b); |


We shall explain how we decide "Improvement" later in this section.

songrijie · 2025-01-07T07:55:43Z

index-advisor.md

+
+```sql
+Recommend Index Set <option> = <value>;
+Recommend Index Show;


Can we make this "Recommend Index Show Options"? We'd better reserve "Recommend Index Show " for future usage.

Agree since it is only used to show options.

songrijie · 2025-01-07T07:57:24Z

index-advisor.md

+
+Here are some current limitations of the index recommendation feature, which we plan to address in the future:
+1. It does not support prepared statements, meaning `RECOMMEND INDEX RUN` cannot recommend indexes for queries executed through the `Prepare` and `Execute` protocol.
+2. It does not provide recommendations for deleting indexes. We need to merge the removing index logic (see below) to the `RECOOEND` comamnd in the future. 


Suggested change

2. It does not provide recommendations for deleting indexes. We need to merge the removing index logic (see below) to the `RECOOEND` comamnd in the future.

2. It does not provide recommendations for deleting indexes. We need to merge the removing index logic (see below) to the `RECOMMEND` command in the future.

Typo

songrijie · 2025-01-07T08:11:50Z

index-advisor.md

+3. A UI for the Index Advisor will be available in the future.
+
+# Removing Unused Indexes
+TiDB provides two system views/tables to help users identify inactive indexes in their workload. Users can either mark such indexes as invisible as a transitional state before dropping them or drop them right away. 


Suggested change

TiDB provides two system views/tables to help users identify inactive indexes in their workload. Users can either mark such indexes as invisible as a transitional state before dropping them or drop them right away.

For v8.0 or higher, TiDB provides two system views/tables to help users identify inactive indexes in their workload. Users can manage to drop these indexes to save the storage and overhead caused. For online systems, it's highly recommended to make the target indexes invisible and observe the impact for one business cycle before dropping them completely.

songrijie · 2025-01-07T08:20:56Z

index-advisor.md

+## View sys.schema_unused_indexes
+
+The `sys.schema_unused_indexes` view identifies indexes that have not been used since the startup of all TiDB instances. The view is defined based on system tables that have schema, table and column information. The view provides the full specification for the index including index, table and schema names. Users can query this view and decide on making indexes invisible or deleting them. 
+


Suggested change

> **Warning:**

>

> As this view shows the unused indexes since last startup of all TiDB instances, please make sure the TiDB instances are alive long enough. Otherwise, it could show false candidates in case certain workloads are not included. SQL `select START_TIME,UPTIME from INFORMATION_SCHEMA.CLUSTER_INFO where TYPE='tidb';` helps identify the ages of all TiDB instances.

songrijie · 2025-01-07T08:31:14Z

index-advisor.md

+
+## View sys.schema_unused_indexes
+
+The `sys.schema_unused_indexes` view identifies indexes that have not been used since the startup of all TiDB instances. The view is defined based on system tables that have schema, table and column information. The view provides the full specification for the index including index, table and schema names. Users can query this view and decide on making indexes invisible or deleting them. 


Suggested change

The `sys.schema_unused_indexes` view identifies indexes that have not been used since the startup of all TiDB instances. The view is defined based on system tables that have schema, table and column information. The view provides the full specification for the index including index, table and schema names. Users can query this view and decide on making indexes invisible or deleting them.

The [`sys.schema_unused_indexes`](/sys-schema/sys-schema-unused-indexes.md) view identifies indexes that have not been used since the startup of all TiDB instances. The view is defined based on system tables that have schema, table and column information. The view provides the full specification for the index including index, table and schema names. Users can query this view and decide on making indexes invisible or deleting them.

songrijie · 2025-01-07T08:34:24Z

index-advisor.md

+
+## View information_schema.tidb_index_usage
+
+This table provides metrics like access patterns, last access time, and rows accessed. Below, we show SQL query recommendations on how to identify unused or inefficient indexes based on this table. 


Suggested change

This table provides metrics like access patterns, last access time, and rows accessed. Below, we show SQL query recommendations on how to identify unused or inefficient indexes based on this table.

[`information_schema.tidb_index_usage`](/information-schema/information-schema-tidb-indexes.md) provides metrics including selectivity buckets, last access time, and rows accessed. Below example shows the queries to identify unused or inefficient indexes based on this table.

songrijie · 2025-01-07T08:39:37Z

index-advisor.md

+WHERE last_access_time IS NULL
+  OR last_access_time < NOW() - INTERVAL 30 DAY;
+
+-- Find indexes with low efficiency


Suggested change

-- Find indexes with low efficiency

-- Find the indexes that are always scanned with over 50% of total records.

songrijie · 2025-01-07T08:40:18Z

index-advisor.md

+WHERE last_access_time IS NOT NULL AND percentage_access_0 + percentage_access_0_1 + percentage_access_1_10 + percentage_access_10_20 + percentage_access_20_50 = 0;
+```
+
+Users should be aware that the data in `tidb_index_usage` may be delayed by up to 5 minutes, and the usage data is reset whenever a TiDB node restarts. Additionally, index usage is only recorded if the table has valid statistics.


Suggested change

Users should be aware that the data in `tidb_index_usage` may be delayed by up to 5 minutes, and the usage data is reset whenever a TiDB node restarts. Additionally, index usage is only recorded if the table has valid statistics.

> **Note:**

>

> Users should be aware that the data in `tidb_index_usage` may be delayed by up to 5 minutes, and the usage data is reset whenever a TiDB node restarts. Additionally, index usage is only recorded if the table has valid statistics.

qw4990 added 2 commits January 2, 2025 16:09

fixup

e5fcdd2

fixup

a9aaf23

ti-chi-bot bot added missing-translation-status This PR does not have translation status info. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 2, 2025

fixup

e9126e0

Oreoxmt self-assigned this Jan 2, 2025

Oreoxmt self-requested a review January 2, 2025 08:16

Oreoxmt added translation/doing This PR's assignee is translating this PR. v9.0 labels Jan 2, 2025

ti-chi-bot bot removed the missing-translation-status This PR does not have translation status info. label Jan 2, 2025

Oreoxmt added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2025

Oreoxmt added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Jan 2, 2025

Oreoxmt reviewed Jan 2, 2025

View reviewed changes

index-advisor.md Show resolved Hide resolved

qw4990 requested review from songrijie and ghazalfamilyusa January 2, 2025 08:37

fixup

c60aaff

songrijie reviewed Jan 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new Index advisor user doc #19867

Add new Index advisor user doc #19867

qw4990 commented Jan 2, 2025 •

edited

Loading

Oreoxmt commented Jan 2, 2025

ti-chi-bot bot commented Jan 2, 2025

ti-chi-bot bot commented Jan 5, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025

ghazalfamilyusa Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025

ghazalfamilyusa Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025 •

edited

Loading

songrijie Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025

songrijie Jan 7, 2025


		# Recommend Index command

		The SQL syntax for the new SQL command `Recommend Index` is shown below. The command can be used for a single SQL query (`For` option) or for a workload (`Run` option). The comamnd also supports setting options for subsequent runs of the command.

	The SQL syntax for the new SQL command `Recommend Index` is shown below. The command can be used for a single SQL query (`For` option) or for a workload (`Run` option). The comamnd also supports setting options for subsequent runs of the command.
	SQL command `RECOMMEND INDEX` is introduced for index advisor tasks. Sub command `RUN` explores historical workloads and saves the recommendations in system tables. With option `FOR`, the command targets particular SQL statement even if it was not executed in the past. The command also accepts extra options for advance control.

	Recommend Index [Run \| For <SQL> \| <Options>]
	Recommend Index Run [ For <SQL> ] [<Options>]

	2. It does not provide recommendations for deleting indexes. We need to merge the removing index logic (see below) to the `RECOOEND` comamnd in the future.
	2. It does not provide recommendations for deleting indexes. We need to merge the removing index logic (see below) to the `RECOMMEND` command in the future.

	TiDB provides two system views/tables to help users identify inactive indexes in their workload. Users can either mark such indexes as invisible as a transitional state before dropping them or drop them right away.
	For v8.0 or higher, TiDB provides two system views/tables to help users identify inactive indexes in their workload. Users can manage to drop these indexes to save the storage and overhead caused. For online systems, it's highly recommended to make the target indexes invisible and observe the impact for one business cycle before dropping them completely.

		## View sys.schema_unused_indexes

		The `sys.schema_unused_indexes` view identifies indexes that have not been used since the startup of all TiDB instances. The view is defined based on system tables that have schema, table and column information. The view provides the full specification for the index including index, table and schema names. Users can query this view and decide on making indexes invisible or deleting them.

+> **Warning:**
+>
+> As this view shows the unused indexes since last startup of all TiDB instances, please make sure the TiDB instances are alive long enough. Otherwise, it could show false candidates in case certain workloads are not included. SQL `select START_TIME,UPTIME from INFORMATION_SCHEMA.CLUSTER_INFO where TYPE='tidb';` helps identify the ages of all TiDB instances.


		## View information_schema.tidb_index_usage

		This table provides metrics like access patterns, last access time, and rows accessed. Below, we show SQL query recommendations on how to identify unused or inefficient indexes based on this table.

	This table provides metrics like access patterns, last access time, and rows accessed. Below, we show SQL query recommendations on how to identify unused or inefficient indexes based on this table.
	[`information_schema.tidb_index_usage`](/information-schema/information-schema-tidb-indexes.md) provides metrics including selectivity buckets, last access time, and rows accessed. Below example shows the queries to identify unused or inefficient indexes based on this table.

	-- Find indexes with low efficiency
	-- Find the indexes that are always scanned with over 50% of total records.

-Users should be aware that the data in `tidb_index_usage` may be delayed by up to 5 minutes, and the usage data is reset whenever a TiDB node restarts. Additionally, index usage is only recorded if the table has valid statistics.
+> **Note:**
+>
+> Users should be aware that the data in `tidb_index_usage` may be delayed by up to 5 minutes, and the usage data is reset whenever a TiDB node restarts. Additionally, index usage is only recorded if the table has valid statistics.

Add new Index advisor user doc #19867

Are you sure you want to change the base?

Add new Index advisor user doc #19867

Conversation

qw4990 commented Jan 2, 2025 • edited Loading

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

Oreoxmt commented Jan 2, 2025

ti-chi-bot bot commented Jan 2, 2025

ti-chi-bot bot commented Jan 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

songrijie Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qw4990 commented Jan 2, 2025 •

edited

Loading

songrijie Jan 7, 2025 •

edited

Loading