diff --git a/docs/design/2018-07-19-row-format.md b/docs/design/2018-07-19-row-format.md index 48b5f274064f0..97928a6f2fcbb 100644 --- a/docs/design/2018-07-19-row-format.md +++ b/docs/design/2018-07-19-row-format.md @@ -16,7 +16,7 @@ In the current row format, each column is encoded with the column ID followed by The new format is defined as follows: -![row-format](./row-format.jpeg) +![row-format](./imgs/row-format.jpeg) * 1 byte codec version diff --git a/docs/design/2018-08-10-restore-dropped-table.md b/docs/design/2018-08-10-restore-dropped-table.md index 2210b565920fe..e6dd7c2a8861d 100644 --- a/docs/design/2018-08-10-restore-dropped-table.md +++ b/docs/design/2018-08-10-restore-dropped-table.md @@ -1,5 +1,5 @@ -# Proposal: +# Proposal: A new command to restore dropped table - Author(s): [winkyao](https://github.com/winkyao) - Last updated: 2018-08-10 diff --git a/docs/design/2018-08-29-new-planner.md b/docs/design/2018-08-29-new-planner.md index 7b5f27c0f1dfd..35875c7ee4bfb 100644 --- a/docs/design/2018-08-29-new-planner.md +++ b/docs/design/2018-08-29-new-planner.md @@ -60,7 +60,7 @@ the future. With the concept of **Group** and **Group Expression**, all the logically equivalent expressions can be stored in a root Group. - ![group and expression](./group-and-expression.png) + ![group and expression](./imgs/group-and-expression.png) - **Transformation Rule** diff --git a/docs/design/2018-10-08-online-DDL.md b/docs/design/2018-10-08-online-DDL.md index b1127433f4487..732ced186bfc8 100644 --- a/docs/design/2018-10-08-online-DDL.md +++ b/docs/design/2018-10-08-online-DDL.md @@ -27,7 +27,7 @@ DDL jobs have multiple state changes during processing. All TiDB servers in a cl So you can simply think that each TiDB server needs two modules to ensure that this condition is met. * One is the load schema module. In order to process SQL normally, each TiDB server needs to be loaded into the latest schema within a lease. * The other one is the handle DDL job module. This module is executed only after the current TiDB server is elected to the owner. Process the job and change the schema. After the owner writes the currently changed schema state to TiKV, we can assume that all TiDB servers in the cluster that can handle SQL after waiting for 2*lease have been updated to this schema. -![Figure 1: Structure flow chart](./structure-flow-chart.png) +![Figure 1: Structure flow chart](./imgs/structure-flow-chart.png)
Figure 1: Structure flow chart
 #### Initial configuration @@ -55,7 +55,7 @@ There are many operations in DDL that need to delete data, such as drop table re ### Parallel flow Currently, only the add index operation in TiDB takes a long time to execute, so an add index operation may block other DDL operations. In order to solve this problem preliminarily and steadily, we have implemented the parallel operation between the general operation and the add index operation between the tables. That is, between different tables, the general operation and the add index operation can be performed in parallel. -![Figure 2: Owner detail flow chart](./owner-detail-flow-chart.png) +![Figure 2: Owner detail flow chart](./imgs/owner-detail-flow-chart.png)
Figure 2: Owner detailed flow chart
 ## Optimization function diff --git a/docs/design/2018-10-22-the-column-pool.md b/docs/design/2018-10-22-the-column-pool.md index 27cb5b73c66b9..c963868e4fa1d 100644 --- a/docs/design/2018-10-22-the-column-pool.md +++ b/docs/design/2018-10-22-the-column-pool.md @@ -52,7 +52,7 @@ implemented as a last in first out stack, which helps to release the unused column in the pool. And in most cases, the required length of a new column is equal to the previous column put into the column pool. -![the column pool](./the-column-pool.png) +![the column pool](./imgs/the-column-pool.png) ## Rationale diff --git a/docs/design/2019-04-11-indexmerge.md b/docs/design/2019-04-11-indexmerge.md index a4b8189fe4500..da6390949880a 100644 --- a/docs/design/2019-04-11-indexmerge.md +++ b/docs/design/2019-04-11-indexmerge.md @@ -1,335 +1,335 @@ -# Proposal: scan a table using IndexMerge -- Author(s) : WHU -- Last updated : May 10 -- Discussion at : - - -## Abstract - -The document proposes using multiple indexes to scan a table when possible, which, in some cases, will improve performance. - -## Background - -In the present TiDB, a SQL statement with conditions that involve multiple indexed attributes only uses one of the conditions as the index filter to build the access condition, while regarding others as table filters. TiDB firstly uses index scan (one index at most) to get handles (rowid in TiDB), which are then used to get rows and check whether the rows satisfy the conditions of table filters. Some relational databases implement a table access path using multiple indexes to improve performance in some cases. - -Here is an example to explain it. The table schema is defined as: - -``` -CREATE TABLE t1 (a int, b int, c int); -CREATE INDEX t1a on t1(a); -CREATE INDEX t1b on t1(b); -CREATE INDEX t1c on t1(c); -``` - -A test SQL statement `SELECT * FROM t1 where a < 2 or b > 50` is used in the example. Currently, TiDB does a table scan and puts `a < 2 or b > 50` as a Selection on top of it. If the selectivity of `a < 2 ` and `b > 50` is low, a better approach would be using indexes on columns `a` and `b` to retrieve rows respectively, and applying a union operation on the result sets. - - -## Proposal - -In short, we can implement access paths using multiple indexes. - -### Planner - -We propose to add the following operators: - -- `IndexMergeReader / PhysicalIndexMergeReader` -- `IndexMergeLookUpReader / PhysicalIndexMergeLookUpReader`. - -Now consider the following types of queries: - -（1）Conditions in conjunctive normal form (CNF), e.g, `select * from t1 where c1 and c2 and c3 and …` - -In this form, each CNF item can be covered by a single index respectively. For example, if we have single column indexes for `t1.a`, `t1.b` and `t1.c` respectively, for SQL `select * from t1 where (a < 10 or a > 100) and b < 10 and c > 1000`, we can use all the three indexes to read the table handles. The result plan for it is: - -``` -PhysicalIndexMergeLookUpReader(IndexMergeIntersect) - IndexScan(t1a) - IndexScan(t1b) - IndexScan(t1c) - TableScan -``` - -For the CNF items not covered by any index, we take them as table filters and convert them to selections on top of the scan node. For SQL `select * from t1 where (a < 10 or c >100) and b < 10`, only item `b < 10` can function as an index access condition, so we will use a single index lookup reader. - -To compare our demo implementation against the master branch, we set up an experiment for the CNF form. The schema and test SQL form we defined are as below: - -``` -Table Schema: - CREATE TABLE T200M(c1 int, c2 int, c3 int, c4 int, c5 int, c6 int, c7 int, c8 int); - CREATE INDEX T200Ma on T200M(a); - CREATE INDEX T200Mb on T200M(b); - -Test SQL Form: - CNF-1 - SELECT * FROM T200M WHERE C1 < $1 AND C2 > $2; - CNF-2 - SELECT * FROM T200M WHERE C1 < $3 AND C2 < $4; -``` - -In the experiment, we load two million rows into table `T200M` with one to two million sequence for all columns. `$1-$4` is altered to obtain equal selectivity on `C1` and `C2`, while the difference between `CNF-1` and `CNF-2` is that `CNF-1` has no final results. The result can be seen in the following graph: - -CNF 200 - -**Note:** `SELECTIVITY`is for the single column. - -（2） Conditions in disjunctive normal form (DNF), e.g, `select * from t1 where c1 or c2 or c3 or …` - -In this form, every DNF item must be covered by a single index. If not, IndexMerge scan cannot be used. For example, SQL `select * from t1 where a > 1 or ( b >1 and b <10)` will generate a possible plan like: - -``` -PhysicalIndexMergeLookUpReader(IndexMergeUnion) - IndexScan(t1a) - IndexScan(t1b) - TableScan -``` - -To compare our demo implementation against the master branch, we set up an experiment for the DNF form. The schema and test SQL form we defined are as below: - -``` -Table Schema: - CREATE TABLE T200(a int, b int, c int); - CREATE INDEX T200a on T2OO(a); - CREATE INDEX T200b on T200(b); - -Test SQL Form: - SELECT * FROM T200 WHERE a < $1 OR b > $2; -``` -We load two million rows into table `T200` with one to two million sequence for all columns. The value of `$1` and `$2` in the test SQL form is altered to obtain the accurate selectivities. The result can be seen in the following graph: - -DNF 200 - - -The structure of `PhysicalIndexMergeLookUpReader` is designed as: - -``` -// PhysicalIndexMergeLookUpReader -type PhysicalIndexMergeLookUpReader struct { - physicalSchemaProducer - - //Follow two plans flat to construct executor pb. - IndexPlans []PhysicalPlan - TablePlans []PhysicalPlan - - indexPlans []PhysicalPlan - tablePlan PhysicalPlan - - IndexMergeType int -} -``` - - -- Field `IndexMergeType` indicates the operations on the results of multiple index scans. Possible values are: - - - 0: not an IndexMerge scan; - - 1: intersection operation on result sets, with a table scan; - - 2: intersection operation on result sets, without the table scan; - - 3: union operation on result sets; must have a table scan; - -In the first version, `PhysicalIndexMergeLookUpReader` and `PhysicalIndexMergeReader` will be implemented as one operator. - - -### IndexMergePath Generate - -For table paths, we first generate all possible IndexMergeOr paths, then possible IndexMergeIntersection paths. - -``` -type IndexMergePath struct { - IndexPath[] - tableFilters - IndexMergeType -} -``` - - -``` -GetIndexMergeUnionPaths(IndexInfos, PushdownConditions){ - var results = nil - foreach cond in PushdownConditions { - if !isOrCondition(cond) { - continue - } - args = flatten(cond,'or') - foreach arg in args { - var indexAccessPaths, imPaths - // Common index paths will be merged later in `CreateIndexMergeUnionPath` - if isAndCondition(arg) { - andArgs = flatten(arg,'and') - indexAccessPaths = buildAccessPath(andArgs, IndexInfos) - } else { - tempArgs = []{arg} - indexAccessPaths = buildAccessPath(tempArgs, IndexInfos) - } - if indexAccessPaths == nil { - imPaths = nil - break - } - imPartialPath = GetIndexMergePartialPath(IndexInfos, indexAccessPaths) - imPaths = append(imPaths, imPartialPath) - } - if imPaths != nil { - possiblePath = CreateIndexMergeUnionPath(imPaths,PushdownConditions,con,IndexInfos) - results = append(results, possiblePath) - } - } - return results -} - - -buildAccessPath(Conditions, IndexInfos){ - var results - for index in IndexInfos { - res = detachCNFCondAndBuildRangeForIndex(Conditions, index, considerDNF = true) - if res.accessCondition = nil { - continue - } - indexPath = CreateIndexAccessPath(index, res) - results = append(results, indexPath) - } - return results -} - -// This function will get a best indexPath for a condition from some alternative paths. -// Now we just take the index which has more columns. -// For exmple: -// (1) -// index1(a,b,c) index2(a,b) index3(a) -// condition: a = 1 will choose index1; a = 1 and b = 2 will also choose index1 -// (2) -// index1(a) index2(b) -// condition: a = 1 and b = 1 -// random choose??? -GetIndexMergePartialPath(IndexInfos, indexAccessPaths) { -} - -// (1) Maybe we will merge some indexPaths -// for example: index1(a) index2(b) -// condition : a < 1 or a > 2 or b < 1 or b > 10 -// imPaths will be [a<1,a>2,b<1,b>10] and we can merge it and get [a<1 or a >2 , b < 1 or b > 10] -// (2)IndexMergePath.tableFilters: -// <1> Remove a condition from PushdownConditions and the rest will be added to tableFitler. -// <2> After the merge operation, if any indexPath's tableFilter is not nil, we should add it into tableFilters - -CreateIndexMergeUnionPath(imPaths,PushdownConditions,cond,IndexInfos) { -} - -``` - - -``` -GetIndexMergeIntersectionPaths(pushDownConditions, usedConditionsInOr, indexInfos) { - var partialPaths - - if len(pushDownConditions) - len(usedConditionsInOr) < 2 { - return nil - } - tableFilters := append(tableFilters, usedConditionsInOr...) - newConsiderConditions := remove(pushDownConditions, usedConditionsInOr) - for cond in newConsiderConditions { - indexPaths = buildAccessPath([]{cond}, indexInfos) - if indexPaths == nil { - tableFiltes = append(tableFilters,cond) - continue - } - indexPath := GetIndexMergePartialPath(indexPaths,indexInfos) - partialPaths = append(partialPaths, indexPath) - } - if len(partialPaths) < 2 { - return nil - } - return CreateIndexMergeIntersectionPath(partialPaths, tableFilters) -} - -// Now, we just use all paths in partialPaths to generate a IndexMergeIntersection. -// We also need to merge possible paths. -// For example: -// index: ix1(a) -// condition： a > 1 and a < 10 -// we will get two partial paths and they all use index ix1. -// IndexMergePath.tableFilters: -// <1> tableFilters -// <2> after merge operation, if any indexPath's tableFilter is not nil, we -// should add indexPath’s tableFilter into IndexMergePath.tableFilters -CreateIndexMergeIntersectionPath(partialPaths, tableFilters) { -} - -``` - -### Executor - -The graph below illustrates an execution of IndexMerge scan. - -Execution Model - -Every index plan in `PhysicalIndexMergeLookUpReader` will start an `IndexWorker` to execute the IndexScan plan and send handles to AndOrWorker. AndOrWorker is responsible for performing set operations (and, or) to getting final handles. Then `AndOrWoker` sends final handles to `TableWokers` to get rows from TiKV. - -Here are some designs for index plans in pipeline mode to be executed without considering the order. - -(1) IndexMergeIntersection - - IndexMergeIntersection - -Take this example: Use set1 to record row IDs returned by ix1 but not sent to tableWorker. Use set2 to record the same thing for ix2. - -If new row IDs comes from ix1, first we check if it is in set2. If yes, we delete it from set2 and send it to tableWorker. Otherwise, we add it into set1. For the above figure, we use the following table to show the processing. - -| new rowid | set1 | set2 | sent to TableWorker | -| :------:| :------: | :------: | :------: | -| 2(ix1) | [2] | [ ] | [ ] | -| 1(ix2) | [2] | [1] | [ ] | -| 5(ix1) | [2,5] |[1] | [ ] | -| 5(ix2) | [2] | [1] | [5] | -| 7(ix1) | [2,7] |[1] | [ ] | -| 2(ix2) | [7] | [1] | [2] | - - - -(2) IndexMergeUnion - - -We have a structure (set) to record which row IDs are accessed. If a new row ID is returned by IndexScan, check if it is in the set. If yes, we just skip it. Otherwise, we add it into the set and send it to tableWorker. - -### Cost Model - -There are three factors to be considered for the cost model: IO, CPU, and Network, as shown below: - -- `IndexMergeType` = 1 - - IO Cost = (totalRowCount + mergedRowCount) * scanFactor - - Network Cost = (totalRowCount + mergedRowCount) * networkFactor - - Cpu Memory Cost = totalRowCount * cpuFactor + totalRowCount * memoryFactor - -- `IndexMergeType` = 2 - - IO Cost = (totalRowCount) * scanFactor - - Network Cost = totalRowCount * networkFactor - - Cpu Memory Cost = totalRowCount * cpuFactor + totalRowCount * memoryFactor - -- `IndexMergeType` = 3 - - IO Cost = (totalRowCount + mergedRowCount) * scanFactor - - Network Cost = (totalRowCount + mergedRowCount) * networkFactor - - Cpu Memory Cost = totalRowCount * cpuFactor + mergedRowCount * memoryFactor - - -**Note**: - -- totalRowCount: sum of handles collected from every index scan. - -- mergedRowCount: number of handles after set operating. - -## Compatibility - -This proposal has no effect on the compatibility. - -## Implementation Procedure - -1. Implement planner operators -1. Enhance `explain` to display the plan -3. Implement executor operators -4. Testing +# Proposal: Access a table using multiple indexes +- Author(s) : WHU +- Last updated : May 10 +- Discussion at : + + +## Abstract + +The document proposes using multiple indexes to scan a table when possible, which, in some cases, will improve performance. + +## Background + +In the present TiDB, a SQL statement with conditions that involve multiple indexed attributes only uses one of the conditions as the index filter to build the access condition, while regarding others as table filters. TiDB firstly uses index scan (one index at most) to get handles (rowid in TiDB), which are then used to get rows and check whether the rows satisfy the conditions of table filters. Some relational databases implement a table access path using multiple indexes to improve performance in some cases. + +Here is an example to explain it. The table schema is defined as: + +``` +CREATE TABLE t1 (a int, b int, c int); +CREATE INDEX t1a on t1(a); +CREATE INDEX t1b on t1(b); +CREATE INDEX t1c on t1(c); +``` + +A test SQL statement `SELECT * FROM t1 where a < 2 or b > 50` is used in the example. Currently, TiDB does a table scan and puts `a < 2 or b > 50` as a Selection on top of it. If the selectivity of `a < 2 ` and `b > 50` is low, a better approach would be using indexes on columns `a` and `b` to retrieve rows respectively, and applying a union operation on the result sets. + + +## Proposal + +In short, we can implement access paths using multiple indexes. + +### Planner + +We propose to add the following operators: + +- `IndexMergeReader / PhysicalIndexMergeReader` +- `IndexMergeLookUpReader / PhysicalIndexMergeLookUpReader`. + +Now consider the following types of queries: + +（1）Conditions in conjunctive normal form (CNF), e.g, `select * from t1 where c1 and c2 and c3 and …` + +In this form, each CNF item can be covered by a single index respectively. For example, if we have single column indexes for `t1.a`, `t1.b` and `t1.c` respectively, for SQL `select * from t1 where (a < 10 or a > 100) and b < 10 and c > 1000`, we can use all the three indexes to read the table handles. The result plan for it is: + +``` +PhysicalIndexMergeLookUpReader(IndexMergeIntersect) + IndexScan(t1a) + IndexScan(t1b) + IndexScan(t1c) + TableScan +``` + +For the CNF items not covered by any index, we take them as table filters and convert them to selections on top of the scan node. For SQL `select * from t1 where (a < 10 or c >100) and b < 10`, only item `b < 10` can function as an index access condition, so we will use a single index lookup reader. + +To compare our demo implementation against the master branch, we set up an experiment for the CNF form. The schema and test SQL form we defined are as below: + +``` +Table Schema: + CREATE TABLE T200M(c1 int, c2 int, c3 int, c4 int, c5 int, c6 int, c7 int, c8 int); + CREATE INDEX T200Ma on T200M(a); + CREATE INDEX T200Mb on T200M(b); + +Test SQL Form: + CNF-1 + SELECT * FROM T200M WHERE C1 < $1 AND C2 > $2; + CNF-2 + SELECT * FROM T200M WHERE C1 < $3 AND C2 < $4; +``` + +In the experiment, we load two million rows into table `T200M` with one to two million sequence for all columns. `$1-$4` is altered to obtain equal selectivity on `C1` and `C2`, while the difference between `CNF-1` and `CNF-2` is that `CNF-1` has no final results. The result can be seen in the following graph: + +CNF 200 + +**Note:** `SELECTIVITY`is for the single column. + +（2） Conditions in disjunctive normal form (DNF), e.g, `select * from t1 where c1 or c2 or c3 or …` + +In this form, every DNF item must be covered by a single index. If not, IndexMerge scan cannot be used. For example, SQL `select * from t1 where a > 1 or ( b >1 and b <10)` will generate a possible plan like: + +``` +PhysicalIndexMergeLookUpReader(IndexMergeUnion) + IndexScan(t1a) + IndexScan(t1b) + TableScan +``` + +To compare our demo implementation against the master branch, we set up an experiment for the DNF form. The schema and test SQL form we defined are as below: + +``` +Table Schema: + CREATE TABLE T200(a int, b int, c int); + CREATE INDEX T200a on T2OO(a); + CREATE INDEX T200b on T200(b); + +Test SQL Form: + SELECT * FROM T200 WHERE a < $1 OR b > $2; +``` +We load two million rows into table `T200` with one to two million sequence for all columns. The value of `$1` and `$2` in the test SQL form is altered to obtain the accurate selectivities. The result can be seen in the following graph: + +DNF 200 + + +The structure of `PhysicalIndexMergeLookUpReader` is designed as: + +``` +// PhysicalIndexMergeLookUpReader +type PhysicalIndexMergeLookUpReader struct { + physicalSchemaProducer + + //Follow two plans flat to construct executor pb. + IndexPlans []PhysicalPlan + TablePlans []PhysicalPlan + + indexPlans []PhysicalPlan + tablePlan PhysicalPlan + + IndexMergeType int +} +``` + + +- Field `IndexMergeType` indicates the operations on the results of multiple index scans. Possible values are: + + - 0: not an IndexMerge scan; + - 1: intersection operation on result sets, with a table scan; + - 2: intersection operation on result sets, without the table scan; + - 3: union operation on result sets; must have a table scan; + +In the first version, `PhysicalIndexMergeLookUpReader` and `PhysicalIndexMergeReader` will be implemented as one operator. + + +### IndexMergePath Generate + +For table paths, we first generate all possible IndexMergeOr paths, then possible IndexMergeIntersection paths. + +``` +type IndexMergePath struct { + IndexPath[] + tableFilters + IndexMergeType +} +``` + + +``` +GetIndexMergeUnionPaths(IndexInfos, PushdownConditions){ + var results = nil + foreach cond in PushdownConditions { + if !isOrCondition(cond) { + continue + } + args = flatten(cond,'or') + foreach arg in args { + var indexAccessPaths, imPaths + // Common index paths will be merged later in `CreateIndexMergeUnionPath` + if isAndCondition(arg) { + andArgs = flatten(arg,'and') + indexAccessPaths = buildAccessPath(andArgs, IndexInfos) + } else { + tempArgs = []{arg} + indexAccessPaths = buildAccessPath(tempArgs, IndexInfos) + } + if indexAccessPaths == nil { + imPaths = nil + break + } + imPartialPath = GetIndexMergePartialPath(IndexInfos, indexAccessPaths) + imPaths = append(imPaths, imPartialPath) + } + if imPaths != nil { + possiblePath = CreateIndexMergeUnionPath(imPaths,PushdownConditions,con,IndexInfos) + results = append(results, possiblePath) + } + } + return results +} + + +buildAccessPath(Conditions, IndexInfos){ + var results + for index in IndexInfos { + res = detachCNFCondAndBuildRangeForIndex(Conditions, index, considerDNF = true) + if res.accessCondition = nil { + continue + } + indexPath = CreateIndexAccessPath(index, res) + results = append(results, indexPath) + } + return results +} + +// This function will get a best indexPath for a condition from some alternative paths. +// Now we just take the index which has more columns. +// For exmple: +// (1) +// index1(a,b,c) index2(a,b) index3(a) +// condition: a = 1 will choose index1; a = 1 and b = 2 will also choose index1 +// (2) +// index1(a) index2(b) +// condition: a = 1 and b = 1 +// random choose??? +GetIndexMergePartialPath(IndexInfos, indexAccessPaths) { +} + +// (1) Maybe we will merge some indexPaths +// for example: index1(a) index2(b) +// condition : a < 1 or a > 2 or b < 1 or b > 10 +// imPaths will be [a<1,a>2,b<1,b>10] and we can merge it and get [a<1 or a >2 , b < 1 or b > 10] +// (2)IndexMergePath.tableFilters: +// <1> Remove a condition from PushdownConditions and the rest will be added to tableFitler. +// <2> After the merge operation, if any indexPath's tableFilter is not nil, we should add it into tableFilters + +CreateIndexMergeUnionPath(imPaths,PushdownConditions,cond,IndexInfos) { +} + +``` + + +``` +GetIndexMergeIntersectionPaths(pushDownConditions, usedConditionsInOr, indexInfos) { + var partialPaths + + if len(pushDownConditions) - len(usedConditionsInOr) < 2 { + return nil + } + tableFilters := append(tableFilters, usedConditionsInOr...) + newConsiderConditions := remove(pushDownConditions, usedConditionsInOr) + for cond in newConsiderConditions { + indexPaths = buildAccessPath([]{cond}, indexInfos) + if indexPaths == nil { + tableFiltes = append(tableFilters,cond) + continue + } + indexPath := GetIndexMergePartialPath(indexPaths,indexInfos) + partialPaths = append(partialPaths, indexPath) + } + if len(partialPaths) < 2 { + return nil + } + return CreateIndexMergeIntersectionPath(partialPaths, tableFilters) +} + +// Now, we just use all paths in partialPaths to generate a IndexMergeIntersection. +// We also need to merge possible paths. +// For example: +// index: ix1(a) +// condition： a > 1 and a < 10 +// we will get two partial paths and they all use index ix1. +// IndexMergePath.tableFilters: +// <1> tableFilters +// <2> after merge operation, if any indexPath's tableFilter is not nil, we +// should add indexPath’s tableFilter into IndexMergePath.tableFilters +CreateIndexMergeIntersectionPath(partialPaths, tableFilters) { +} + +``` + +### Executor + +The graph below illustrates an execution of IndexMerge scan. + +Execution Model + +Every index plan in `PhysicalIndexMergeLookUpReader` will start an `IndexWorker` to execute the IndexScan plan and send handles to AndOrWorker. AndOrWorker is responsible for performing set operations (and, or) to getting final handles. Then `AndOrWoker` sends final handles to `TableWokers` to get rows from TiKV. + +Here are some designs for index plans in pipeline mode to be executed without considering the order. + +(1) IndexMergeIntersection + + IndexMergeIntersection + +Take this example: Use set1 to record row IDs returned by ix1 but not sent to tableWorker. Use set2 to record the same thing for ix2. + +If new row IDs comes from ix1, first we check if it is in set2. If yes, we delete it from set2 and send it to tableWorker. Otherwise, we add it into set1. For the above figure, we use the following table to show the processing. + +| new rowid | set1 | set2 | sent to TableWorker | +| :------:| :------: | :------: | :------: | +| 2(ix1) | [2] | [ ] | [ ] | +| 1(ix2) | [2] | [1] | [ ] | +| 5(ix1) | [2,5] |[1] | [ ] | +| 5(ix2) | [2] | [1] | [5] | +| 7(ix1) | [2,7] |[1] | [ ] | +| 2(ix2) | [7] | [1] | [2] | + + + +(2) IndexMergeUnion + + +We have a structure (set) to record which row IDs are accessed. If a new row ID is returned by IndexScan, check if it is in the set. If yes, we just skip it. Otherwise, we add it into the set and send it to tableWorker. + +### Cost Model + +There are three factors to be considered for the cost model: IO, CPU, and Network, as shown below: + +- `IndexMergeType` = 1 + + IO Cost = (totalRowCount + mergedRowCount) * scanFactor + + Network Cost = (totalRowCount + mergedRowCount) * networkFactor + + Cpu Memory Cost = totalRowCount * cpuFactor + totalRowCount * memoryFactor + +- `IndexMergeType` = 2 + + IO Cost = (totalRowCount) * scanFactor + + Network Cost = totalRowCount * networkFactor + + Cpu Memory Cost = totalRowCount * cpuFactor + totalRowCount * memoryFactor + +- `IndexMergeType` = 3 + + IO Cost = (totalRowCount + mergedRowCount) * scanFactor + + Network Cost = (totalRowCount + mergedRowCount) * networkFactor + + Cpu Memory Cost = totalRowCount * cpuFactor + mergedRowCount * memoryFactor + + +**Note**: + +- totalRowCount: sum of handles collected from every index scan. + +- mergedRowCount: number of handles after set operating. + +## Compatibility + +This proposal has no effect on the compatibility. + +## Implementation Procedure + +1. Implement planner operators +1. Enhance `explain` to display the plan +3. Implement executor operators +4. Testing diff --git a/docs/design/README.md b/docs/design/README.md index 6106cc3a5de1c..a5afa94461b7b 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -33,9 +33,15 @@ Writing a design document can promote us to think deliberately and gather knowle - [Proposal: Support a Global Column Pool](./2018-10-22-the-column-pool.md) - [Proposal: Join Reorder Design v1](./2018-10-20-join-reorder-dp-v1.md) - [Proposal: Support Window Functions](./2018-10-31-window-functions.md) +- [Proposal: Access a table using multiple indexes](./2019-04-11-indexmerge.md) ### Completed - [Proposal: A new aggregate function execution framework](./2018-07-01-refactor-aggregate-framework.md) +- [Proposal: TiDB DDL architecture](./2018-10-08-online-DDL.md) - [Proposal: Infer the System Timezone of a TiDB cluster via TZ environment variable](./2018-09-10-adding-tz-env.md) +- [Proposal: Table Partition](./2018-10-19-table-partition.md) +- [Proposal: Implement View Feature](./2018-10-24-view-support.md) +- [Proposal: Support restoring SQL text from an AST tree](./2018-11-29-ast-to-sql-text.md) +- [Proposal: Support Plugin](./2018-12-10-plugin-framework.md) - [Proposal: Support Skyline Pruning](./2019-01-25-skyline-pruning.md) diff --git a/docs/design/group-and-expression.png b/docs/design/imgs/group-and-expression.png similarity index 100% rename from docs/design/group-and-expression.png rename to docs/design/imgs/group-and-expression.png diff --git a/docs/design/owner-detail-flow-chart.png b/docs/design/imgs/owner-detail-flow-chart.png similarity index 100% rename from docs/design/owner-detail-flow-chart.png rename to docs/design/imgs/owner-detail-flow-chart.png diff --git a/docs/design/row-format.jpeg b/docs/design/imgs/row-format.jpeg similarity index 100% rename from docs/design/row-format.jpeg rename to docs/design/imgs/row-format.jpeg diff --git a/docs/design/structure-flow-chart.png b/docs/design/imgs/structure-flow-chart.png similarity index 100% rename from docs/design/structure-flow-chart.png rename to docs/design/imgs/structure-flow-chart.png diff --git a/docs/design/the-column-pool.png b/docs/design/imgs/the-column-pool.png similarity index 100% rename from docs/design/the-column-pool.png rename to docs/design/imgs/the-column-pool.png