Skip to content

Commit

Permalink
Revise conflict data resolution enhancement during data import (#16861)…
Browse files Browse the repository at this point in the history
… (#16873)
  • Loading branch information
ti-chi-bot authored Mar 28, 2024
1 parent f23d87a commit 3d8adb9
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 7 deletions.
2 changes: 1 addition & 1 deletion tidb-lightning/tidb-lightning-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ strategy = ""
# Note that this parameter is only used in scenarios where the target table is empty.
# parallel-import = false

# Starting from v8.0.0, the `duplicate-resolution` parameter is deprecated. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# The `duplicate-resolution` parameter is deprecated starting from v8.0.0 and will be removed in a future release. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# Whether to detect and resolve duplicate records (unique key conflict) in the physical import mode.
# The following resolution algorithms are supported:
# - none: does not detect duplicate records, which has the best performance of the two algorithms.
Expand Down
7 changes: 4 additions & 3 deletions tidb-lightning/tidb-lightning-error-resolution.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This document introduces TiDB Lightning error types, how to query the errors, an

- `lightning.max-error`: the tolerance threshold of type error
- `conflict.strategy`, `conflict.threshold`, and `conflict.max-record-rows`: configurations related to conflicting data
- `tikv-importer.duplicate-resolution` (deprecated in v8.0.0): the conflict handling configuration that can only be used in the physical import mode
- `tikv-importer.duplicate-resolution` (deprecated in v8.0.0 and will be removed in a future release): the conflict handling configuration that can only be used in the physical import mode
- `lightning.task-info-schema-name`: the database where conflicting data is stored when TiDB Lightning detects conflicts

For more information, see [TiDB Lightning (Task)](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task).
Expand Down Expand Up @@ -194,7 +194,7 @@ In this example, a data source is prepared with some known errors.
EOF
```
3. Configure TiDB Lightning to enable strict SQL mode, use the Local-backend to import data, delete duplicates, and skip up to 10 errors.
3. Configure TiDB Lightning to enable strict SQL mode, use the Local-backend to import data, replace duplicates, and skip up to 10 errors.
{{< copyable "shell-regular" >}}
Expand All @@ -207,8 +207,9 @@ In this example, a data source is prepared with some known errors.
[tikv-importer]
backend = 'local'
sorted-kv-dir = '/tmp/lightning-tmp/'
duplicate-resolution = 'remove'
[conflict]
strategy = 'replace'
[mydumper]
data-source-dir = '.'
[tidb]
Expand Down
6 changes: 3 additions & 3 deletions tidb-lightning/tidb-lightning-physical-import-mode-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ strategy = ""
# Import mode. "local" means using the physical import mode.
backend = "local"

# Starting from v8.0.0, the `duplicate-resolution` parameter is deprecated. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# The `duplicate-resolution` parameter is deprecated starting from v8.0.0 and will be removed in a future release. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# The method to resolve the conflicting data.
duplicate-resolution = 'remove'

Expand Down Expand Up @@ -104,7 +104,7 @@ Conflicting data refers to two or more records with the same primary key or uniq
There are two versions for conflict detection:

- The new version of conflict detection, controlled by the `conflict` configuration item.
- The old version of conflict detection (deprecated in v8.0.0), controlled by the `tikv-importer.duplicate-resolution` configuration item.
- The old version of conflict detection (deprecated in v8.0.0 and will be removed in a future release), controlled by the `tikv-importer.duplicate-resolution` configuration item.

### The new version of conflict detection

Expand All @@ -131,7 +131,7 @@ The new version of conflict detection controls whether to enable preprocess conf

### The old version of conflict detection (deprecated in v8.0.0)

Starting from v8.0.0, the old version of conflict detection (`tikv-importer.duplicate-resolution`) is deprecated. If `tikv-importer.duplicate-resolution` is `remove` and `conflict.strategy` is not configured, TiDB Lightning automatically enables the new version of conflict detection by assigning the value of `conflict.strategy` to `"replace"`. Note that `tikv-importer.duplicate-resolution` and `conflict.strategy` cannot be configured at the same time, as it will result in an error.
Starting from v8.0.0, the old version of conflict detection (`tikv-importer.duplicate-resolution`) is deprecated. The `tikv-importer.duplicate-resolution` parameter will be removed in a future release. If `tikv-importer.duplicate-resolution` is `remove` and `conflict.strategy` is not configured, TiDB Lightning automatically enables the new version of conflict detection by assigning the value of `conflict.strategy` to `"replace"`. Note that `tikv-importer.duplicate-resolution` and `conflict.strategy` cannot be configured at the same time, as it will result in an error.

- For versions between v7.3.0 and v7.6.0, TiDB Lightning enables the old version of conflict detection when `tikv-importer.duplicate-resolution` is not an empty string.
- For v7.2.0 and earlier versions, TiDB Lightning only supports the old version of conflict detection.
Expand Down

0 comments on commit 3d8adb9

Please sign in to comment.