Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revise conflict data resolution enhancement during data import #16861

Merged
merged 5 commits into from
Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tidb-lightning/tidb-lightning-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ strategy = ""
# Note that this parameter is only used in scenarios where the target table is empty.
# parallel-import = false

# Starting from v8.0.0, the `duplicate-resolution` parameter is deprecated. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# The `duplicate-resolution` parameter is deprecated starting from v8.0.0 and will be removed in a future release. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# Whether to detect and resolve duplicate records (unique key conflict) in the physical import mode.
# The following resolution algorithms are supported:
# - none: does not detect duplicate records, which has the best performance of the two algorithms.
Expand Down
7 changes: 4 additions & 3 deletions tidb-lightning/tidb-lightning-error-resolution.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This document introduces TiDB Lightning error types, how to query the errors, an

- `lightning.max-error`: the tolerance threshold of type error
- `conflict.strategy`, `conflict.threshold`, and `conflict.max-record-rows`: configurations related to conflicting data
- `tikv-importer.duplicate-resolution` (deprecated in v8.0.0): the conflict handling configuration that can only be used in the physical import mode
- `tikv-importer.duplicate-resolution` (deprecated in v8.0.0 and will be removed in a future release): the conflict handling configuration that can only be used in the physical import mode
- `lightning.task-info-schema-name`: the database where conflicting data is stored when TiDB Lightning detects conflicts

For more information, see [TiDB Lightning (Task)](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task).
Expand Down Expand Up @@ -194,7 +194,7 @@ In this example, a data source is prepared with some known errors.
EOF
```

3. Configure TiDB Lightning to enable strict SQL mode, use the Local-backend to import data, delete duplicates, and skip up to 10 errors.
3. Configure TiDB Lightning to enable strict SQL mode, use the Local-backend to import data, replace duplicates, and skip up to 10 errors.

{{< copyable "shell-regular" >}}

Expand All @@ -207,8 +207,9 @@ In this example, a data source is prepared with some known errors.
[tikv-importer]
backend = 'local'
sorted-kv-dir = '/tmp/lightning-tmp/'
duplicate-resolution = 'remove'

[conflict]
strategy = 'replace'
[mydumper]
data-source-dir = '.'
[tidb]
Expand Down
6 changes: 3 additions & 3 deletions tidb-lightning/tidb-lightning-physical-import-mode-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ strategy = ""
# Import mode. "local" means using the physical import mode.
backend = "local"

# Starting from v8.0.0, the `duplicate-resolution` parameter is deprecated. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# The `duplicate-resolution` parameter is deprecated starting from v8.0.0 and will be removed in a future release. For more information, see <https://docs.pingcap.com/tidb/dev/tidb-lightning-physical-import-mode-usage#the-old-version-of-conflict-detection-deprecated-in-v800>.
# The method to resolve the conflicting data.
duplicate-resolution = 'remove'

Expand Down Expand Up @@ -104,7 +104,7 @@ Conflicting data refers to two or more records with the same primary key or uniq
There are two versions for conflict detection:

- The new version of conflict detection, controlled by the `conflict` configuration item.
- The old version of conflict detection (deprecated in v8.0.0), controlled by the `tikv-importer.duplicate-resolution` configuration item.
- The old version of conflict detection (deprecated in v8.0.0 and will be removed in a future release), controlled by the `tikv-importer.duplicate-resolution` configuration item.

### The new version of conflict detection

Expand All @@ -131,7 +131,7 @@ The new version of conflict detection controls whether to enable preprocess conf

### The old version of conflict detection (deprecated in v8.0.0)

Starting from v8.0.0, the old version of conflict detection (`tikv-importer.duplicate-resolution`) is deprecated. If `tikv-importer.duplicate-resolution` is `remove` and `conflict.strategy` is not configured, TiDB Lightning automatically enables the new version of conflict detection by assigning the value of `conflict.strategy` to `"replace"`. Note that `tikv-importer.duplicate-resolution` and `conflict.strategy` cannot be configured at the same time, as it will result in an error.
Starting from v8.0.0, the old version of conflict detection (`tikv-importer.duplicate-resolution`) is deprecated. The `tikv-importer.duplicate-resolution` parameter will be removed in a future release. If `tikv-importer.duplicate-resolution` is `remove` and `conflict.strategy` is not configured, TiDB Lightning automatically enables the new version of conflict detection by assigning the value of `conflict.strategy` to `"replace"`. Note that `tikv-importer.duplicate-resolution` and `conflict.strategy` cannot be configured at the same time, as it will result in an error.

- For versions between v7.3.0 and v7.6.0, TiDB Lightning enables the old version of conflict detection when `tikv-importer.duplicate-resolution` is not an empty string.
- For v7.2.0 and earlier versions, TiDB Lightning only supports the old version of conflict detection.
Expand Down
Loading