Skip to content

Commit

Permalink
Update Parameter Name from tikv-importer.incremental-import to `tik…
Browse files Browse the repository at this point in the history
…v-importer.parallel-import` (pingcap#14350)
  • Loading branch information
lyzx2001 authored Aug 7, 2023
1 parent 98a5537 commit b79d940
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 7 deletions.
4 changes: 2 additions & 2 deletions tidb-lightning/tidb-lightning-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ driver = "file"
# The default value is `false`.
# When you use parallel import mode, you must set the parameter to `true`,
# but the premise is that no data exists in the target table, that is, all data can only be imported by TiDB Lightning.
# Note that this parameter **is not for incremental data import** and is only used in scenarios where the target table is empty.
# incremental-import = false
# Note that this parameter is only used in scenarios where the target table is empty.
# parallel-import = false

# The listening address of tikv-importer when backend is "importer". Change it to the actual address.
addr = "172.16.31.10:8287"
Expand Down
8 changes: 4 additions & 4 deletions tidb-lightning/tidb-lightning-distributed-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,13 @@ You can use TiDB Lightning to import data in parallel in the following scenarios
>
> - Parallel import only supports initialized empty tables in TiDB and does not support migrating data to tables with data written by existing services. Otherwise, data inconsistencies may occur.
>
> - Parallel import is usually used in the physical import mode. You need to configure `incremental-import = true`.
> - Parallel import is usually used in the physical import mode. You need to configure `parallel-import = true`.
>
> - Apply only one backend at a time when using multiple TiDB Lightning instances to import data to the same target. For example, you cannot import data to the same TiDB cluster in both the physical and logical import modes at the same time.
## Considerations

To use parallel import, you need to configure `incremental-import = true`. When TiDB Lightning is started, it registers meta data in the downstream TiDB cluster and automatically detects whether there are other instances migrating data to the target cluster at the same time. If there is, it automatically enters the parallel import mode.
To use parallel import, you need to configure `parallel-import = true`. When TiDB Lightning is started, it registers meta data in the downstream TiDB cluster and automatically detects whether there are other instances migrating data to the target cluster at the same time. If there is, it automatically enters the parallel import mode.

But when migrating data in parallel, you need to take the following into consideration:

Expand Down Expand Up @@ -95,7 +95,7 @@ data-source-dir = "/path/to/source-dir"
[tikv-importer]
# Whether to allow importing data into tables that already have data. The default value is `false`.
# When using parallel import, because multiple TiDB Lightning instances import a table at the same time, this configuration item must be set to `true`.
incremental-import = true
parallel-import = true
# "local": The default mode. It applies to large dataset import, for example, greater than 1 TiB. However, during the import, downstream TiDB is not available to provide services.
# "tidb": You can use this mode for small dataset import, for example, smaller than 1 TiB. During the import, downstream TiDB is available to provide services.
backend = "local"
Expand Down Expand Up @@ -175,7 +175,7 @@ type = "sql"
[tikv-importer]
# Whether to allow importing data into tables that already have data. The default value is `false`.
# When using parallel import, because multiple TiDB Lightning instances import a table at the same time, this configuration item must be set to `true`.
incremental-import = true
parallel-import = true
```

You can modify the configuration of the other instance to only import the `05001 ~ 10000` data files.
Expand Down
2 changes: 1 addition & 1 deletion tidb-lightning/tidb-lightning-prechecks.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ The following table describes each check item and detailed explanation.
| Exceedingly Large CSV files in the data file | >= 5.3.0 | When there are CSV files larger than 10 GiB in the backup file and auto-slicing is not enabled (StrictFormat=false), it will impact the import performance. The purpose of this check is to remind you to ensure the data is in the right format and to enable auto-slicing. |
| Recovery from breakpoints | >= 5.3.0 | This check ensures that no changes are made to the source file or schema in the database during the breakpoint recovery process that would result in importing the wrong data. |
| Import into an existing table | >= 5.3.0 | When importing into an already created table, it checks, as much as possible, whether the source file matches the existing table. Check if the number of columns matches. If the source file has column names, check if the column names match. When there are default columns in the source file, it checks if the default columns have Default Value, and if they have, the check passes. |
| Whether the target table is empty | >= 5.3.1 | TiDB Lightning automatically exits with an error if the target table is not empty. If parallel import mode is enabled (`incremental-import = true`), this check item will be skipped. |
| Whether the target table is empty | >= 5.3.1 | TiDB Lightning automatically exits with an error if the target table is not empty. If parallel import mode is enabled (`parallel-import = true`), this check item will be skipped. |

0 comments on commit b79d940

Please sign in to comment.