Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dumpling: merge userguide into docs #19890

Merged
merged 6 commits into from
Jan 20, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions dumpling-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -411,3 +411,38 @@
| `--tidb-mem-quota-query` | The memory limit of exporting SQL statements by a single line of Dumpling command, and the unit is byte. For v4.0.10 or later versions, if you do not set this parameter, TiDB uses the value of the `mem-quota-query` configuration item as the memory limit value by default. For versions earlier than v4.0.10, the parameter value defaults to 32 GB. | 34359738368 |
| `--params` | Specifies the session variable for the connection of the database to be exported. The required format is `"character_set_client=latin1,character_set_connection=latin1"` |
| `-c` or `--compress` | Compresses the CSV and SQL data and table structure files exported by Dumpling. It supports the following compression algorithms: `gzip`, `snappy`, and `zstd`. | "" |

## Output filename template
hfxsd marked this conversation as resolved.
Show resolved Hide resolved

The `--output-filename-template` argument defines the naming convention for output files, excluding the file extensions. It accepts strings in the [Go `text/template` syntax](https://golang.org/pkg/text/template/).

The following fields are available for the template:

* `.DB`: the database name
* `.Table`: the table name or the object name
* `.Index`: the 0-based sequence number of the file when a table is split into multiple files, indicating which part is being dumped. For example, `{{printf "%09d" .Index}}` means formatting `.Index` as a 9-digit number with leading zeros.

Database and table names might contain special characters (such as `/`) that are not allowed in file systems. To handle this issue, Dumpling provides the `fn` function to percent-encode these special characters:

* U+0000 to U+001F (control characters)
* `/`, `\`, `<`, `>`, `:`, `"`, `*`, `?` (invalid Windows path characters)
* `.` (database or table name separator)
* `-`, if used as part of `-schema`

For example, using `--output-filename-template '{{fn .Table}}.{{printf "%09d" .Index}}'`, Dumpling will write the table `db.tbl:normal` into files named `tbl%3Anormal.000000000.sql`, `tbl%3Anormal.000000001.sql`, and so on.

Check failure on line 432 in dumpling-overview.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [PingCAP.Latin] Use 'such as' instead of 'and so on.'. Raw Output: {"message": "[PingCAP.Latin] Use 'such as' instead of 'and so on.'.", "location": {"path": "dumpling-overview.md", "range": {"start": {"line": 432, "column": 214}}}, "severity": "ERROR"}

In addition to output data files, you can define `--output-filename-template` to replace file names of the schema files. The following table shows the default configurations.

| Name | Content |
|------|---------|
| data | `{{fn .DB}}.{{fn .Table}}.{{.Index}}` |
| schema | `{{fn .DB}}-schema-create` |
| table | `{{fn .DB}}.{{fn .Table}}-schema` |
| event | `{{fn .DB}}.{{fn .Table}}-schema-post` |
| function | `{{fn .DB}}.{{fn .Table}}-schema-post` |
| procedure | `{{fn .DB}}.{{fn .Table}}-schema-post` |
| sequence | `{{fn .DB}}.{{fn .Table}}-schema-sequence` |
| trigger | `{{fn .DB}}.{{fn .Table}}-schema-triggers` |
| view | `{{fn .DB}}.{{fn .Table}}-schema-view` |

For example, using `--output-filename-template '{{define "table"}}{{fn .Table}}.$schema{{end}}{{define "data"}}{{fn .Table}}.{{printf "%09d" .Index}}{{end}}'`, Dumpling will write the schema of the table `db.tbl:normal` into a file named `tbl%3Anormal.$schema.sql`, and write the data into files `tbl%3Anormal.000000000.sql`, `tbl%3Anormal.000000001.sql`, and so on.

Check failure on line 448 in dumpling-overview.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [PingCAP.Latin] Use 'such as' instead of 'and so on.'. Raw Output: {"message": "[PingCAP.Latin] Use 'such as' instead of 'and so on.'.", "location": {"path": "dumpling-overview.md", "range": {"start": {"line": 448, "column": 357}}}, "severity": "ERROR"}
Loading