From 135686ade3473a6f9a3f8d1685de0d42389b9d9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 3 Jan 2025 08:20:18 +0100 Subject: [PATCH] dumpling: Update README (#58679) --- dumpling/README.md | 22 ++++----- dumpling/docs/cn/user-guide.md | 82 ---------------------------------- dumpling/docs/en/user-guide.md | 82 ---------------------------------- 3 files changed, 8 insertions(+), 178 deletions(-) delete mode 100644 dumpling/docs/cn/user-guide.md delete mode 100644 dumpling/docs/en/user-guide.md diff --git a/dumpling/README.md b/dumpling/README.md index 076ae57a4fb32..442d6bd474d90 100644 --- a/dumpling/README.md +++ b/dumpling/README.md @@ -1,35 +1,31 @@ 🥟 Dumpling ============ -[![Build Status](https://travis-ci.org/pingcap/dumpling.svg?branch=master)](https://travis-ci.org/pingcap/dumpling) -[![codecov](https://codecov.io/gh/pingcap/dumpling/branch/master/graph/badge.svg)](https://codecov.io/gh/pingcap/dumpling) -[![API Docs](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white)](https://pkg.go.dev/github.com/pingcap/dumpling) -[![Go Report Card](https://goreportcard.com/badge/github.com/pingcap/dumpling)](https://goreportcard.com/report/github.com/pingcap/dumpling) -[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpingcap%2Fdumpling.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpingcap%2Fdumpling?ref=badge_shield) +[![API Docs](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white)](https://pkg.go.dev/github.com/pingcap/tidb/dumpling) -**Dumpling** is a tool and a Go library for creating SQL dump from a MySQL-compatible database. +**Dumpling** is a tool and a Go library for creating an SQL dump from a MySQL-compatible database. It is intended to replace `mysqldump` and `mydumper` when targeting TiDB. -You may read the [design document](https://github.com/pingcap/community/blob/master/rfc/2019-12-06-dumpling.md), [English user guide](docs/en/user-guide.md) and [中文使用手册](docs/cn/user-guide.md) for details. +You may read the original [design document](https://github.com/pingcap/community/blob/master/archive/misc/rfc/2019-12-06-dumpling.md). The end-user documentation is available here: [en](https://docs.pingcap.com/tidb/stable/dumpling-overview), [zh](https://docs.pingcap.com/zh/tidb/stable/dumpling-overview), [ja](https://docs.pingcap.com/ja/tidb/stable/dumpling-overview). Features -------- -> Dumpling is currently in early development stage, and most features are incomplete. Contributions are welcomed! +> Dumpling is in active development. Contributions are welcomed! - [x] SQL dump is split into multiple files (like `mydumper`) for easy management. - [x] Export multiple tables in parallel to speed up execution. - [x] Multiple output formats: SQL, CSV, ... -- [ ] Write to cloud storage (S3, GCS) natively +- [x] Write to cloud storage (S3, GCS) natively - [x] Advanced table filtering -Any questions? Let's discuss on [TiDB Internals forum](https://internals.tidb.io/)! +Any questions? Discord, Slack and other contact options can be found [here](https://docs.pingcap.com/tidb/stable/support) Building -------- 0. Under directory `tidb` -1. Install Go 1.16 or above +1. Install Go 1.23.4 or above 2. Run `make build_dumpling` to compile. The output is in `bin/dumpling`. 3. Run `make dumpling_unit_test` to run the unit tests. 4. Run `make dumpling_integration_test` to run integration tests. For integration test: @@ -46,6 +42,4 @@ Building License ------- -Dumpling is under the Apache 2.0 license. See the [LICENSE](./LICENSE) file for details. - -[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpingcap%2Fdumpling.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpingcap%2Fdumpling?ref=badge_large) +Dumpling is under the Apache 2.0 license. See the [LICENSE](../LICENSE) file for details. diff --git a/dumpling/docs/cn/user-guide.md b/dumpling/docs/cn/user-guide.md deleted file mode 100644 index b330a905d6321..0000000000000 --- a/dumpling/docs/cn/user-guide.md +++ /dev/null @@ -1,82 +0,0 @@ -# Dumpling 使用手册 - -[Dumpling](https://github.com/pingcap/tidb/blob/master/dumpling) 是支持以 SQL 文本或者 CSV 格式将 MySQL/TiDB 数据导出的工具。 - -设计初衷是为了替代 [Mydumper](https://github.com/pingcap/mydumper), 所以基本用法可以参考 Mydumper, -当然在实现中没有完全照搬 Mydumper, 因此存在与 Mydumper 不同的用法。 - -下表罗列了一些主要参数 - -| 主要参数 | | -| --------| --- | -| -B 或 --database | 导出指定数据库 | -| -T 或 --tables-list | 导出指定数据表 | -| -f 或 --filter | 导出能匹配模式的表，语法可参考 [table-filter](https://github.com/pingcap/tidb-tools/blob/master/pkg/table-filter/README.md)（只有英文版） | -| --case-sensitive | table-filter 是否大小写敏感，默认为 false 不敏感 | -| -h 或 --host| 链接节点地址(默认 "127.0.0.1")| -| -t 或 --threads | 备份并发线程数| -| -r 或 --rows |将 table 划分成 row 行数据，一般针对大表操作并发生成多个文件。| -| --loglevel | 日志级别 {debug,info,warn,error,dpanic,panic,fatal} (默认 "info") | -| -d 或 --no-data | 不导出数据, 适用于只导出 schema 场景 | -| --no-header | 导出 table csv 数据，不生成 header | -| -W 或 --no-views| 不导出 view, 默认 true | -| -m 或 --no-schemas | 不导出 schema , 只导出数据 | -| -s 或--statement-size | 控制 Insert Statement 的大小，单位 bytes | -| -F 或 --filesize | 将 table 数据划分出来的文件大小, 需指明单位 (如 `128B`, `64KiB`, `32MiB`, `1.5GiB`) | -| --filetype| 导出文件类型 csv/sql (默认 sql) | -| -o 或 --output | 设置导出文件路径 | -| --output-filename-template | 设置导出文件名模版，详情见下 | -| -S 或 --sql | 根据指定的 sql 导出数据，该指令不支持并发导出 | -| --consistency | flush: dump 前用 FTWRL
snapshot: 通过 tso 指定 dump 位置
lock: 对需要 dump 的所有表执行 lock tables read
none: 不加锁 dump，无法保证一致性
auto: MySQL flush, TiDB snapshot| -| --snapshot | snapshot tso, 只在 consistency=snapshot 下生效 | -| --where | 对备份的数据表通过 where 条件指定范围 | -| -p 或 --password | 链接密码 | -| -P 或 --port | 链接端口，默认 4000 | -| -u 或 --user | 默认 root | - -更多具体用法可以使用 -h, --help 进行查看。 - -## Mydumper 相关参考 - -[Mydumper usage](https://github.com/maxbube/mydumper/blob/master/docs/mydumper_usage.rst) - -[TiDB Mydumper 使用文档](https://pingcap.com/docs-cn/stable/reference/tools/mydumper/) - -## Dumpling 下载链接 - -[nightly](https://download.pingcap.org/dumpling-nightly-linux-amd64.tar.gz) - -## 导出文件名模版 - -`--output-filename-template` 参数指定了所有文件的命名方式（不含扩展名）。它使用 [Go 的 `text/template` 语法](https://golang.org/pkg/text/template/)。 - -模板可使用以下字段： - -* `.DB` — 库名 -* `.Table` — 表名、物件名称。 -* `.Index` — 由 0 开始的序列号，代表当前导出的表中的哪一份文件 - -库和表名中可能包含 `/` 之类的特殊字符，而这些字符不能用在文件系统中。因此，Dumpling 提供了一个 `fn` 函数来对这些特殊字符进行百分号编码。它们是： - -* U+0000 到 U+001F (控制字符) -* `/`、`\`、`<`、`>`、`:`、`"`、`*`、`?` (无效的 Windows 路径字符) -* `.` (库/表名分隔符) -* `-`，当出现在 `-schema` 字串里 - -例如，使用 `--output-filename-template '{{fn .Table}}.{{printf "%09d" .Index}}'` 后，Dumpling 会把表 `"db"."tbl:normal"` 导出到 `tbl%3Anormal.000000000.sql`、`tbl%3Anormal.000000001.sql` 等文件。 - -除数据文件外，Dumpling 还支持透过子模版自定义命名表结构文件的名称。默认的配置是： - -| 模版名 | 默认内容 | -|------|---------| -| data | `{{fn .DB}}.{{fn .Table}}.{{.Index}}` | -| schema | `{{fn .DB}}-schema-create` | -| table | `{{fn .DB}}.{{fn .Table}}-schema` | -| event | `{{fn .DB}}.{{fn .Table}}-schema-post` | -| function | `{{fn .DB}}.{{fn .Table}}-schema-post` | -| procedure | `{{fn .DB}}.{{fn .Table}}-schema-post` | -| sequence | `{{fn .DB}}.{{fn .Table}}-schema-sequence` | -| trigger | `{{fn .DB}}.{{fn .Table}}-schema-triggers` | -| view | `{{fn .DB}}.{{fn .Table}}-schema-view` | - -例如，使用 `--output-filename-template '{{define "table"}}{{fn .Table}}.$schema{{end}}{{define "data"}}{{fn .Table}}.{{printf "%09d" .Index}}{{end}}'`后，Dumpling 会把表 `"db"."tbl:normal"` 的结构写到 `tbl%3Anormal.$schema.sql`，以及把数据写到 `tbl%3Anormal.000000000.sql`。 diff --git a/dumpling/docs/en/user-guide.md b/dumpling/docs/en/user-guide.md deleted file mode 100644 index e98ae1cf15b5e..0000000000000 --- a/dumpling/docs/en/user-guide.md +++ /dev/null @@ -1,82 +0,0 @@ -# Dumpling User Guide - -**Dumpling** is a tool and a Go library for creating SQL dump (CSV/SQL format) from a MySQL-compatible database. - -It is intended to replace `mysqldump` and `mydumper` when targeting TiDB; as a result, its basic usage is similar to that of Mydumper. - -The following table lists the major parameters of Dumpling. - - -| Parameter | Description | -| --------| --- | -| -B or --database | Dump the specified databases. | -| -T or --tables-list | Dump the specified tables | -| -f or --filter | Dump only the tables matching the patterns. See [table-filter](https://github.com/pingcap/tidb-tools/blob/master/pkg/table-filter/README.md) for syntax. | -| --case-sensitive | whether the filter should be case-sensitive, default false(insensitive) | -| -h or --host | Host to connect to. (default: `127.0.0.1`) | -| -t or --threads | Number of threads for concurrent backup. | -| -r or --rows | Split table into multiple files by number of rows. This allows Dumpling to generate multiple files concurrently. (default: unlimited) | -| --loglevel | Log level. {debug, info, warn, error, dpanic, panic, fatal}. (default: `info`) | -| -d or --no-data | Don't dump data, for schema-only case. | -| --no-header | Dump table CSV without header. | -| -W or --no-views | Don't dump views. (default: `true`) | -| -m or --no-schemas | Don't dump schemas, dump data only. | -| -s or --statement-size | Control the size of Insert Statement. Unit: byte. | -| -F or --filesize | The approximate size of the output file. The unit should be explicitly provided (such as `128B`, `64KiB`, `32MiB`, `1.5GiB`) | -| --filetype| The type of dump file. (sql/csv, default "sql") | -| -o or --output | Output directory. The default value is based on time. | -| --output-filename-template | Output file name templates. See below for details. | -| -S or --sql | Dump data with given sql. This argument doesn't support concurrent dump | -| --consistency | Which consistency control to use (default `auto`):
`flush`: Use FTWRL (flush tables with read lock)
`snapshot`: use a snapshot at a given timestamp
`lock`: execute lock tables read for all tables that need to be locked
`none`: dump without locking. It cannot guarantee consistency
`auto`: `flush` on MySQL, `snapshot` on TiDB | -| --snapshot | Snapshot position. Valid only when consistency=snapshot. | -| --where | Specify the dump range by `where` condition. Dump only the selected records. | -| -p or --password | User password. | -| -P or --port | TCP/IP port to connect to. (default: `4000`) | -| -u or --user | Username with privileges to run the dump. (default "root") | - -To see more detailed usage, run the flag `-h` or `--help`. - -## Mydumper Reference - -[Mydumper usage](https://github.com/maxbube/mydumper/blob/master/docs/mydumper_usage.rst) - -[TiDB Mydumper Instructions](https://pingcap.com/docs/stable/reference/tools/mydumper/) - -## Download - -Download the nightly version of Dumpling [here](https://download.pingcap.org/dumpling-nightly-linux-amd64.tar.gz). - -## Output filename template - -The `--output-filename-template` argument specifies how all files are named, before including the file extensions. It accepts strings in the [Go `text/template` syntax](https://golang.org/pkg/text/template/). - -The following fields are available to the template: - -* `.DB` — database name -* `.Table` — table name or object name -* `.Index` — when a table is split into multiple files, this is the 0-based sequence number indicating which part we are dumping - -The database and table names may contain special characters like `/` not acceptable in the file system. Thus, Dumpling also provided a function `fn` to percent-escape these special characters: - -* U+0000 to U+001F (control characters) -* `/`, `\`, `<`, `>`, `:`, `"`, `*`, `?` (invalid Windows path characters) -* `.` (database/table name separator) -* `-`, if appeared as part of `-schema` - -For instance, using `--output-filename-template '{{fn .Table}}.{{printf "%09d" .Index}}'`, Dumpling will write the table `"db"."tbl:normal"` into files named like `tbl%3Anormal.000000000.sql`, `tbl%3Anormal.000000001.sql`, etc. - -Besides the data files, you could also define named templates to replace the file name of the schema files. The default are configuration is: - -| Name | Content | -|------|---------| -| data | `{{fn .DB}}.{{fn .Table}}.{{.Index}}` | -| schema | `{{fn .DB}}-schema-create` | -| table | `{{fn .DB}}.{{fn .Table}}-schema` | -| event | `{{fn .DB}}.{{fn .Table}}-schema-post` | -| function | `{{fn .DB}}.{{fn .Table}}-schema-post` | -| procedure | `{{fn .DB}}.{{fn .Table}}-schema-post` | -| sequence | `{{fn .DB}}.{{fn .Table}}-schema-sequence` | -| trigger | `{{fn .DB}}.{{fn .Table}}-schema-triggers` | -| view | `{{fn .DB}}.{{fn .Table}}-schema-view` | - -For instance, using `--output-filename-template '{{define "table"}}{{fn .Table}}.$schema{{end}}{{define "data"}}{{fn .Table}}.{{printf "%09d" .Index}}{{end}}'`, Dumpling will write the schema of the table `"db"."tbl:normal"` into the file `tbl%3Anormal.$schema.sql`, and data into the files like `tbl%3Anormal.000000000.sql`.