DM v2.0.7

[2.0.7] 2021-09-23

Bug fixes

• Fix the error that binlog event is purged when switching enable-gtid in source configuration from false to true #2094
• Fix the memory leak problem of schema-tracker #2133

Improvements

• Disable background statistic job in schema tracker to reduce CPU consumption #2065
• Support regular expressions rules for online DDL shadow and trash tables #2139

DM v2.0.6

[2.0.6] 2021-08-13

Bug fixes

• Fix the issue that the metadata inconsistency between DDL infos and upstream tables in the optimistic sharding DDL mode causes DM-master panic #1971

DM v2.0.5

[2.0.5] 2021-07-30

Improvements

• Support for filtering certain DML using SQL expressions #1832
• Add config import/export command to import and export cluster sources and tasks configuration files for downgrade #1921
• Optimize safe-mode to improve replication efficiency #1920
• Maximize compatibility with upstream SQL_MODE #1894
• Support upstream using both pt and gh-ost online DDL modes in one task #1918
• Improve the efficiency of replication of DECIMAL types #1841
• Support for automatic retry of transaction-related retryable errors #1916

Bug fixes

• Fix the issue that the inconsistency of upstream and downstream primary keys might lead to data loss #1919
• Fix the issue that too many upstream sources cause cluster upgrade failure and DM-master OOM #1868
• Fix the issue of the configuration item case-sensitive #1886
• Fix the issue that the default value of tidb_enable_change_column_type inside DM is wrong #1843
• Fix the issue that the auto_random column in downstream may causes task interruption #1847
• Fix the issue that operate-schema set -flush command causes DM-worker panic #1829
• Fix the issue that DDL fails to coordinate within DM-worker due to repeated execution of the same DDL in pessimistic mode #1816
• Fix the issue that wrong configuration causes DM-worker panic #1842
• Fix the issue that redoing tasks causes loader panic #1822
• Fix the issue that DM binlog file name is not timely updated after upstream master-slave switch #1874
• Fix the issue of incorrect value of replication delay monitoring #1880
• Fix the issue that block-allow-list fails to filter online DDL in some cases #1867
• Fix the issue that the task cannot be stopped manually due to the error after automatic resuming #1917

DM v1.0.7

[1.0.7] 2021-06-21

Bug fixes

• Fix the issue that data may be lost after a task restarts from interruption #1783

DM v2.0.4

[2.0.4] 2021-06-18

Improvements

• Support rescheduling and automatically resuming tasks after a DM-worker goes offline first and then comes back online during the full import #1784
• Add the metric replicationLagGauge to monitor replication delay #1759
• Restore schemas in parallel during the full import #1701
• Support automatically adjusting the time_zone settings of both the upstream and downstream databases #1714
• Improve the speed of rolling back incremental replication tasks after the tasks meet errors #1705
• Automatically adjust GTID according to checkpoints when GTID is enabled during the incremental replication #1745
• Detect the versions of upstream and downstream databases and record the versions in log files #1693
• Use the schema from the dump stage of the full export as the initial schema for the incremental replication task of the same data source #1754
• Decrease the time that the safe mode lasts after the incremental task is restarted to 1 minute to improve the replication speed #1779
• Improve the usability of dmctl
  • Support setting the address of DM-master as an environment variable #1726
  • Support specifying the master-addr parameter anywhere in a dmctl command #1771
  • Use the encrypt/decrypt command instead of the --decrypt/-encrypt parameter to encrypt or decrypt the database password #1771

Bug fixes

• Fix the issue that data may be lost after a non-GTID task restarts from interruption #1781
• Fix the issue that the data source binding information may be lost after upgrading a DM cluster which has been downgraded before #1713
• Fix the issue that etcd reports that the wal directory does not exist when DM-master restarts #1680
• Fix the issue that the number of error messages reported from precheck exceeds the grpc limit #1688
• Fix the issue that DM-worker panics when replicating unsupported statements from a MariaDB database of an earlier version #1734
• Fix the issue that DM does not update the metric of relay log disk capacity #1753
• Fix the issue that DM may panic when getting the master status of the upstream database binlog #1774

DM v2.0.3

[2.0.3] 2021-05-11

Improvements

• Support deleting residual DDL locks using the command unlock-ddl-lock after the migration task is stopped #1612
• Support limiting the number of errors and warnings that DM reports during the precheck process #1621
• Optimize the behavior of the command query-status to get the status of upstream binlogs #1630
• Optimize the format of sharded tables’ migration status output by the command query-status in the pessimistic mode #1650
• Print help message first when dmtcl processes commands with the --help input #1637
• Automatically remove the related information from monitoring panels after a DDL lock is deleted #1631
• Automatically remove the related task status from monitoring panels after a task is stopped or completed #1614

Bug fixes

• Fix the issue that DM-master becomes out of memory after DM is updated to v2.0.2 in the process of shard DDL coordination using the optimistic mode #1643 #1649
• Fix the issue that the source binding information is lost when DM is started for the first time after updated to v2.0.2 #1649
• Fix the issue that the flag in the command operate-source show -s does not take effect #1587
• Fix the issue that the command operate-source stop <config-file> fails because DM cannot connect to the source #1587
• Fix the finer-grained issue that some migration errors might be wrongly ignored #1599
• Fix the issue that the migration is interrupted when DM filters online DDL statements according to binlog event filtering rules that are configured #1668

v1.0.4-hotfix-2

Hotfix again (#1641)

DM v2.0.2

[2.0.2] 2021-04-09

Improvements

• Relay log GA
  • The relay log feature is no longer enabled by setting the source configuration file.Now, the feature is enabled by running commands in dmctl for specified DM-workers #1499
  • DM sends the commands query-status -s and purge-relay to all DM-workers that pull relay logs #1533
  • Align the relay unit‘s behavior of pulling and sending binlogs with that of the secondary MySQL database #1390
  • Reduce the scenarios where relay logs need to be purged #1400
  • Support sending heartbeat events when the relay log feature is enabled to display task progress with regular updates #1404
• Optimistic sharding DDL mode
  • Optimize operations for resolving DDL conflicts #1496 #1506 #1518 #1551
  • Adjust the DDL coordination behavior in the optimistic mode to avoid data inconsistency in advance #1510 #1512
• Support automatically recognizing the switching of upstream data sources when the source configuration needs no update, for example, when the IP address does not change #1364
• Precheck the privileges of the upstream MySQL instance at a finer granularity #1336
• Support configuring binlog event filtering rules in the source configuration file #1370
• When binding an idle upstream data source to an idle DM-worker node, DM-master nodes firstly choose the most recent binding of that DM-worker node #1373
• Improve the stability of DM automatically getting the SQL mode from the binlog file #1382 #1552
• Support automatically parsing GTIDs of different formats in the source configuration file #1385
• Extend DM-worker’s TTL for keepalive to reduce scheduling caused by poor network #1405
• Support reporting an error when the configuration file contains configuration items that are not referenced #1410
• Improve the display of a GTID set by sorting it in dictionary order #1424
• Optimize monitoring and alerting rules #1438
• Support manually transferring an upstream data source to a specified DM-worker #1492
• Add configurations of etcd compaction and disk quota #1521

Bug fixes

• Fix the issue of data loss during the full data migration occurred because DM frequently restarts the task #1378
• Fix the issue that an incremental replication task fails to start when the binlog position is not specified together with GTID in the task configuration #1393
• Fix the issue that DM-worker’s binding relationships become abnormal when the disk and network environments are poor #1396
• Fix the issue that enabling the relay log feature might cause data loss when the GTIDs specified in upstream binlog previous_gtids events are not consecutive #1390 #1430
• Disable the heartbeat feature of DM v1.0 to avoid the failure of high availability scheduling #1467
• Fix the issue that the migration fails if the upstream binlog sequence number is larger than 999999 #1476
• Fix the issue that DM commands hang when DM gets stuck in pinging the upstream and downstream databases #1477
• Fix the issue that the full import fails when the upstream database enables the ANSI_QUOTES mode #1497
• Fix the issue that DM might duplicate binlog events when the GTID and the relay log are enabled at the same time #1525

DM v2.0.1

[2.0.1] 2020-12-25

Improvements

• Support the relay log feature in high availability scenarios #1353
  • DM-worker supports storing relay logs only locally.
  • In scenarios where a DM-worker node is down or is offline due to network fluctuations, the newly scheduled DM-worker pulls the upstream binlog again.
• Restrict the handle-error command to only handle DDL errors to avoid misuse #1303
• Support simultaneously connecting multiple DM-master nodes and automatically switching connected nodes in dmctl #1349
• Add the get-config command to get the configuration of migration tasks and DM components #1348
• Support migrating SQL statements like ALTER TABLE ADD COLUMN (xx, xx) #1345
• Support automatically filtering SQL statements like CREATE/ALTER/DROP EVENT #1343
• Support checking whether server-id is set for the upstream MySQL/MariaDB instance before the incremental replication task starts #1315
• Support replicating schemas and tables with sql in their names during the full import #1259

Bug fixes

• Fix the issue that restarting a task might cause fail to initial unit Sync of subtask error #1274
• Fix the issue that the pause-task command might be blocked when it is executed during the full import #1269 #1277
• Fix the issue that DM fails to create a data source for a MariaDB instance when enable-gtid: true is configured #1344
• Fix the issue that the query-status command might be blocked when it is executed #1293
• Fix the issue that concurrently coordinating multiple DDL statements in the pessimistic shard DDL mode might block the task #1263
• Fix the issue that running the pause-task command might get the meaningless sql: connection is already closed error #1304
• Fix the issue that the full migration fails when the upstream instance does not have the REPLICATION privilege #1326
• Fix the issue that the route-rules configuration of a shard merge task does not take effect in the full import when the SQL_MODE of the task contains ANSI_QUOTES #1314
• Fix the issue that DM fails to automatically apply the SQL_MODE of the upstream database during the incremental replication #1307
• Fix the issue that DM logs the fail to parse binlog status_vars warning when automatically parsing the SQL_MODE of the upstream database #1299

DM v2.0.0-GA

[2.0.0] 2020-10-30

Improvements

• Optimize the setting of safe-mode to ensure the eventual consistency of data when the upstream database, such as Amazon Aurora and Aliyun RDS, does not support FTWRL in the full export #981 #1017
• Support automatic configuration of sql_mode for data migration based on the global sql_mode of upstream and downstream databases and sql_mode of binlog events #1005 #1071 #1137
• Support automatic configuration of the max_allowed_packet from DM to the downstream TiDB, based on the global max_allowed_packet value of the downstream TiDB #1071
• Optimize the incremental replication speed compared with DM 2.0 RC version #1203
• Improve performance by using optimistic transaction to migrate data to TiDB by default #1107
• Support DM-worker automatically fetching and using the list of DM-master nodes in the cluster #1180
• Disable auto-resume behavior for more errors that cannot be automatically recovered #979 #1085 #1216

Bug fixes

• Fix the issue that failure to automatically set the default value of statement-size for full export might cause the packet for query is too large error or the OOM issue in TiDB #1133
• Fix DM-worker panic when there are concurrent checkpoint operations during the full import #1182
• Fix the issue that the migration task might have table checkpoint position * less than global checkpoint position error and be interrupted after the upstream MySQL/MariaDB instance is restarted #1041
• Fix the issue that migration tasks might be interrupted when the upstream database does not enable GTID #1123
• Fix the issue that the DM-master node does not start properly after conflicts occur during the shard DDL coordination #1199
• Fix the issue that the incremental replication might be too slow when there are multiple common indexes in the table to be migrated #1063
• Fix the issue that the progress display is abnormal after restarting the migration task during the full import #1043
• Fix the issue that paused migration subtasks cannot be obtained by query-status after being scheduled to another DM-worker #1183
• Fix the issue that FileSize might not take effect during the full export #1191
• Fix the issue that the -s parameter in extra-args does not take effect during the full export #1196
• Fix the issue that enabling the online DDL feature might cause not allowed operation: alter multiple tables in one statement error #1192
• Fix the issue that during the incremental replication, the migration task might be interrupted when the DDL statements to be migrated are associated with other tables, such as DDL statements related to foreign keys #1101 #1108
• Fix the issue that database names and table names with character / are not correctly parsed during the full migration #991
• Fix the issue that after failing to migrate DDL statements to the downstream TiDB database during the incremental replication, migration tasks might not be paused and the corresponding error cannot be obtained from query-status #1059
• Fix the issue that concurrently coordinating multiple DDL statements in the optimistic shard DDL mode might block the task #1051
• Fix the issue that a DM-master might try to forward requests to other DM-master nodes after it becomes the leader #1157
• Fix the issue that DM cannot parse GRANT CREATE TABLESPACE during the precheck #1113
• Fix the issue that migration tasks are interrupted when migrating DROP TABLE statements but corresponding tables don’t exist #990
• Fix the issue that operate-schema might not work properly when the --source parameter is specified #1106
• Fix the issue that list-member cannot be executed correctly after enabling TLS #1050
• Fix the issue that mixing https and http in the config items might cause the cluster to not work properly after enabling TLS #1220
• Fix the issue that the HTTP API cannot work properly after configuring the cert-allowed-cn parameter for DM-masters #1036
• Fix the issue that for incremental replication tasks, the configuration check fails when binlog-gtid is only specified in the meta of the task configuration #987
• Fix the issue that in the interactive mode, dmctl cannot correctly execute some commands starting or ending with blank characters #1202
• Fix the issue that the converting NULL to string is unsupported error is output to the log file during the full export #1014
• Fix the issue that the progress might be displayed as NaN during the full import #1209

Action required

• When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together