Skip to content

AutoMigrate can break clickhouse data if you change a column type #125

Open
@SergFilippov

Description

@SergFilippov

GORM Playground Link

go-gorm/playground#639

Description

When changing the column type AutoMigrate performs ALTER TABLE MODIFY COLUMN on each field every time migration is started.

If the old type is compatible with the new one (e.g. Int32->Int64), the type is changed and there seems to be no problem. But if not (e.g. from String to Int64) then ALTER TABLE MODIFY COLUMN is executed and a mutation is created which fails with an error:

code: 341, message: Exception happened during execution of mutation 'mutation_2.txt' with part '202308_1_1_0' reason: 'Code: 6. DB::Exception: Cannot parse string 'string value' as Int32: syntax error at begin of string. Note: there are toInt32OrZero and toInt32OrNull functions, which returns zero/NULL instead of throwing exception.: while executing 'FUNCTION _CAST(value :: 0, 'Int32' :: 2) -> _CAST(value, 'Int32') Int32 : 3': (while reading from part /var/lib/clickhouse/store/a35/a3515250-be07-4ea7-a399-d406026db762/202308_1_1_0/ located on disk default of type local): While executing MergeTreeSequentialSource. (CANNOT_PARSE_TEXT) (version 23.7.2.25 (official build))'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query

This actually changes the data type in Clickhouse and breaks data. The query "SELECT * FROM events" gives an error:

Code: 6. DB::Exception: Cannot parse string 'string value' as Int32: syntax error at begin of string. Note: there are toInt32OrZero and toInt32OrNull functions, which returns zero/NULL instead of throwing exception.: while executing 'FUNCTION _CAST(value :: 0, 'Int32' :: 2) -> _CAST(value, 'Int32') Int32 : 3': (while reading from part /var/lib/clickhouse/store/50c/50c405ac-c069-4d7f-b816-ec2aec6ed96a/202308_1_1_0/ located on disk default of type local): While executing MergeTreeInOrder. (CANNOT_PARSE_TEXT) (version 23.7.2.25 (official build))

You can see the example of code in main.go in playground above

Expected behavior

If the old type is incompatible with the new AutoMigrate does not execute the ALTER TABLE MODIFY COLUMN query, the mutation is not created.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions