You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Changing the data type of a field that is referenced by a generated column or CHECK constraint is forbidden as this may change the result returned by that expression.
This is checked in AlterDeltaTableCommands for manual type changes and ImplicitMetadataOperation for type changes applied during schema evolution. These checks don't work correctly for fields nested inside a map or array.
This is because generated column and CHECK constraint expression use paths that referenced a specific element in the array or map, e.g. col.arr[0] whereas the path that the check looks for is a generic schema path to that element: col.element
Steps to reproduce
Create a table with schema a struct<arr: array<struct<x: smallint>>> and insert an int value for field x.
test("disallow column type evolution - nesting") {
Observed results
The write succeeds
Expected results
The operation should fail
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?
Yes. I can contribute a fix for this bug independently.
Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
No. I cannot contribute a bug fix at this time.
The text was updated successfully, but these errors were encountered:
johanl-db
changed the title
[BUG][Spark] Changing type of map/array field referenced by a generated column or CHECK constraint not failing
[BUG][Spark] Changing type of map/array field referenced by a generated column or CHECK constraint not failing correctly
Sep 3, 2024
Bug
Describe the problem
See #3601 (comment)
Changing the data type of a field that is referenced by a generated column or CHECK constraint is forbidden as this may change the result returned by that expression.
This is checked in AlterDeltaTableCommands for manual type changes and ImplicitMetadataOperation for type changes applied during schema evolution. These checks don't work correctly for fields nested inside a map or array.
This is because generated column and CHECK constraint expression use paths that referenced a specific element in the array or map, e.g.
col.arr[0]
whereas the path that the check looks for is a generic schema path to that element:col.element
Steps to reproduce
Create a table with schema
a struct<arr: array<struct<x: smallint>>>
and insert anint
value for fieldx
.See similar test:
delta/spark/src/test/scala/org/apache/spark/sql/delta/GeneratedColumnSuite.scala
Line 737 in 5d63094
Observed results
The write succeeds
Expected results
The operation should fail
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?
The text was updated successfully, but these errors were encountered: