Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-636] Support GEOMETRY in Spark SQL table creation DDL #1543

Merged
merged 11 commits into from
Aug 12, 2024

Conversation

zhangfengcdt
Copy link
Contributor

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

This PR add the geometry type in the table creation DDL. For example:

CREATE TABLE T_TEST_EXPLICIT_GEOMETRY (GEO_COL GEOMETRY)

CREATE TABLE T_TEST_EXPLICIT_GEOMETRY_2 (INT_COL INT, GEO_COL GEOMETRY)

Before the change, it throws error below:

DataType geometryee is not supported.(line 1, pos 62)

== SQL ==
CREATE TABLE T_TEST_EXPLICIT_GEOMETRY_2 (INT_COL INT, GEO_COL GEOMETRY)
--------------------------------------------------------------^^^

How was this patch tested?

new unit test: SQLSyntaxTestScala

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@zhangfengcdt zhangfengcdt marked this pull request as ready for review August 7, 2024 21:49
@zhangfengcdt
Copy link
Contributor Author

There are breaking changes to the SparkSqlParser and SparkSqlAstBuilder, that removed the configuration from the constructor (committed on Nov 14, 2020 ). The spark 3.0.3 thus failed.

apache/spark@156704b

@jiayuasu
Copy link
Member

What happens to the spark-3.0 profile?

I understand that it could create some redundant code but I believe we should still replicate the tests to all profiles. What do you think? @zhangfengcdt

@zhangfengcdt
Copy link
Contributor Author

What happens to the spark-3.0 profile?

I understand that it could create some redundant code but I believe we should still replicate the tests to all profiles. What do you think? @zhangfengcdt

@jiayuasu I've just added the unit tests to all spark versions we currently support. For spark-3.0, I also refactor the custom sedona sql parser constructor to take the extra conf as the legacy spark codebase requires.

@jiayuasu jiayuasu added this to the sedona-1.6.1 milestone Aug 12, 2024
@jiayuasu jiayuasu merged commit 43cab48 into apache:master Aug 12, 2024
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

datatype geometry is not supported when 'create table xxx (geom geometry)'
2 participants