Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery RANGE (and INTERVAL) support #1292

Open
0rnella opened this issue Sep 12, 2024 · 1 comment
Open

BigQuery RANGE (and INTERVAL) support #1292

0rnella opened this issue Sep 12, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@0rnella
Copy link

0rnella commented Sep 12, 2024

Context

I have a table in BigQuery with columns of the data type RANGE. When I attempt to read the table with the BigQuery spark connector (in a BigQuery stored procedure), I get the following error:

py4j.protocol.Py4JJavaError: An error occurred while calling o918.load.

: java.lang.IllegalStateException: Unexpected type: RANGE
	at com.google.cloud.spark.bigquery.SchemaConverters.getStandardDataType(SchemaConverters.java:421)
	at com.google.cloud.spark.bigquery.SchemaConverters.lambda$getDataType$3(SchemaConverters.java:340)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at com.google.cloud.spark.bigquery.SchemaConverters.getDataType(SchemaConverters.java:340)
	at com.google.cloud.spark.bigquery.SchemaConverters.convert(SchemaConverters.java:286)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at com.google.cloud.spark.bigquery.SchemaConverters.toSpark(SchemaConverters.java:69)
	at com.google.cloud.spark.bigquery.v2.Spark3Util.lambda$createBigQueryTableInstance$0(Spark3Util.java:63)
	at com.google.cloud.spark.bigquery.v2.Spark31BigQueryTable.schema(Spark31BigQueryTable.java:87)
	at com.google.cloud.spark.bigquery.v2.Spark31BigQueryTableProvider.inferSchema(Spark31BigQueryTableProvider.java:40)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:90)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:140)
	at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:210)
	at scala.Option.flatMap(Option.scala:283)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:208)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

I also get a similar error with columns of type INTERVAL, but I do realize that the INTERVAL data type is still pre-GA as of now, so that is pretty much expected.

Neither RANGE nor INTERVAL (again, the latter is totally fine) is currently listed among the Data Types in the README.
Are there plans yet to introduce support for RANGE?

Please let me know if I can provide any more information, and thank you.

@isha97
Copy link
Member

isha97 commented Sep 23, 2024

Adding support for RANGE and INTERVAL is on our plan but we don't have any eta for that yet.

@isha97 isha97 added the enhancement New feature or request label Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants