Skip to content

Commit

Permalink
[SPARK-48340][PYTHON][FOLLOWUP] Support TimestampNTZ infer schema mis…
Browse files Browse the repository at this point in the history
…s prefer_timestamp_ntz

### What changes were proposed in this pull request?
Add UT

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Added UT

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#46750 from AngersZhuuuu/SPARK-48340-FOLLOWUP.

Lead-authored-by: Angerszhuuuu <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
AngersZhuuuu and HyukjinKwon committed May 27, 2024
1 parent 90c1637 commit 49da3a4
Showing 1 changed file with 14 additions and 1 deletion.
15 changes: 14 additions & 1 deletion python/pyspark/sql/tests/test_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ def __init__(self):
Row(a=1),
Row("a")(1),
A(),
Row(b=Row(c=datetime.datetime(1970, 1, 1, 0, 0))),
]

df = self.spark.createDataFrame([data])
Expand All @@ -214,6 +215,7 @@ def __init__(self):
"struct<a:bigint>",
"struct<a:bigint>",
"struct<a:bigint>",
"struct<b:struct<c:timestamp>>",
]
self.assertEqual(actual, expected)

Expand All @@ -236,14 +238,25 @@ def __init__(self):
Row(a=1),
Row(a=1),
Row(a=1),
Row(b=Row(c=datetime.datetime(1970, 1, 1, 0, 0))),
]
self.assertEqual(actual, expected)

with self.sql_conf({"spark.sql.timestampType": "TIMESTAMP_NTZ"}):
with self.sql_conf({"spark.sql.session.timeZone": "America/Sao_Paulo"}):
df = self.spark.createDataFrame([(datetime.datetime(1970, 1, 1, 0, 0),)])
data = [
(
datetime.datetime(1970, 1, 1, 0, 0),
Row(a=Row(a=datetime.datetime(1970, 1, 1, 0, 0))),
)
]
df = self.spark.createDataFrame(data)
self.assertEqual(list(df.schema)[0].dataType.simpleString(), "timestamp_ntz")
self.assertEqual(df.first()[0], datetime.datetime(1970, 1, 1, 0, 0))
self.assertEqual(
list(df.schema)[1].dataType.simpleString(), "struct<a:struct<a:timestamp_ntz>>"
)
self.assertEqual(df.first()[1], Row(a=Row(a=datetime.datetime(1970, 1, 1, 0, 0))))

df = self.spark.createDataFrame(
[
Expand Down

0 comments on commit 49da3a4

Please sign in to comment.