feat: `Datetime(time_unit, time_zone)` and `Duration(time_unit)` types #960

FBruzzesi · 2024-09-13T07:40:29Z

What type of PR is this? (check all applicable)

Related issues

Closes [Enh]: Add time units and time zone specifics #467

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below.

Introduces time units and time zones in Datetime type.

FBruzzesi · 2024-09-13T07:41:00Z

tests/series_only/cast_test.py

@@ -75,13 +75,13 @@ def test_cast_date_datetime_pandas() -> None:
    df = df.select(nw.col("a").cast(nw.Datetime))
    result = nw.to_native(df)
    expected = pd.DataFrame({"a": [datetime(2020, 1, 1), datetime(2020, 1, 2)]}).astype(
-        {"a": "timestamp[ns][pyarrow]"}
+        {"a": "timestamp[us][pyarrow]"}


Changes the default to the polars one

FBruzzesi · 2024-09-13T07:42:00Z

tests/expr_and_series/cast_test.py

+        nw.col("date")
+        .cast(nw.Datetime("ms", time_zone="Europe/Rome"))
+        .cast(nw.String())
+        .str.slice(offset=0, length=19)


19: number of characters of "2024-01-01 01:00:00". The format right after that is different for each backend

FBruzzesi · 2024-09-13T07:43:07Z

narwhals/_pandas_like/utils.py

+    pd_datetime_rgx = (
+        r"^datetime64\[(?P<time_unit>ms|us|ns)(?:, (?P<time_zone>[a-zA-Z\/]+))?\]$"
+    )
+    pa_datetime_rgx = r"^timestamp\[(?P<time_unit>ms|us|ns)(?:, tz=(?P<time_zone>[a-zA-Z\/]+))?\]\[pyarrow\]$"


Please try to break these 🙈

FBruzzesi · 2024-09-13T08:32:18Z

narwhals/_pandas_like/utils.py

+        # Pandas does not support "ms" or "us" time units before version 1.5.0
+        # Let's overwrite with "ns"
+        if implementation is Implementation.PANDAS and backend_version < (
+            1,
+            5,
+            0,
+        ):  # pragma: no cover
+            time_unit = "ns"


I don't think we can do much else here

FBruzzesi · 2024-09-13T08:47:40Z

@MarcoGorelli any experience on how to locate timezone database at "C:\Users\runneradmin\Downloads\tzdata" ? 😂

MarcoGorelli · 2024-09-14T13:18:31Z

wow, nice one! i'll try breaking it a bit, but this is amazing, been wanting to do this for a while 🚀

MarcoGorelli · 2024-09-14T18:06:27Z

I think the dtype comparison isn't quite right:

In [17]: s
Out[17]:
shape: (1,)
Series: '' [datetime[μs, Asia/Kathmandu]]
[
        2019-12-31 18:15:00 +0545
]

In [18]: nw.from_native(s, allow_series=True).dtype == nw.Datetime('us')
Out[18]: True

In [19]: s.dtype == pl.Datetime('us')
Out[19]: False

FBruzzesi · 2024-09-14T18:12:38Z

I think the dtype comparison isn't quite right:

In [17]: s
Out[17]:
shape: (1,)
Series: '' [datetime[μs, Asia/Kathmandu]]
[
        2019-12-31 18:15:00 +0545
]

In [18]: nw.from_native(s, allow_series=True).dtype == nw.Datetime('us')
Out[18]: True

In [19]: s.dtype == pl.Datetime('us')
Out[19]: False

Nice catch! Thanks! Will fix later on 👌

MarcoGorelli · 2024-09-14T20:42:28Z

🤔 interesting, so elif dtype in {nw.Datetime, nw.Date}: breaks if we add the time_unit and time_zone attributes

narwhals/dtypes.py

…e-datetime

MarcoGorelli · 2024-09-29T20:49:42Z

Looks like this might be it 🥳 🍾 can't believe it...this took hours...worth it though. It means we can change the main narwhals namespace, with zero impact on Altair users. as it turns out, the stable api was really worth doing... this is so rewarding 🕺

i think the nightly ci failure is unrelated

i'll check this again tomorrow, then hopefully we can make a release with this in on Tuesday

FBruzzesi · 2024-09-29T21:28:44Z

That's awesome! I will take some time this upcoming week to check what happened in detail!
Also plotly has a test with specific time-zone, thus I am looking forward to the release of this feature 😁

However, to use the "edge" dtypes (or in general features), should we

+ import narwhals as nw
- import narwhals.stable.v1 as nw

?

MarcoGorelli · 2024-09-30T12:14:08Z

narwhals/_pandas_like/utils.py

+    pd_duration_rgx = r"^timedelta64\[(?P<time_unit>ms|us|ns)\]$"
+    pa_duration_rgx = r"^duration\[(?P<time_unit>ms|us|ns)\]\[pyarrow\]$"


pandas / pyarrow support 'second' time unit, I think that should be allowed to pass through

Just by passing it along or doing manipulation for the user?

i think we can just pass it through - adding a commit soon

MarcoGorelli · 2024-09-30T12:15:36Z

However, to use the "edge" dtypes (or in general features), should we

it's fine to use time_unit / time_zone with stable.v1, it's just probably a good idea to avoid anything which calls __hash__. For example, instead of series.dtype in {nw.Datetime, nw.Date}, it'd be safer to do series.dtype == nw.Datetime or series.dtype == nw.Date

MarcoGorelli · 2024-09-30T15:30:45Z

narwhals/_pandas_like/utils.py

+        du_time_unit = getattr(dtype, "time_unit", "us")
+        return (
+            f"duration[{du_time_unit}][pyarrow]"
+            if dtype_backend == "pyarrow-nullable"
+            else f"timedelta64[{du_time_unit}]"
+        )


we need to do the same pre-1.5.0 check here

narwhals/_pandas_like/utils.py

MarcoGorelli

alright, happy with this 🥳 let's ship it?

FBruzzesi · 2024-10-01T08:10:59Z

Thanks for taking care of this Marco! But most importantly, this is the cutest gif so far! How did you discover it?
FYI: @anopsy

anopsy · 2024-10-01T08:15:49Z

This is uber cute!

narwhals-dev#960)

FBruzzesi added 3 commits September 12, 2024 22:41

WIP

121f6f8

order matters?

4896df2

datetime test and polars fix

cd2ed40

github-actions bot added the enhancement New feature or request label Sep 13, 2024

FBruzzesi commented Sep 13, 2024

View reviewed changes

FBruzzesi added 3 commits September 13, 2024 09:47

rm NoneType

eb1468e

pandas pre 1.5

e71f9c3

no cover backend version branch

32385d0

FBruzzesi commented Sep 13, 2024

View reviewed changes

FBruzzesi added 4 commits September 13, 2024 11:53

add pytz to dev requirements for testing

3abeaf8

merge main

c5b7635

xfail pyarrow table on windows

4415e3c

Duration(time_unit)

5309d4f

FBruzzesi changed the title ~~feat: time zone aware Datetime type~~ feat: Datetime(time_unit, time_zone) and Duration(time_unit) types Sep 14, 2024

Merge branch 'main' into feat/time-zone-aware-datetime

85fdd80

FBruzzesi added 2 commits September 14, 2024 19:40

Merge branch 'main' into feat/time-zone-aware-datetime

91bfb7a

add Datetime and Duration methods, as in polars

20e36a1

downstream?

ec1cb5e

revert

2147ec6

MarcoGorelli reviewed Sep 14, 2024

View reviewed changes

narwhals/dtypes.py Show resolved Hide resolved

FBruzzesi added 3 commits September 14, 2024 23:01

hash class only

0f69ec1

else case no cover

22836a0

Merge branch 'main' into feat/time-zone-aware-datetime

a1f56bc

MarcoGorelli added 4 commits September 28, 2024 09:27

Merge remote-tracking branch 'upstream/main' into feat/time-zone-awar…

e94b517

…e-datetime

broken, but getting there?

180b86e

Merge remote-tracking branch 'upstream/main' into feat/time-zone-awar…

da884e8

…e-datetime

fixup

114be74

MarcoGorelli added 5 commits September 29, 2024 21:51

reduce diff

587d917

stableify duration too

dd050a8

test duration too

b4de1f7

try removing pytz

458f2a2

try fix ci

34c27ef

MarcoGorelli added 4 commits September 29, 2024 22:29

try fix ci

0de71a6

try fix ci

d105911

try fix ci

a773d85

try fix ci

0149431

MarcoGorelli reviewed Sep 30, 2024

View reviewed changes

MarcoGorelli added 2 commits September 30, 2024 13:49

allow s time unit

2249af0

test second resolution

942a77b

MarcoGorelli reviewed Sep 30, 2024

View reviewed changes

narwhals/_pandas_like/utils.py Outdated Show resolved Hide resolved

MarcoGorelli added 2 commits September 30, 2024 17:14

override duration time unit for pandas pre 2.0

ad38667

🏷️

38898a8

MarcoGorelli approved these changes Sep 30, 2024

View reviewed changes

pre-2.0 pandas

43da4c3

MarcoGorelli merged commit 93d2fc7 into main Sep 30, 2024
25 checks passed

FBruzzesi deleted the feat/time-zone-aware-datetime branch October 1, 2024 08:11

akmalsoliev pushed a commit to akmalsoliev/narwhals that referenced this pull request Oct 15, 2024

feat: Datetime(time_unit, time_zone) and Duration(time_unit) types (

ae81983

narwhals-dev#960)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `Datetime(time_unit, time_zone)` and `Duration(time_unit)` types #960

feat: `Datetime(time_unit, time_zone)` and `Duration(time_unit)` types #960

FBruzzesi commented Sep 13, 2024 •

edited

Loading

FBruzzesi Sep 13, 2024

FBruzzesi Sep 13, 2024

FBruzzesi Sep 13, 2024

FBruzzesi Sep 13, 2024

FBruzzesi commented Sep 13, 2024

MarcoGorelli commented Sep 14, 2024

MarcoGorelli commented Sep 14, 2024

FBruzzesi commented Sep 14, 2024

MarcoGorelli commented Sep 14, 2024

MarcoGorelli commented Sep 29, 2024

FBruzzesi commented Sep 29, 2024

MarcoGorelli Sep 30, 2024

FBruzzesi Sep 30, 2024

MarcoGorelli Sep 30, 2024

MarcoGorelli commented Sep 30, 2024

MarcoGorelli Sep 30, 2024

MarcoGorelli left a comment

FBruzzesi commented Oct 1, 2024 •

edited

Loading

anopsy commented Oct 1, 2024

		pd_duration_rgx = r"^timedelta64\[(?P<time_unit>ms\|us\|ns)\]$"
		pa_duration_rgx = r"^duration\[(?P<time_unit>ms\|us\|ns)\]\[pyarrow\]$"

feat: Datetime(time_unit, time_zone) and Duration(time_unit) types #960

feat: Datetime(time_unit, time_zone) and Duration(time_unit) types #960

Conversation

FBruzzesi commented Sep 13, 2024 • edited Loading

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FBruzzesi commented Sep 13, 2024

MarcoGorelli commented Sep 14, 2024

MarcoGorelli commented Sep 14, 2024

FBruzzesi commented Sep 14, 2024

MarcoGorelli commented Sep 14, 2024

MarcoGorelli commented Sep 29, 2024

FBruzzesi commented Sep 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli commented Sep 30, 2024

Choose a reason for hiding this comment

MarcoGorelli left a comment

Choose a reason for hiding this comment

FBruzzesi commented Oct 1, 2024 • edited Loading

anopsy commented Oct 1, 2024

feat: `Datetime(time_unit, time_zone)` and `Duration(time_unit)` types #960

feat: `Datetime(time_unit, time_zone)` and `Duration(time_unit)` types #960

FBruzzesi commented Sep 13, 2024 •

edited

Loading

FBruzzesi commented Oct 1, 2024 •

edited

Loading