Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FSTORE-1218] Fix as_of and read_changes doc #1216

Merged
merged 9 commits into from
Feb 19, 2024
Merged

[FSTORE-1218] Fix as_of and read_changes doc #1216

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bde7954
add warning that as_of and read_changes methods are not availble in S…
Feb 7, 2024
e8eff76
added doc as_of and read_changes
Feb 7, 2024
d60e7f8
Fix doc banner for as_of and read_changes
Feb 7, 2024
e535706
Fixed as_of doc in both query and fg section
Feb 7, 2024
1780506
reprase as_of and read_changes banner for python availability
Feb 7, 2024
86a0ac1
Rephrase as_of and read_changes HUDI warnings. Merge with the partial…
Feb 7, 2024
00e15e5
rephrased as_of and read_changes warnings. restored old warnings as well
Feb 8, 2024
643c05a
Merge branch 'logicalclocks:master' into FSTORE-1218
Feb 15, 2024
eda44e1
Merge branch 'master' into FSTORE-1218
vatj Feb 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions python/hsfs/constructor/query.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,9 @@ def as_of(
):
"""Perform time travel on the given Query.

!!! warning
HUDI supports Time Travel and Incremental Query via Spark context, exclusively in PySpark/Spark

This method returns a new Query object at the specified point in time. Optionally, commits before a
specified point in time can be excluded from the query. The Query can then either be read into a Dataframe
or used further to perform joins or construct a training dataset.
Expand Down Expand Up @@ -315,9 +318,6 @@ def as_of(
.as_of(..., ...)
```

!!! warning
This function only works for queries on feature groups with time_travel_format='HUDI'.

!!! warning
Excluding commits via exclude_until is only possible within the range of the Hudi active timeline.
By default, Hudi keeps the last 20 to 30 commits in the active timeline.
Expand Down
9 changes: 5 additions & 4 deletions python/hsfs/feature_group.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2110,7 +2110,8 @@ def read_changes(
`as_of(end_wallclock_time, exclude_until=start_wallclock_time).read(read_options=read_options)`
instead.

This function only works on feature groups with `HUDI` time travel format.
!!! warning "Warning"
HUDI supports Time Travel and Incremental Query via Spark context, exclusively in PySpark/Spark
vatj marked this conversation as resolved.
Show resolved Hide resolved

# Arguments
start_wallclock_time: Start time of the time travel query. Strings should be formatted in one of the following formats `%Y-%m-%d`, `%Y-%m-%d %H`, `%Y-%m-%d %H:%M`,
Expand Down Expand Up @@ -2847,6 +2848,9 @@ def as_of(
):
"""Get Query object to retrieve all features of the group at a point in the past.

!!! warning
HUDI supports Time Travel and Incremental Query via Spark context, exclusively in PySpark/Spark
vatj marked this conversation as resolved.
Show resolved Hide resolved

This method selects all features in the feature group and returns a Query object
at the specified point in time. Optionally, commits before a specified point in time can be
excluded from the query. The Query can then either be read into a Dataframe
Expand Down Expand Up @@ -2911,9 +2915,6 @@ def as_of(
.as_of("2020-10-20", exclude_until="2020-10-19")
```

!!! warning
This function only works for feature groups with time_travel_format='HUDI'.

vatj marked this conversation as resolved.
Show resolved Hide resolved
!!! warning
Excluding commits via exclude_until is only possible within the range of the Hudi active timeline.
By default, Hudi keeps the last 20 to 30 commits in the active timeline.
Expand Down
Loading