-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sensor Agent Doc Integration Example #1180
Changes from 22 commits
3303706
badf470
766c99b
c4212cb
e6662e9
bfd74ef
52bb657
3665a47
cc20402
41eedbe
edc829b
ba0cbd1
be664cb
60a89ca
bbb06a8
04d5f9e
dc15e86
704b4eb
ddf6d9b
6dffcc9
42b1ee6
9867fd3
6076178
ca50b8d
c81cf13
45bdfaf
a1a6b9e
b53f2bd
b9f6ea0
71f51a8
3f751ed
e42c965
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
(sensor)= | ||
|
||
# Sensor | ||
|
||
```{eval-rst} | ||
.. tags:: Data, Basic | ||
``` | ||
|
||
## Run the example on the Flyte cluster | ||
|
||
To run the provided example on the Flyte cluster, use the following command: | ||
|
||
``` | ||
pyflyte run --remote \ | ||
https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/sensor/sensor/file_sensor_example.py wf | ||
``` | ||
|
||
```{auto-examples-toc} | ||
sensor | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
# %% [markdown] | ||
# # File Sensor | ||
# | ||
# This example shows how to use the `FileSensor` to detect files appearing in your local or remote filesystem. | ||
# | ||
# To begin, import the required libraries. | ||
|
||
# %% | ||
from flytekit import task, workflow | ||
from flytekit.sensor.file_sensor import FileSensor | ||
|
||
# %% [markdown] | ||
# Create a FileSensor task. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I recommend changing this to "Next, create a FileSensor task" and moving the next code block directly under this sentence. |
||
# | ||
# The sensor will search for the file at the specified path. If the file exists, it will return a succeed status. Otherwise, the sensor will continue running until the file is added to the directory. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I recommend changing this to "To use the FileSensor created in the previous step, you must specify the |
||
# | ||
# You can also use the S3 or GCS file system. | ||
|
||
# %% | ||
sensor = FileSensor(name="test_file_sensor") | ||
|
||
# %% [markdown] | ||
# You have to specify the `path` parameter. | ||
# | ||
# In the sandbox, you can use the s3 path. | ||
# | ||
# We have already set the minio credentials in the agent by default. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I recommend moving this section the very bottom and putting it in a note, like so:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note that you may have to tweak the formatting to get the code snippet to appear correctly in the note -- I just started, and am not entirely familiar with how these docs are formatted. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @neverett Please review again! Thanks really much! |
||
# | ||
# If you test it locally, you have to set the credentials to your environment variables. | ||
# ```{prompt} bash | ||
# export FLYTE_AWS_ENDPOINT="http://localhost:30002" | ||
# export FLYTE_AWS_ACCESS_KEY_ID="minio" | ||
# export FLYTE_AWS_SECRET_ACCESS_KEY="miniostorage" | ||
# ``` | ||
|
||
|
||
# %% | ||
@task() | ||
def t1(): | ||
print("SUCCEEDED") | ||
|
||
|
||
@workflow() | ||
def wf(): | ||
sensor(path="s3://my-s3-bucket/file.txt") >> t1() | ||
|
||
|
||
if __name__ == "__main__": | ||
wf() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I recommend "First" instead of "To begin"