Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Real time deck support #1704

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Yicheng-Lu-llll
Copy link
Member

@Yicheng-Lu-llll Yicheng-Lu-llll commented Jun 22, 2023

TL;DR

This PR adds real-time deck support.

After this PR, User can use flytekit.Deck.persist() to generate the html based on current collected metrics/information. So that user can see the real time deck even the task is running or has already failed. See the below example:

import flytekit
from flytekit import Resources, task, workflow
from flytekit.core.utils import timeit

@task(
    disable_deck=False,
    limits=Resources(mem="4Gi", cpu="1"),
)
def t1():
    import time

    for i in range(2):
        # timeit measure the time used in the block and shown in time line deck. See https://github.com/flyteorg/flytekit/pull/1581. 
        # Or you can add information to your own deck. See https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/deck.html.
        with timeit(f"iteration {i}"):
            time.sleep(50)
        flytekit.Deck.persist()


@workflow
def wf():
    t1()


if __name__ == "__main__":
    wf()
截屏2023-06-23 22 39 08 截屏2023-06-23 22 39 23

real time deck support includes:

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Signed-off-by: Yicheng-Lu-llll <[email protected]>
Signed-off-by: Yicheng-Lu-llll <[email protected]>
Signed-off-by: Yicheng-Lu-llll <[email protected]>
@fg91
Copy link
Member

fg91 commented Jun 30, 2023

Just realizing we could use this feature to solve this problem: provide the ability to display links to e.g. wandb runs which are generated at runtime before the task ends 🎉

@cosmicBboy
Copy link
Contributor

This also solves the problem with eager mode, i.e. we can re-render the Flyte deck of subtasks being kicked-off by the eager workflow in real-time.

@@ -72,6 +72,12 @@ def name(self) -> str:
def html(self) -> str:
return self._html

@classmethod
def persist(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think self should be cls

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I see. Thank you for pointing out!

@fg91
Copy link
Member

fg91 commented Jul 25, 2023

How does this effort here relate to the discussion in flyteorg/flyte#3838 btw?
(cc @hamersaw)

@hamersaw
Copy link
Contributor

hamersaw commented Jul 25, 2023

@fg91 yeah, this is one of the issues that is effectively blocked, I would rather merge this with a re-usable flytekit -> flytepropeller communication mechanism.

@ketian-indeed
Copy link

Sorry to bump up on this PR, but this seems like a pretty useful feature to us. Is there any update on it? Thx!

@kumare3
Copy link
Contributor

kumare3 commented May 29, 2024

@ketian-indeed would love to understand the type of visualization you want

@ketian-indeed
Copy link

@kumare3, thanks for the quick response. I'm thinking of a use case like dumping a link (e.g. Datadog/WandB/MLflow links that are valuable to check while a model training is ongoing) to the Deck when a task starts and being able to see the link on the Deck while the task is running. I believe the Deck is currently shown once the task is completed - please correct me if I'm wrong.

@kumare3
Copy link
Contributor

kumare3 commented May 30, 2024

Hi @ketian-indeed we are infact adding support for W&B, Comet and neptune natively into flyte, Check this out. #2449

Also Datadog you should be able to add already using,
https://docs.flyte.org/en/latest/user_guide/productionizing/configuring_logging_links_in_the_ui.html#configure-logging

Please let me know what you think?

But, i also like the idea of Realtime decks - we will be adding support for this, but not like this PR. slightly different.

@ketian-indeed
Copy link

Thanks, @kumare3! Configuring logging links in the UI seems to be a better solution for some of our use cases. I'll test that out.
Meanwhile, I'm also looking forward to the real-time deck support, which I believe will provide even more flexibility for visualization.

@kumare3
Copy link
Contributor

kumare3 commented May 30, 2024

I think that is fair. We will prioritize this work in summer. But please add ideas / use cases here that way it is very easy to ensure we address. If you are open to contributing that would be awesome too.

@fg91
Copy link
Member

fg91 commented May 30, 2024

But, i also like the idea of Realtime decks - we will be adding support for this, but not like this PR. slightly different.

I think that is fair. We will prioritize this work in summer. But please add ideas / use cases here that way it is very easy to ensure we address. If you are open to contributing that would be awesome too.

This, this, and this discussion are relevant in this context.

TL;DR multiple use cases are currently blocked by not having a real-time communication channel between a task pod and propeller, e.g. live updates in the eager mode deck or adding a log link dynamically at runtime:

@task
def train():
    run_url = my_experiment_tracking_api.init_run()
    flytekit.current_context().add_log_link(run_url)

@kumare3 kumare3 mentioned this pull request Oct 9, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants