Review the new DVC sharing experiments workflow and unify it with VS Code

[shcheklein]

With iterative/dvc#9074 and https://github.com/iterative/studio/issues/5050 DVC introduced a way to connect to Studio and a workflow that sends experiments to Studio.

We need to review and unify/make it compatible:

• Token management Studio token: make it global, make it easier to setup dvc#9265 (Use dvc config to store and access studio.token #3768)
• Auto-sharing mode - does it contradict or not with existing DVC workflow? - no, leave as is
• Drop custom client implementation (Use exp push to share experiments to Studio #3701)
• Consolidate share actions into one - Review the new DVC sharing experiments workflow and unify it with VS Code #3574 (comment) (Remove commit and share and share as branch options #3771 + Integrate remaining share experiment command with the extension (exp push) #3781)
• Improvement: Sharing multiple experiments action Review the new DVC sharing experiments workflow and unify it with VS Code #3574 (comment) (Add multi-select versions of exp push #3792)
• When a user is not already connected to Studio prompt with a toast to connect on exp push Review the new DVC sharing experiments workflow and unify it with VS Code #3574 (comment) (Integrate remaining share experiment command with the extension (exp push) #3781)
• Add instructions to live share outside of the extension context (Review the new DVC sharing experiments workflow and unify it with VS Code #3574 (comment)) - not needed (Review the new DVC sharing experiments workflow and unify it with VS Code #3574 (comment))
• Sharing experiments. Can we detect that it's already being shared? Show a link?

[dberenbaum]

Can we detect that it's already being shared? Show a link?

We could fetch from the git remote or use dvc exp list to see what's been pushed. Do we need anything specific to Studio?

I'm a bit behind on the Studio efforts to have more shareable links, but I think we can follow it in https://github.com/iterative/studio/issues/5419 and https://github.com/iterative/studio/issues/5150.

• Auto-sharing mode - does it contradict or not with existing DVC workflow?

I guess it means this:

I don't think it contradicts. Does it do anything besides add the STUDIO_TOKEN to experiment actions @mattseddon? I think it's needed since there's no other obvious way to enable this behavior when working in the extension.

• Do we still need to use our own client implementation or not?

I hope not.

[dberenbaum]

• Sharing experiments. Can we detect that it's already being shared? Show a link?

I don't know if we have a good way to detect what experiments have been live shared. For exp refs, we could use dvc exp list to check the git remote. If they exist there, there should be a record in Studio, although it's possible it hasn't been retrieved/parsed yet by Studio.

I would add an item to consolidate and unify the sharing options. Here's what I see today:

Overall, I think we can have a single "Share" action that relies on dvc exp push now and drop the existing different ways to share. This will hopefully make sharing more straightforward.

Now Studio should show any experiments that are pushed to the Git remote via dvc exp push, so we shouldn't need to rely on the live metrics API anymore. dvc exp push will notify Studio if STUDIO_TOKEN is set or if the token is available in studio.token in the DVC config (depends on iterative/dvc#9257; we will likely recommend using dvc config --global studio.token xxx for CLI users).

dvc exp push will push the cache by default. Not sure if we need an option to disable that, but we should at least be aware that it could make pushing slow.

[shcheklein]

Sharing experiments. Can we detect that it's already being shared? Show a link?

I think DVCLive creates some special signal file with a process pid? Can we utilize it cc @mattseddon @daavoo .

On the "share" actions. I agree, @dberenbaum we can add item to the list to consolidate this and simplify. We still keep an apply action though, right? So that I can make locally a commit out of an experiment at least. We just don't send these commits, neither we try to do dvc push, etc.

[dberenbaum]

We still keep an apply action though, right? So that I can make locally a commit out of an experiment at least. We just don't send these commits, neither we try to do dvc push, etc.

Agreed, we can keep local actions like "Apply to workspace" and "Create new branch," I just don't think we need to combine them with the sharing action.

[mattseddon]

I would add an item to consolidate and unify the sharing options. Here's what I see today:

Overall, I think we can have a single "Share" action that relies on dvc exp push now and drop the existing different ways to share. This will hopefully make sharing more straightforward.

Now Studio should show any experiments that are pushed to the Git remote via dvc exp push, so we shouldn't need to rely on the live metrics API anymore. dvc exp push will notify Studio if STUDIO_TOKEN is set or if the token is available in studio.token in the DVC config (depends on iterative/dvc#9257; we will likely recommend using dvc config --global studio.token xxx for CLI users).

I'm happy to drop Commit and Share + Branch and Share and update Share to Studio to use exp push and label it with simply Share. My one question would be "how do we promote Studio as a sharing option"? Currently, with the custom implementation of the Studio client and the command's label, we can do a few things:

1. Promote Studio as a sharing option.
2. We can check to see if there is a Studio token set for the extension and redirect to the setup page if there isn't one present.
3. If we receive a 401 we can notify the user that they need to update their token.

Should we drop this logic altogether and not worry about the edge cases?
How do we show users that sharing to Studio is an option?

It feels like we need an API that we can use at startup to check whether or not the user has a valid token and if not prompt them to add one/update. Otherwise, we can't provide much help to users once they leave the happy path.

dvc exp push will push the cache by default. Not sure if we need an option to disable that, but we should at least be aware that it could make pushing slow.

From testing out with the demo project moving from the old API to using exp push blew the time out to share an experiment from 1s to ~90s, after the initial push this dropped to 14s. It would be good to be able to reduce that time in any way possible.

[mattseddon]

FWIW I also ran into the need to authenticate the process running DVC with Github today. I need to work out how to get access to the credentials using the VS Code environment.

[dberenbaum]

I'm happy to drop Commit and Share + Branch and Share and update Share to Studio to use exp push and label it with simply Share. My one question would be "how do we promote Studio as a sharing option"? Currently, with the custom implementation of the Studio client and the command's label, we can do a few things:

Sorry, I'm getting confused by all the different ways to share. Do you mean how do we promote live sharing to Studio, or any kind of sharing to Studio?

It feels like we need an API that we can use at startup to check whether or not the user has a valid token and if not prompt them to add one/update.

Not sure I follow all the suggestions, but can we start simpler by checking whether the user has a token at all and assume it's valid if it exists? I think checking the token validity is less of a priority for now.

From testing out with the demo project moving from the old API to using exp push blew the time out to share an experiment from 1s to ~90s, after the initial push this dropped to 14s.

Are you testing with checkpoints?

[mattseddon]

Sorry, I'm getting confused by all the different ways to share. Do you mean how do we promote live sharing to Studio, or any kind of sharing to Studio?

Any kind of sharing to Studio. How do we highlight this as a collaboration option?

Note: I can see from iterative/dvc.org#4470 that if a user has a token present in their config then they can live share experiments but not sure what the default behaviour is.

Not sure I follow all the suggestions, but can we start simpler by checking whether the user has a token at all and assume it's valid if it exists? I think checking the token validity is less of a priority for now.

I think we need a Studio API that we can check auth details against but this should be fixed/made obsolete by https://github.com/iterative/studio/issues/5158.

A couple of follow-up questions:

• Can I use dvc config studio.token to check if a user has a Studio token in their config? (if yes I can use it as an environment variable if "live sharing" is enabled)
• When setting the token into a user's config should I set it as --local, --global or --system? (my guess is --global)

If users have already set this up I think we should still prompt them to add again as this makes it explicit that we are saving it in their DVC config (rather than expending effort migrating the secret).

Are you testing with checkpoints?

In this instance no, checkpoints were dropped on the branch that I was testing with.