Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing models does not alert experiments #568

Open
bderenzi opened this issue Jul 31, 2024 · 1 comment
Open

Removing models does not alert experiments #568

bderenzi opened this issue Jul 31, 2024 · 1 comment

Comments

@bderenzi
Copy link
Collaborator

Right now the model used is determined by the llm string and the llm_provider_id. If we change the available llm models (e.g., because a provider is deprecating old models), the experiment is never alerted / there is no check.

Possible solutions:

  • When we edit a provider we could check the old set of models against the new and for any that have been removed we could generate an error / warning to go and update the experiment to first remove dependency on the model that is getting removed.

That solution is a little annoying for the user unless in the same edit-provider UI we allow them to select a replacement. For example, imagine the case where we're removing llama-3 and adding llama-3.1. The steps would be:

  1. Edit the provider to remove llama-3 and add llama-3.1
  2. Get an error that projects X, Y, Z are all reliant on llama-3 so we can't remove them yet
  3. Change the edit to just add llama-3.1
  4. Go manually update all projects relying on the old model
  5. Return to the edit provider screen to remove llama-3

An alternative:

  1. Edit the provider to remove llama-3 and add llama-3.1
  2. Get an error that projects X, Y, Z are all reliant on llama-3 so we can't remove them yet. Ask for a replacement model that will be applied to all experiments that are affected
  3. Confirm replacement and be done.
@SmittieC
Copy link
Collaborator

Option 2 is deffs better UX. +1 for that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

2 participants