Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MLFlow ResponseError('too many 500 error responses') if try to log to deleted experiment #20555

Open
millskyle opened this issue Jan 21, 2025 · 0 comments · May be fixed by #20556
Open

MLFlow ResponseError('too many 500 error responses') if try to log to deleted experiment #20555

millskyle opened this issue Jan 21, 2025 · 0 comments · May be fixed by #20556
Labels
bug Something isn't working needs triage Waiting to be triaged by maintainers ver: 2.5.x

Comments

@millskyle
Copy link

Bug description

If one tries to log to a deleted MLFlow experiment, (e.g. you delete the experiment in the MLFlow UI, go on holidays, come back renewed and refreshed and are greeted by your previously working code no longer working), Lightning throws a ResponseError('too many 500 error responses').

I will create a PR with a proposed simple fix.

What version are you seeing the problem on?

v2.5

How to reproduce the bug

Error messages and logs

mlflow.exceptions.MlflowException: API request to http://localhost:5000/api/2.0/mlflow/runs/create failed with exception HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/runs/create (Caused by ResponseError('too many 500 error responses'))

Environment

Current environment
#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):

More info

No response

@millskyle millskyle added bug Something isn't working needs triage Waiting to be triaged by maintainers labels Jan 21, 2025
@millskyle millskyle linked a pull request Jan 21, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Waiting to be triaged by maintainers ver: 2.5.x
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant