Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ray cluster is not queued with MCAD #725

Open
Sara-KS opened this issue Dec 13, 2022 · 5 comments
Open

Ray cluster is not queued with MCAD #725

Sara-KS opened this issue Dec 13, 2022 · 5 comments

Comments

@Sara-KS
Copy link

Sara-KS commented Dec 13, 2022

I am trying to set up a Ray cluster with 1 head node and 1 worker node using the MCAD option. When I get to the point where I try to launch Ray with the CodeFlare CLI I run into this streaming output

✔  Choice 9  Choose a Pod Scheduler  · Use the Multi-user Enhanced Kubernetes Scheduler
✔  Choice 10  Choose Pod Scheduler for MCAD  · My administrator has already installed and configured MCAD
▶ Stream out Events from the Ray Head Node
No resources found in preprocessing-pipelines namespace.
Waiting for Ray Head node
No resources found in preprocessing-pipelines namespace.
Waiting for Ray Head node 
...

In a separate terminal window, oc get appwrappers returns with no app wrappers listed. This same results happens regardless of how I configure the Ray cluster resources in the Ray Resource Requirements step.

@starpit
Copy link
Collaborator

starpit commented Dec 13, 2022

thanks for the bug report. what does helm ls -n preprocessing-pipelines show? there is a known defect where if a helm chart is leftover from a failed prior startup, it blocks a fresh redeploy.

@Sara-KS
Copy link
Author

Sara-KS commented Dec 13, 2022

Thanks Nick, that was exactly it. When I uninstalled the previous helm chart it was able to make progress

@Sara-KS Sara-KS closed this as completed Dec 13, 2022
@starpit
Copy link
Collaborator

starpit commented Dec 13, 2022

thanks for checking that out! i'll reopen this, since it is a bug we should fix!

@starpit starpit reopened this Dec 13, 2022
@starpit
Copy link
Collaborator

starpit commented Dec 13, 2022

oh, and a bit of fyis from my prior investigations on this: a failed helm install will actually leave around a helm chart, but in a broken state (rather than just leaving around nothing except errors on your console). nice.

then, the guidebook for this only checks to see if the helm chart is installed (not its state). double nice.

@Sara-KS
Copy link
Author

Sara-KS commented Dec 13, 2022

Good to know! I will keep that in mind and post here if I run into a related issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants