Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail gracefully #194

Open
Veldhoen opened this issue Dec 20, 2024 · 0 comments
Open

Fail gracefully #194

Veldhoen opened this issue Dec 20, 2024 · 0 comments

Comments

@Veldhoen
Copy link
Member

Veldhoen commented Dec 20, 2024

Currently, the ASR task in Airflow fails every so often when the Whisper service gets evicted or OOMKilled. The triggerer will inquire about the status of a task ID that does not exist in the newly started Whisper service pod.
It would be nice if we can prevent this somehow, so that the same resources doesn't have to be processed for ASR again.
We should think about a good way to go about this. One proposal that we might discuss:

On receiving sigterm, stay alive until the task has finished AND Airflow trigger has been informed (that is: the API has sent a Status.DONE: status.HTTP_200_OK response upon a @api.get("/tasks/{task_id}") request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant