Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timedout tasks do not show correct error messages #1469

Open
shrinandj opened this issue Jun 28, 2023 · 0 comments · May be fixed by #2128
Open

Timedout tasks do not show correct error messages #1469

shrinandj opened this issue Jun 28, 2023 · 0 comments · May be fixed by #2128

Comments

@shrinandj
Copy link
Contributor

Steps that led to this issue:

  • Create a task with a timeout decorator. In this case, the timeout was 60 seconds.
  • Run the task on Kubernetes. This sets the Kubernetes jobs' activeDeadlineSeconds to 60 seconds.
  • If the task did not complete, the error seen by the user is:
[KILLED BY ORCHESTRATOR]
    Kubernetes error:
    Completed. This could be a transient error. Use @retry to retry.

Only after looking at the pod in K8s, it was seen that the pod had timed out. The status of the pod showed:

...
  containerStatuses:
  - containerID: containerd://99e036b3ba08df84f126ea41d05bb0842ea06540de49359d8c7d83e8aecc790b
...
   state:
      terminated:
        containerID: containerd://99e036b3ba08df84f126ea41d05bb0842ea06540de49359d8c7d83e8aecc790b
        exitCode: 137
        reason: Error
...
  message: Pod was active on the node longer than the specified deadline
  phase: Failed
...
  qosClass: Burstable
  reason: DeadlineExceeded

It will be good if the correct error message is brought up onto the user console so that they can see the timeout error.
@madhur-ob madhur-ob linked a pull request Nov 4, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix timeout reason for k8s madhur-ob/metaflow
1 participant
@shrinandj