Kubernetes pod stuck on ContainerCreating / leader changed

[chbk]

I'm trying to run a Nextflow pipeline on a Kubernetes cluster, but randomly getting these issues.

First issue is that pods created by the pipeline will occasionally get stuck on the ContainerCreating status indefinitely:

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    11m                 default-scheduler  Successfully assigned default/nf-e063a0c262f14fd07067e86006d4d8b0 to gs-7e42puiub44i-node-9
  Warning  FailedMount  6m53s               kubelet            Unable to attach or mount volumes: unmounted volumes=[vol-6], unattached volumes=[default-token-zn89s vol-6]: timed out waiting for the condition
  Warning  FailedMount  3s (x4 over 9m11s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[vol-6], unattached volumes=[vol-6 default-token-zn89s]: timed out waiting for the condition

Second issue gives this error:

Error executing process > 'merge (2)'
Caused by:
  Request GET
/api/v1/namespaces/default/pods/nf-049122184f941fb21d514f8a9025ac1f/status
returned an error code=500
  {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {
      },
      "status": "Failure",
      "message": "etcdserver: leader changed",
      "code": 500
  }
WARN: Killing pending tasks (49)

I don't know if the two are related. It seems they can happen anytime on any pod. When either of these issues appear, relaunching the pipeline with the -resume option works, but they still appear on other pods later on.

Has anyone encountered this before?

[pditommaso]

The first error it's hard to say. It looks the pod cannot mount the volume, you should ask your cluster admin why that's happening.

In the second it looks the pod was interrupted, not sure why, maybe due to a scaling event. Also here your sysadmin could provide more info. On the nextflow side, you can try adding an error retry policy so that the job gets re-executed on such error. Find more at this link https://www.nextflow.io/docs/latest/process.html#maxretries

[chbk]

Thank you @pditommaso I'll try that out.