Master Gives Up When Scheduling Workers

[kmg-stripe]

Hello @Andyz26 ! We are investigating an issue where a resource cluster doesn't have enough resources to assign workers to task executors. Our ASG-based resource provider will scale-up, but the scheduler gives up before all of the nodes are up. We see log messages for retrying the assignments, but see the scheduler gives up before exhausting all retries.

I see you recently merged this: 7f996f8

We originally thought that max retries was too low and our problem would go away when deploying an artifact with your change, but we still see the scheduler giving up without exhausting retries. Just checking to see if you happen to be investigating something similar...

cc: @codyrioux

[Andyz26]

@kmg-stripe sorry i missed this thread. Did you end up finding anything? Was it related to the behavior change on job at "accepted" state?

[kmg-stripe]

ah yes, sorry for the very, very late response, but we sorted this out.