Skip to content

Commit

Permalink
Fix done jobs being aborted
Browse files Browse the repository at this point in the history
Fixes #885. The problem was that after reading the logs, the run status
wasn't done yet at the moment the CLI aborted it.
Added waiting time for run to finish.
  • Loading branch information
r4victor authored and peterschmidt85 committed Feb 12, 2024
1 parent dc0a472 commit a921c1e
Showing 1 changed file with 10 additions and 4 deletions.
14 changes: 10 additions & 4 deletions src/dstack/_internal/cli/commands/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,16 @@ def _command(self, args: argparse.Namespace):
else:
console.print("[error]Failed to attach, exiting...[/]")

run.refresh()
if run.status.is_finished():
_print_fail_message(run)
abort_at_exit = False
# After reading the logs, the run may not be marked as finished immediately.
# Give the run some time to transit into a finished state before aborting it.
for _ in range(5):
run.refresh()
if run.status.is_finished():
if run.status == RunStatus.FAILED:
_print_fail_message(run)
abort_at_exit = False
break
time.sleep(1)
except KeyboardInterrupt:
try:
if not confirm_ask("\nStop the run before detaching?"):
Expand Down

0 comments on commit a921c1e

Please sign in to comment.