From 376531d1f29b59c85cef8829034c9f9295ea1633 Mon Sep 17 00:00:00 2001 From: Luis Perez Date: Thu, 27 Jun 2024 07:46:27 -0700 Subject: [PATCH] Allow batches 30s of cleanup time on external kills If a batch is killed by Tron (i.e., hitting max_runtime) or a human (i.e., tronctl kill), give them 30s to handle any cleanup. Batches that wish to use this feature will need to catch the SIGTERM signal and do whatever cleanup they'd like to do in that handler :) --- docs/source/jobs.rst | 7 +++++++ requirements-minimal.txt | 2 +- requirements.txt | 2 +- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/source/jobs.rst b/docs/source/jobs.rst index ae8199f0b..2b6ce4979 100644 --- a/docs/source/jobs.rst +++ b/docs/source/jobs.rst @@ -395,6 +395,13 @@ AM). Cleanup Actions --------------- +There are two possible approaches for cleanup that will depend on when you'd +like the cleanup to occur. You can either: +* Configure your code to cleanup on recieving a SIGTERM (NOTE: we have a + hardcoded limit of 30s before we send a SIGKILL and force-kill the action) + to handle cleanup whenever Tron or a human decides to tronctl kill a run +* OR, use a cleanup action as detailed below to handle cleanup after a job run + has completed. Cleanup actions run after the job succeeds or fails. They are specified just like regular actions except that there is only one per job and it has no name diff --git a/requirements-minimal.txt b/requirements-minimal.txt index f4711d882..11684cc6a 100644 --- a/requirements-minimal.txt +++ b/requirements-minimal.txt @@ -21,7 +21,7 @@ pytimeparse pytz PyYAML>=5.1 requests -task_processing[mesos_executor,k8s]>=1.1.0 +task_processing[mesos_executor,k8s]>=1.2.0 Twisted>=19.7.0 urllib3>=1.24.2 Werkzeug>=0.15.3 diff --git a/requirements.txt b/requirements.txt index 5e0bcfc70..b070dfe99 100644 --- a/requirements.txt +++ b/requirements.txt @@ -80,7 +80,7 @@ setuptools==65.5.1 six==1.15.0 sshpubkeys==3.1.0 stack-data==0.6.2 -task-processing==1.1.0 +task-processing==1.2.0 traitlets==5.0.0 Twisted==22.10.0 typing-extensions==4.5.0