Cause of the delay in reloading crontab?

[palkarbedki]

Hi

We have a script that tries to scan the etc/crond directory, populate crontab file and compares it with the crontab file that supercronic uses to schedule, when there are changes tries reload the crontab.

When we try to reload the cron tab by using the below command, we notice a delay between the time the command is issued to reload the crontab and the read crontab .

kill -USR2 $pid

Please help to understand the cause for the delay? Do we have any options to avoid this delay? As this seem to cause the minutely jobs to skip an iteration due to the delay during reload time.

Mar 13 09:39:00 supercronic: time="2024-03-13T09:39:00Z" level=info msg=starting iteration=0 job.command="populate_and_load_crontab_from_etc_crond.sh" job.position=0 job.schedule="* * * * "
Mar 13 09:39:50 supercronic: time="2024-03-13T09:39:50Z" level=info msg="received user defined signal 2, reloading crontab"
Mar 13 09:39:50 supercronic: time="2024-03-13T09:39:50Z" level=info msg="job succeeded" iteration=0 job.command="populate_and_load_crontab_from_etc_crond.sh" job.position=0 job.schedule=" * * * *"
Mar 13 09:40:27 supercronic: time="2024-03-13T09:40:27Z" level=info msg="read crontab: "

Thanks

[UserNotFound]

Am I understanding it correctly that supercronic itself is running a job that sends the signal to reload the configuration?

If so, does this seem like a reasonable approximation?

FROM debian

RUN apt-get update \
  && apt-get install -y curl\
  && rm -rf /var/lib/apt/lists/*

# Latest releases available at https://github.com/aptible/supercronic/releases
ENV SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/v0.2.29/supercronic-linux-amd64 \
    SUPERCRONIC=supercronic-linux-amd64 \
    SUPERCRONIC_SHA1SUM=cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b

RUN curl -fsSLO "$SUPERCRONIC_URL" \
 && echo "${SUPERCRONIC_SHA1SUM}  ${SUPERCRONIC}" | sha1sum -c - \
 && chmod +x "$SUPERCRONIC" \
 && mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" \
 && ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic

RUN echo "kill -USR2 1" > doit.sh
RUN echo "* * * * * sh doit.sh" > crontab

CMD ["supercronic", "crontab"]

Which, if this test is accurate, shows supercronic has no issue reloading itself in less than a second:

$ docker build . --tag test1 && docker run -it test1
INFO[2024-03-22T03:39:00Z] starting iteration=0 job.command="sh doit.sh" job.position=0 job.schedule="* * * * "
INFO[2024-03-22T03:39:00Z] received user defined signal 2, reloading crontab
INFO[2024-03-22T03:39:00Z] waiting for jobs to finish
INFO[2024-03-22T03:39:00Z] job succeeded iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * *"
INFO[2024-03-22T03:39:00Z] read crontab: crontab

INFO[2024-03-22T03:36:37Z] read crontab: crontab
INFO[2024-03-22T03:37:00Z] starting iteration=0 job.command="sh doit.sh" job.position=0 job.schedule="* * * * "
INFO[2024-03-22T03:37:00Z] received user defined signal 2, reloading crontab
INFO[2024-03-22T03:37:00Z] waiting for jobs to finish
INFO[2024-03-22T03:37:00Z] job succeeded iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * "
INFO[2024-03-22T03:37:00Z] read crontab: crontab
INFO[2024-03-22T03:38:00Z] starting iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * "
INFO[2024-03-22T03:38:00Z] received user defined signal 2, reloading crontab
INFO[2024-03-22T03:38:00Z] waiting for jobs to finish
INFO[2024-03-22T03:38:00Z] job succeeded iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * "
INFO[2024-03-22T03:38:00Z] read crontab: crontab
INFO[2024-03-22T03:39:00Z] starting iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * "
INFO[2024-03-22T03:39:00Z] received user defined signal 2, reloading crontab
INFO[2024-03-22T03:39:00Z] waiting for jobs to finish
INFO[2024-03-22T03:39:00Z] job succeeded iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * *"
INFO[2024-03-22T03:39:00Z] read crontab: crontab

I do notice that the logs you shared are missing the "waiting for jobs to finish" message (maybe you omitted it for brevity?), and in my testing with two jobs I am able to reproduce an explainable delay: the exit and reload signals do both wait for currently running jobs to finish:

* * * * * sh doit.sh
*/2 * * * * sleep 90

Yields the result

INFO[2024-03-22T03:43:38Z] read crontab: crontab
INFO[2024-03-22T03:44:00Z] starting iteration=0 job.command="sh doit.sh" job.position=0 job.schedule="* * * * "
INFO[2024-03-22T03:44:00Z] starting iteration=0 job.command="sleep 90" job.position=1 job.schedule="/2 * * * "
INFO[2024-03-22T03:44:00Z] job succeeded iteration=0 job.command="sh doit.sh" job.position=0 job.schedule=" * * * "
INFO[2024-03-22T03:44:00Z] received user defined signal 2, reloading crontab
INFO[2024-03-22T03:44:00Z] waiting for jobs to finish
INFO[2024-03-22T03:45:30Z] job succeeded iteration=0 job.command="sleep 90" job.position=1 job.schedule="/2 * * * *"
INFO[2024-03-22T03:45:30Z] read crontab: crontab

Does this seem to be your issue?

[palkarbedki]

Sorry, I missed the 'waiting for jobs to finish' log in my earlier post. Yes, I see the reload waits for the jobs to finish causing the jobs to miss an iteration . Do you have any suggestions to avoid this?

We have multiple jobs that runs every minute and say one of the job gets removed from the schedule, then from the next iteration on, we do not want this job to be scheduled at the same time the other jobs shouldn't get delayed /miss an iteration during the reload of the crontab.