Race Condition: Error is thrown if multiple processes try to roll over a log file

[christophertubbs]

Encountered while testing the evaluation service:

Traceback (most recent call last):
  File "/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/handlers.py", line 70, in emit
    self.doRollover()
  File "/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/handlers.py", line 394, in doRollover
    self.rotate(self.baseFilename, dfn)
  File "/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/handlers.py", line 111, in rotate
    os.rename(source, dest)
FileNotFoundError: [Errno 2] No such file or directory: 'DMOD/python/services/evaluationservice/dmod/evaluationservice/evaluations.log' -> 'DMOD/python/services/evaluationservice/dmod/evaluationservice/evaluations.log.2024-08-13'

Two processes (runners for the evaluation service) tried to roll over the rotating handler at the same time. Python doesn't perform interprocessing communication when logging, so it does not consider this a race condition or something python itself should fix.

Something needs to be put into place to prevent this in the future.

This is default handling. The environment variable LOGGING_CONFIGURATION may be set to change how logging operates in the case that a remote handler may be needed or file writing needs to be removed in a particular environment.

This is low priority since it may be hard to hit.