Partially overwritten taskout.log after system reinstallation

[jikortus]

Hello,

we have a Beaker task that performs system (RHEL) reinstallation based on various input parameters. Basically what the task does can be split up into following phases:

• Setup phase. Configuration for our reinstallation tool used by this task is collected, which includes restraint configuration (config.conf) content stored in a kickstart file used for system reinstallation.
• Reinstallation phase 1. The reinstallation tool is executed, it uploads the kickstart file, downloads appropriate kernel and initramdisk and configures GRUB with a new installation boot entry. Then the system is rebooted using rhts-reboot.
• Reinstallation phase 2. The installation itself - Anaconda (installer) performs new system installation based on the previous setup (mainly kickstart). Installation logs are collected and uploaded to Beaker throughout the installation using custom python scripts invoked in Anaconda's pre-installation phase in kickstart. When the main part of installation is done, system repositories including Beaker/restraint ones are configured, restraint/beakerlib/... packages are installed, offsets in restraint configuration updated based on lengths of logs already uploaded to Beaker controller, restraint configuration file content is restored, and also a flag file indicating that a new installation completed is created on the new filesystem. The last step is a reboot into the newly installed system.
• Reinstallation phase 3. The existence of the reinstallation flag file is checked, indicating that the reinstallation was successful.
• Teardown. The flag file is removed.

There's one issue we have with the main task log file (taskout.log) though: Its content is stored as expected up to the end of phase 1. Then new test/log sections from phase 2 appear and the individual Anaconda logs are uploaded fine, nevertheless the taskout.log gets partially overwritten in phase 3 instead of having the log content from phase 3 appended at the end. The general logs (resultoutputfile.log) for each phase are present as expected.

It seems that this misbehaviour can be related to improper logs offset handling by restraint. I spotted several issues during experiments with the task: Sometimes the [offsets_*] section is missing in restraint configuration file, but in this particular task it appears after some time. I also noticed that sometimes the taskout.log content from phase 1 (before reboot) can have some bits at the end - it looks like the log content is not uploaded to Beaker completely.

I tried to take several measures to fix or workaround this issue, only with partial success: I modified the task to ensure the [offsets_*] section exists in restraint configuration file before reboot and I added a delay before reboot to ensure the pre-reinstallation logs are uploaded completely. Also I changed the post-installation kickstart script to restore all restraint data (/var/lib/restraint, /var/tmp/restraintd) instead of just the configuration file. I'm not sure if all of those changes were necessary, but this way at least the taskout.log content from pre-reinstallation phase was preserved, however the rest (phase 3 and cleanup) was missing. EDIT: Those changes didn't help either, the beginning of the log is still overwritten with post-reinstallation logs.

Unfortunately I couldn't find some minimal reproducer to post here, but if there are any suggestions on what to try or what data to gather, I can experiment with it further. I'm not able to provide the Beaker task and the reinstallation tool's code here publicly as it is part of our internal testing suite.