Restraint false reports panic of the previous job

[HouMinXi]

I recently got a false positive report from beaker. After investigating, I found a call trace that occurred when the test machine shutdown and prepared to install a new distro on the current job.

Here is a simple reproducer.

1. keep the power setting is 'Leave on' on beaker web page.
2. Create a script that triggers panic

cat /usr/local/bin/calltrace.sh

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
2. Write the script to systemctl and configure it to trigger after shutdown.

cat /usr/lib/systemd/system/custom_shutdown.service

[Unit]
Description=close services before reboot and shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/calltrace.sh

[Install]
WantedBy=halt.target reboot.target shutdown.target

chmod 777 /usr/local/bin/calltrace.sh

systmectl enable custom_shutdown.service

3. Submit a stand-alone job

<job retention_tag="60days">
  <whiteboard>[20240504~20:17] [[email protected] S:1] /kernel/networking/rt-kernel/debug_restraint: {--machine=dell-per740-83.rhts.eng.pek2.redhat.com --systype=Prototype --param=NAY=yes --param=NIC_DRIVER=ice {--ks-meta=redhat_ca_cert no_networks} --param=SET_DMESG_CHECK_KEY=yes --reserve=}</whiteboard>
  <recipeSet priority="High">
    <recipe whiteboard="[20240504~20:17] [[email protected] S:1] /kernel/networking/rt-kernel/debug_restraint: {--machine=dell-per740-83.rhts.eng.pek2.redhat.com --systype=Prototype --param=NAY=yes --param=NIC_DRIVER=ice {--ks-meta=redhat_ca_cert no_networks} --param=SET_DMESG_CHECK_KEY=yes --reserve=}" role="None" ks_meta="harness='restraint-rhts beakerlib' redhat_ca_cert redhat_ca_cert no_networks disabled_root_access" kernel_options="" kernel_options_post="">
      <autopick random="false"/>
      <watchdog panic="None"/>
      <packages/>
      <ks_appends>
        <ks_append><![CDATA[
%post --log=/root/my-ks-post.log

downhostname=download.devel.redhat.com

#generate buildroot repo
rtype=rel-eng
rtype=nightly
verx=$(rpm -E %rhel)
case $verx in
8|9)
	buildrootUrl=http://$downhostname/rhel-$verx/$rtype/BUILDROOT-$verx/latest-BUILDROOT-$verx-RHEL-$verx/compose/Buildroot/$(arch)/os/
	cat <<-EOF >/etc/yum.repos.d/beaker-buildroot.repo
	[beaker-buildroot]
	name=beaker-buildroot
	baseurl=$buildrootUrl
	enabled=1
	gpgcheck=0
	skip_if_unavailable=1
	EOF
	;;
esac

#update restraint plugins
_rpath=share/restraint/plugins/task_run.d
_path=qa/rhts/lookaside/bkr-client-improved/$_rpath
(cd /usr/$_rpath && curl -k -Ls --retry 64 --retry-delay 2 --remote-name-all http://$downhostname/$_path/{25_environment,27_task_require})
(cd /usr/${_rpath%/*}/completed.d && curl -k -Ls --retry 64 --retry-delay 2 -O http://$downhostname/${_path%/*}/completed.d/85_sync_multihost_tasks)
chmod a+x  /usr/$_rpath/* /usr/${_rpath%/*}/completed.d/*

#add more post actions:

%end


                ]]></ks_append>
      </ks_appends>
      <repos/>
      <distroRequires>
                <and>
                    <distro_method op="=" value="nfs"/>
                    <distro_name op="=" value="RHEL-9.4.0-20240430.42"/>
                    <distro_variant op="=" value=""/>
                    <distro_arch op="=" value="x86_64"/>
                </and>
                <or>
                </or>
                <not>
                    <or>
                    </or>
                </not>
            </distroRequires>
      <hostRequires>
                <and>
                    <hostname op="like" value="dell-per740-83.rhts.eng.pek2.redhat.com"/>
                </and>
                <system_type op="=" value="Prototype"/>
            </hostRequires>
      <partitions>
            </partitions>
      <task name="/distribution/check-install" role="STANDALONE">
        <params>
          <param name="_FETCH_URL" value="no"/>
          <param name="DISTRO_BUILD" value="RHEL-9.4.0-20240430.42"/>
        </params>
      </task>
      <task name="/kernel/networking/rt-kernel/debug_restraint" role="STANDALONE">
        <fetch url="https://gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2#networking/rt-kernel/debug_restraint"/>
        <params>
          <param name="NAY" value="yes"/>
          <param name="NIC_DRIVER" value="ice"/>
          <param name="SET_DMESG_CHECK_KEY" value="yes"/>
          <param name="DISTRO_BUILD" value="RHEL-9.4.0-20240430.42"/>
        </params>
      </task>
      <task name="/distribution/reservesys" role="STANDALONE">
        <params>
          <param name="_FETCH_URL" value="no"/>
          <param name="RESERVETIME" value="356400"/>
        </params>
      </task>
    </recipe>
  </recipeSet>
</job>

4. The beaker web page shows that this job triggered panic.
   https://beaker.engineering.redhat.com/recipes/16070209#task177261937

First, the power setting of the test machine is 'leave on', which also provides the initialization conditions for this test. When the job of step 3 is submitted, the machine triggers a shutdown and startup operation. The call trace of step 2 is triggered when shutting down. It should be noted that the call trace has been recorded in the beaker job submitted this time. https://beaker.engineering.redhat.com/recipes/16070209/logs/console.log

Finally,

[HouMinXi]

I don’t understand why panic is printed before the last task is executed.