"Replace a VM in staging" fails on "delete NIC" step

[sandbergja]

Expected behavior

The "Replace a VM in staging" playbook works from tower

Actual behavior

In several cases, it fails with this error:

"pyVmomi.VmomiSupport.vim.fault.GenericVmConfigFault: (vim.fault.GenericVmConfigFault) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         key = 'msg.vigor.hotRemoveStillExists',
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               dynamicType = <unset>,
               dynamicProperty = (vmodl.DynamicProperty) [],
               key = '1',
               value = 'ethernet0'
            }
         ],
         message = 'The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.'
      }
   ],
   reason = 'The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.'
}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File \"<stdin>\", line 107, in <module>
  File \"<stdin>\", line 99, in _ansiballz_main
  File \"<stdin>\", line 47, in invoke_module
  File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_community.vmware.vmware_guest_network_payload_2swyahpq/ansible_community.vmware.vmware_guest_network_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_network.py\", line 829, in <module>
  File \"/tmp/ansible_community.vmware.vmware_guest_network_payload_2swyahpq/ansible_community.vmware.vmware_guest_network_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_network.py\", line 823, in main
  File \"/tmp/ansible_community.vmware.vmware_guest_network_payload_2swyahpq/ansible_community.vmware.vmware_guest_network_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_network.py\", line 610, in _nic_absent
  File \"/tmp/ansible_community.vmware.vmware_guest_network_payload_2swyahpq/ansible_community.vmware.vmware_guest_network_payload.zip/ansible_collections/community/vmware/plugins/module_utils/vmware.py\", line 158, in wait_for_task
  File \"<string>\", line 3, in raise_from
ansible_collections.community.vmware.plugins.module_utils.vmware.TaskError: ('The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.', None)
",

After this happens, when we look in Vsphere, we can see that:

• The new VM did not get an IP address
• The new VM did not get the MAC address from the old vm (which is in the ToBeDeleted folder)

Steps to replicate

1. Go to tower
2. Run the Replace a VM in staging template on a staging vm
3. If it succeeds, run it again on another vm until it fails.

Impact of this bug

Devs can't reliably replace VMs without asking ops to fix it in vsphere

Instances of this failure

• https://ansible-tower.princeton.edu/#/jobs/playbook/9853/output
• https://ansible-tower.princeton.edu/#/jobs/playbook/9706/output
• https://ansible-tower.princeton.edu/#/jobs/playbook/9683/output
• https://ansible-tower.princeton.edu/#/jobs/playbook/9070/output