v0.4.0 - Rewrote reboot behavior (#14)

* new reboot_if_pending type
* fix bug in is_patchday() logic
* tag more resources
* use anchors for dependencies
* use onlyif for conditional execs
* refactor & deduplicate
* strip cmd_opts
* pre-patch reboot only when allowed
* default patch windows to "reboot: ifneeded"
* Update version, readme and changelog

Author: Kevin Reeuwijk

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All notable changes to this project will be documented in this file.
 
+## Release 0.4.0
+
+**Features**
+- Completely rewrote the reboot behavior, so that pending reboot detections fully works both before patching and after patching, in the same Puppet run. There is no more dependency on the `reboots.reboot_required` portion of the `pe_patch`/`os_patching` fact, all logic is now internal and no longer requires multiple Puppet runs.
+- Changed the default schedules to `reboot: ifneeded` (was `reboot: always`), now that the pending reboot logic has improved so much
+- Ensured that pre_reboot commands will now trigger when necessary (only one scenario can happen at a time):
+  - when an OS pending reboot is detected at the start of a run (before patching)
+  - when an OS pending reboot is detected at the end of a run (after patching)
+- Forced pre_reboot commands (which are essentially Exec resources) to use the `posix` provider on Linux and the `powershell` provider on Windows, so that the pending reboot detection logic can be injected to the resource dynamically.
+
 ## Release 0.3.0
 
 **Features**

README.md

@@ -28,8 +28,8 @@ This module will leverage the fact data provided by either the [albatrossflavour
 Once available patches are known via the above facts, the module will install the patches during the configured patch window.
 * For Linux operating systems, this happens through the native Package resource.
 * For Windows operating systems, this happens through the `windows_updates::kb` class, which requires the [noma4i/windows_updates](https://forge.puppet.com/noma4i/windows_updates) module to be installed.
-* By default, a reboot is performed at the end of a patch run that actually installed patches. You can change this behavior though.
-* You can define pre-patch, post-patch and pre-reboot commands for patching runs.
+* By default, a reboot is only performed when necessary at the end of a patch run that actually installed patches. You can change this behavior though, to either always reboot or never reboot.
+* You can define pre-patch, post-patch and pre-reboot commands for patching runs. We recommend that for Windows, you use Powershell-based commands for these. Specifically for pre-reboot commands on Windows, you *must* use Powershell-based commands.
 
 ### Setup Requirements
 
@@ -51,7 +51,7 @@ or
 class {'patching_as_code':}
 ```
 This enables automatic detection of available patches, and puts all the nodes in the `primary` patch group.
-By default this will patch your systems on the 3rd Friday of the month, between 22:00 and midnight (00:00), and perform a reboot.
+By default this will patch your systems on the 3rd Friday of the month, between 22:00 and midnight (00:00), and perform a reboot if necessary.
 On PE 2019.8 or newer this will not automatically classify the `pe_patch` class, so that you can control this through PE's builtin "PE Patch Management" node groups.
 
 To allow patching_as_code to control & declare the `pe_patch` class, change the declaration to:
@@ -60,7 +60,7 @@ class {'patching_as_code':
   classify_pe_patch => true
 }
 ```
-This will change the behavior to also declare the `pe_patch` class, and match its `patch_group` parameter with this module's `patch_group` parameter. In this scenario, make sure you do not classify your nodes with `pe_patch` via the node groups or other means.
+This will change the behavior to also declare the `pe_patch` class, and match its `patch_group` parameter with this module's `patch_group` parameter. In this scenario, make sure you do not classify your nodes with `pe_patch` via the "PE Patch Management" node groups or other means.
 
 ## Usage
 
@@ -71,15 +71,15 @@ patching_as_code::patch_group: early
 ```
 The module provides 5 patch groups out of the box:
 ```
-testing:   patches every 2nd Thursday of the month, between 07:00 and 09:00, performs a reboot
-early:     patches every 3rd Monday   of the month, between 20:00 and 22:00, performs a reboot
-primary:   patches every 3rd Friday   of the month, between 22:00 and 00:00, performs a reboot
-secondary: patches every 3rd Saturday of the month, between 22:00 and 00:00, performs a reboot
-late:      patches every 4th Saturday of the month, between 22:00 and 00:00, performs a reboot
+testing:   patches every 2nd Thursday of the month, between 07:00 and 09:00, performs a reboot if needed
+early:     patches every 3rd Monday   of the month, between 20:00 and 22:00, performs a reboot if needed
+primary:   patches every 3rd Friday   of the month, between 22:00 and 00:00, performs a reboot if needed
+secondary: patches every 3rd Saturday of the month, between 22:00 and 00:00, performs a reboot if needed
+late:      patches every 4th Saturday of the month, between 22:00 and 00:00, performs a reboot if needed
 ```
 There are also 2 special built-in patch groups:
 ```
-always:    patches immediately when a patch is available, can patch in any agent run, performs a reboot
+always:    patches immediately when a patch is available, can patch in any agent run, performs a reboot if needed
 never:     never performs any patching and does not reboot
 ```
 
@@ -177,8 +177,8 @@ You can control additional commands that get executed at specific times, to faci
 2) Run pre-patching commands
 3) Install patches
 4) Run post-patching commands
-5) If reboots are enabled, run pre-reboot commands
-6) If reboots are enabled, reboot system
+5) If reboots are enabled, run pre-reboot commands (if a reboot is pending, or when reboots are set to `always`)
+6) If reboots are enabled, reboot system (if a reboot is pending, or when reboots are set to `always`)
 
 To define the pre/post-patching and pre-reboot commands, you need to create hashes in Hiera. The commands will be executed as `Exec` resources, and you can use any of the [allowed attributes](https://puppet.com/docs/puppet/6.17/types/exec.html#exec-attributes) for that resource (just don't use metaparameters). There are 3 hashes you can define:
 ```
@@ -203,6 +203,8 @@ patching_as_code::pre_patch_commands:
 ```
 As you can see, it's just like defining `Exec` resources.
 
+Note that specifically for `patching_as_code::pre_reboot_commands`, the `provider:`, `onlyif:` and `unless:` parameters will be ignored, as these are overwritten by the internal logic to detect pending reboots. On Linux the `provider:` is forced to `posix`, on Windows it is forced to `powershell`.
+
 ## Limitations
 
 This solution will patching to initiate whenever an agent run occurs inside the patch window. On Windows, patch runs for Cumulative Updates can take a long time, so you may want to tune the hours of your patch windows to account for a patch run getting started near the end of the window and still taking a significant amount of time.

REFERENCE.md

@@ -113,7 +113,8 @@ Options:
 
 * **:command** `String`: The pre-reboot command to execute
 * **:path** `String`: The path for the command
-* **:provider** `String`: The provider for the command
+
+Note: the provider for the command gets forced to `posix` on Linux and `powershell` on Windows
 
 ##### `use_pe_patch`
 

data/common.yaml

@@ -6,31 +6,31 @@ patching_as_code::patch_schedule:
     count_of_week: 2
     hours: 07:00 - 09:00
     max_runs: 4
-    reboot: always
+    reboot: ifneeded
   early:
     day_of_week: Monday
     count_of_week: 3
     hours: 20:00 - 22:00
     max_runs: 4
-    reboot: always
+    reboot: ifneeded
   primary:
     day_of_week: Friday
     count_of_week: 3
     hours: 22:00 - 00:00
     max_runs: 4
-    reboot: always
+    reboot: ifneeded
   secondary:
     day_of_week: Saturday
     count_of_week: 3
     hours: 22:00 - 00:00
     max_runs: 4
-    reboot: always
+    reboot: ifneeded
   late:
     day_of_week: Saturday
     count_of_week: 4
     hours: 22:00 - 00:00
     max_runs: 4
-    reboot: always
+    reboot: ifneeded
 
 patching_as_code::blocklist: []
 patching_as_code::allowlist: []

functions/is_patchday.pp

@@ -19,7 +19,7 @@ function patching_as_code::is_patchday(
   $som_weekday  = Integer($startofmonth.strftime('%u'))
 
   # Calculate first occurence of same weekday
-  if $day_number - $som_weekday <= 0 {
+  if $day_number - $som_weekday < 0 {
     $firstocc = 1 + 7 + $day_number - $som_weekday
   } else {
     $firstocc = 1 + $day_number - $som_weekday

lib/patching_as_code/pending_reboot.ps1

@@ -0,0 +1,24 @@
+function Get-PendingReboot {
+    #Copied from http://ilovepowershell.com/2015/09/10/how-to-check-if-a-server-needs-a-reboot/
+    #Adapted from https://gist.github.com/altrive/5329377
+    #Based on <http://gallery.technet.microsoft.com/scriptcenter/Get-PendingReboot-Query-bdb79542>
+
+    $rebootPending = $false
+
+    if (Get-ChildItem "HKLM:\Software\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending" -EA Ignore) { $rebootPending = $true }
+    if (Get-Item "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired" -EA Ignore) { $rebootPending = $true }
+    if (Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager" -Name PendingFileRenameOperations -EA Ignore) { $rebootPending = $true }
+    try {
+        $util = [wmiclass]"\\.\root\ccm\clientsdk:CCM_ClientUtilities"
+        $status = $util.DetermineIfRebootPending()
+        if (($null -ne $status) -and $status.RebootPending) {
+            $rebootPending = $true
+        }
+    }
+    catch { }
+
+    # return result
+    $rebootPending
+}
+
+Get-PendingReboot

lib/patching_as_code/pending_reboot.sh

@@ -0,0 +1,27 @@
+if [ -f '/usr/bin/needs-restarting' ]
+then
+  case $(facter os.release.major) in
+    7|8)
+      /usr/bin/needs-restarting -r 2>/dev/null 1>/dev/null
+      if [ $? -eq 0 ]
+      then
+        echo "true"
+      fi
+    ;;
+    6)
+      /usr/bin/needs-restarting 2>/dev/null 1>/dev/null
+      if [ $? -gt 0 ]
+      then
+        echo "true"
+      fi
+    ;;
+  esac
+fi
+
+if [ $(facter osfamily) = 'Debian' ] || [ $(facter osfamily) = 'Suse' ]
+then
+  if [ -f '/var/run/reboot-required' ]
+  then
+    echo "true"
+  fi
+fi

lib/puppet/type/patch_package.rb

@@ -12,15 +12,6 @@
     desc 'Puppet schedule to link package resource to'
   end
 
-  newparam(:cache_clean, array_matching: :all) do
-    desc 'Name of Exec resource to clean cache'
-
-    munge do |values|
-      values = [values] unless values.is_a? Array
-      values
-    end
-  end
-
   newparam(:triggers, array_matching: :all) do
     desc 'Resources to notify after updating the package'
 
@@ -32,7 +23,7 @@
 
   # All parameters are required
   validate do
-    [:name, :patch_window, :cache_clean, :triggers].each do |param|
+    [:name, :patch_window, :triggers].each do |param|
       raise Puppet::Error, "Required parameter missing: #{param}" unless @parameters[param]
     end
   end
@@ -41,14 +32,6 @@
   # If package is found, update resource one-time for patching
   # If package is not found, create a one-time package resource
   def pre_run_check
-    # Validate :cache_clean
-    cache_clean = parameter(:cache_clean)
-    cache_clean.value.each do |res|
-      retrieve_resource_reference(res)
-    rescue ArgumentError => e
-      raise Puppet::Error, "Parameter cache_clean failed: #{e} at #{@file}:#{@line}"
-    end
-
     # Validate :triggers
     triggers = parameter(:triggers)
     triggers.value.each do |res|
@@ -71,20 +54,21 @@ def pre_run_check
         Puppet.send('notice', "#{package_res} (managed) will be updated by Patching_as_code")
         catalog.resource(package_res)['ensure'] = 'latest'
         catalog.resource(package_res)['schedule'] = self[:patch_window]
-        catalog.resource(package_res)['require'] = Array(catalog.resource(package_res)['require']) + cache_clean.value
+        catalog.resource(package_res)['before'] = Array(catalog.resource(package_res)['before']) + ['Anchor[patching_as_code::patchday::end]']
+        catalog.resource(package_res)['require'] = Array(catalog.resource(package_res)['require']) + ['Anchor[patching_as_code::patchday::start]']
         catalog.resource(package_res)['notify'] = Array(catalog.resource(package_res)['notify']) + triggers.value
       else
         Puppet.send('notice', "#{package_res} (managed) will not be updated by Patching_as_code, due to the package enforcing a specific version")
       end
     else
       Puppet.send('notice', "#{package_res} (unmanaged) will be updated by Patching_as_code")
-      catalog.add_resource(Puppet::Type.type('package').new(
-                             title: package,
-                             ensure: 'latest',
-                             schedule: self[:patch_window],
-                             require: cache_clean.value,
-                             notify: triggers.value,
-                           ))
+      catalog.create_resource('package',
+                              title: package,
+                              ensure: 'latest',
+                              schedule: self[:patch_window],
+                              before: 'Anchor[patching_as_code::patchday::end]',
+                              require: 'Anchor[patching_as_code::patchday::start]',
+                              notify: triggers.value)
     end
   end
 

lib/puppet/type/reboot_if_pending.rb

@@ -0,0 +1,128 @@
+# rubocop:disable Style/FrozenStringLiteralComment
+# rubocop:enable Style/FrozenStringLiteralComment
+Puppet::Type.newtype(:reboot_if_pending) do
+  @doc = 'Perform a clean reboot if it was pending before this agent run'
+
+  newparam(:name) do
+    isnamevar
+    desc 'Name of this resource (has no function)'
+  end
+
+  newparam(:patch_window) do
+    desc 'Puppet schedule to link the reboot resource to'
+  end
+
+  newparam(:os) do
+    desc 'OS type from kernel fact'
+  end
+
+  # All parameters are required
+  validate do
+    [:name, :patch_window, :os].each do |param|
+      raise Puppet::Error, "Required parameter missing: #{param}" unless @parameters[param]
+    end
+  end
+
+  # Add a reboot resource to the catalog if a pending reboot is detected
+  def pre_run_check
+    # Check for pending reboots
+    pending_reboot = false
+    kernel = parameter(:os).value.downcase
+    case kernel
+    when 'windows'
+      sysroot = ENV['SystemRoot']
+      powershell = "#{sysroot}\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"
+      # get the script path relative to the Puppet Type
+      checker_script = File.join(
+        __dir__,
+        '..',
+        '..',
+        'patching_as_code',
+        'pending_reboot.ps1',
+      )
+      pending_reboot = Puppet::Util::Execution.execute("#{powershell} -ExecutionPolicy Unrestricted -File #{checker_script}").chomp.to_s.downcase == 'true'
+    when 'linux'
+      # get the script path relative to the Puppet Type
+      checker_script = File.join(
+        __dir__,
+        '..',
+        '..',
+        'patching_as_code',
+        'pending_reboot.sh',
+      )
+      pending_reboot = Puppet::Util::Execution.execute("/bin/sh #{checker_script}").chomp.to_s.downcase == 'true'
+    else
+      raise Puppet::Error, "Patching as Code - Unsupported Operating System type: #{kernel}"
+    end
+    return unless pending_reboot
+
+    Puppet.send('notice', 'Patching as Code - Pending OS reboot detected, node will reboot at start of patch window today')
+    ## Reorganize dependencies for pre-patch, post-patch and pre-reboot exec resources:
+    pre_patch_resources = []
+    post_patch_resources = []
+    pre_reboot_resources = []
+    catalog.resources.each do |res|
+      next unless res.type.to_s == 'exec'
+      next unless res['tag'].is_a? Array
+      next unless (res['tag'] & ['patching_as_code_pre_patching', 'patching_as_code_post_patching', 'patching_as_code_pre_reboot']).any?
+
+      if res['tag'].include?('patching_as_code_pre_patching')
+        pre_patch_resources << res
+      elsif res['tag'].include?('patching_as_code_post_patching')
+        post_patch_resources << res
+      elsif res['tag'].include?('patching_as_code_pre_reboot')
+        pre_reboot_resources << res
+      end
+    end
+    ## pre-patch resources should gain Reboot[Patching as Code - Pending OS reboot] for require
+    pre_patch_resources.each do |res|
+      catalog.resource(res.to_s)['require'] = Array(catalog.resource(res.to_s)['require']) << 'Reboot[Patching as Code - Pending OS reboot]'
+    end
+    ## post-patch resources should lose existing before dependencies
+    post_patch_resources.each do |res|
+      catalog.resource(res.to_s)['before'] = []
+    end
+    ## pre-reboot resources should lose existing dependencies
+    pre_reboot_resources.each do |res|
+      catalog.resource(res.to_s)['require'] = []
+      catalog.resource(res.to_s)['before']  = []
+    end
+
+    catalog.add_resource(Puppet::Type.type('reboot').new(
+                           title: 'Patching as Code - Pending OS reboot',
+                           apply: 'immediately',
+                           schedule: parameter(:patch_window).value,
+                           before: "Class[patching_as_code::#{kernel}::patchday]",
+                           require: pre_reboot_resources,
+                         ))
+
+    catalog.add_resource(Puppet::Type.type('notify').new(
+                           title: 'Patching as Code - Performing Pending OS reboot before patching...',
+                           schedule: parameter(:patch_window).value,
+                           notify: 'Reboot[Patching as Code - Pending OS reboot]',
+                           before: "Class[patching_as_code::#{kernel}::patchday]",
+                           require: pre_reboot_resources,
+                         ))
+  end
+
+  def retrieve_resource_reference(res)
+    case res
+    when Puppet::Type      # rubocop:disable Lint/EmptyWhen
+    when Puppet::Resource  # rubocop:disable Lint/EmptyWhen
+    when String
+      begin
+        Puppet::Resource.new(res)
+      rescue ArgumentError
+        raise ArgumentError, "#{res} is not a valid resource reference"
+      end
+    else
+      raise ArgumentError, "#{res} is not a valid resource reference"
+    end
+
+    resource = catalog.resource(res.to_s)
+
+    raise ArgumentError, "#{res} is not in the catalog" unless resource
+
+    resource
+  end
+end