Skip to content

Commit

Permalink
v0.4.0 - Rewrote reboot behavior (#14)
Browse files Browse the repository at this point in the history
* new reboot_if_pending type
* fix bug in is_patchday() logic
* tag more resources
* use anchors for dependencies
* use onlyif for conditional execs
* refactor & deduplicate
* strip cmd_opts
* pre-patch reboot only when allowed
* default patch windows to "reboot: ifneeded"
* Update version, readme and changelog
  • Loading branch information
Kevin Reeuwijk authored Feb 4, 2021
1 parent 01979a3 commit 93602b9
Show file tree
Hide file tree
Showing 13 changed files with 344 additions and 107 deletions.
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

All notable changes to this project will be documented in this file.

## Release 0.4.0

**Features**
- Completely rewrote the reboot behavior, so that pending reboot detections fully works both before patching and after patching, in the same Puppet run. There is no more dependency on the `reboots.reboot_required` portion of the `pe_patch`/`os_patching` fact, all logic is now internal and no longer requires multiple Puppet runs.
- Changed the default schedules to `reboot: ifneeded` (was `reboot: always`), now that the pending reboot logic has improved so much
- Ensured that pre_reboot commands will now trigger when necessary (only one scenario can happen at a time):
- when an OS pending reboot is detected at the start of a run (before patching)
- when an OS pending reboot is detected at the end of a run (after patching)
- Forced pre_reboot commands (which are essentially Exec resources) to use the `posix` provider on Linux and the `powershell` provider on Windows, so that the pending reboot detection logic can be injected to the resource dynamically.

## Release 0.3.0

**Features**
Expand Down
26 changes: 14 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ This module will leverage the fact data provided by either the [albatrossflavour
Once available patches are known via the above facts, the module will install the patches during the configured patch window.
* For Linux operating systems, this happens through the native Package resource.
* For Windows operating systems, this happens through the `windows_updates::kb` class, which requires the [noma4i/windows_updates](https://forge.puppet.com/noma4i/windows_updates) module to be installed.
* By default, a reboot is performed at the end of a patch run that actually installed patches. You can change this behavior though.
* You can define pre-patch, post-patch and pre-reboot commands for patching runs.
* By default, a reboot is only performed when necessary at the end of a patch run that actually installed patches. You can change this behavior though, to either always reboot or never reboot.
* You can define pre-patch, post-patch and pre-reboot commands for patching runs. We recommend that for Windows, you use Powershell-based commands for these. Specifically for pre-reboot commands on Windows, you *must* use Powershell-based commands.

### Setup Requirements

Expand All @@ -51,7 +51,7 @@ or
class {'patching_as_code':}
```
This enables automatic detection of available patches, and puts all the nodes in the `primary` patch group.
By default this will patch your systems on the 3rd Friday of the month, between 22:00 and midnight (00:00), and perform a reboot.
By default this will patch your systems on the 3rd Friday of the month, between 22:00 and midnight (00:00), and perform a reboot if necessary.
On PE 2019.8 or newer this will not automatically classify the `pe_patch` class, so that you can control this through PE's builtin "PE Patch Management" node groups.

To allow patching_as_code to control & declare the `pe_patch` class, change the declaration to:
Expand All @@ -60,7 +60,7 @@ class {'patching_as_code':
classify_pe_patch => true
}
```
This will change the behavior to also declare the `pe_patch` class, and match its `patch_group` parameter with this module's `patch_group` parameter. In this scenario, make sure you do not classify your nodes with `pe_patch` via the node groups or other means.
This will change the behavior to also declare the `pe_patch` class, and match its `patch_group` parameter with this module's `patch_group` parameter. In this scenario, make sure you do not classify your nodes with `pe_patch` via the "PE Patch Management" node groups or other means.

## Usage

Expand All @@ -71,15 +71,15 @@ patching_as_code::patch_group: early
```
The module provides 5 patch groups out of the box:
```
testing: patches every 2nd Thursday of the month, between 07:00 and 09:00, performs a reboot
early: patches every 3rd Monday of the month, between 20:00 and 22:00, performs a reboot
primary: patches every 3rd Friday of the month, between 22:00 and 00:00, performs a reboot
secondary: patches every 3rd Saturday of the month, between 22:00 and 00:00, performs a reboot
late: patches every 4th Saturday of the month, between 22:00 and 00:00, performs a reboot
testing: patches every 2nd Thursday of the month, between 07:00 and 09:00, performs a reboot if needed
early: patches every 3rd Monday of the month, between 20:00 and 22:00, performs a reboot if needed
primary: patches every 3rd Friday of the month, between 22:00 and 00:00, performs a reboot if needed
secondary: patches every 3rd Saturday of the month, between 22:00 and 00:00, performs a reboot if needed
late: patches every 4th Saturday of the month, between 22:00 and 00:00, performs a reboot if needed
```
There are also 2 special built-in patch groups:
```
always: patches immediately when a patch is available, can patch in any agent run, performs a reboot
always: patches immediately when a patch is available, can patch in any agent run, performs a reboot if needed
never: never performs any patching and does not reboot
```

Expand Down Expand Up @@ -177,8 +177,8 @@ You can control additional commands that get executed at specific times, to faci
2) Run pre-patching commands
3) Install patches
4) Run post-patching commands
5) If reboots are enabled, run pre-reboot commands
6) If reboots are enabled, reboot system
5) If reboots are enabled, run pre-reboot commands (if a reboot is pending, or when reboots are set to `always`)
6) If reboots are enabled, reboot system (if a reboot is pending, or when reboots are set to `always`)

To define the pre/post-patching and pre-reboot commands, you need to create hashes in Hiera. The commands will be executed as `Exec` resources, and you can use any of the [allowed attributes](https://puppet.com/docs/puppet/6.17/types/exec.html#exec-attributes) for that resource (just don't use metaparameters). There are 3 hashes you can define:
```
Expand All @@ -203,6 +203,8 @@ patching_as_code::pre_patch_commands:
```
As you can see, it's just like defining `Exec` resources.

Note that specifically for `patching_as_code::pre_reboot_commands`, the `provider:`, `onlyif:` and `unless:` parameters will be ignored, as these are overwritten by the internal logic to detect pending reboots. On Linux the `provider:` is forced to `posix`, on Windows it is forced to `powershell`.

## Limitations

This solution will patching to initiate whenever an agent run occurs inside the patch window. On Windows, patch runs for Cumulative Updates can take a long time, so you may want to tune the hours of your patch windows to account for a patch run getting started near the end of the window and still taking a significant amount of time.
3 changes: 2 additions & 1 deletion REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,8 @@ Options:

* **:command** `String`: The pre-reboot command to execute
* **:path** `String`: The path for the command
* **:provider** `String`: The provider for the command

Note: the provider for the command gets forced to `posix` on Linux and `powershell` on Windows

##### `use_pe_patch`

Expand Down
10 changes: 5 additions & 5 deletions data/common.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,31 +6,31 @@ patching_as_code::patch_schedule:
count_of_week: 2
hours: 07:00 - 09:00
max_runs: 4
reboot: always
reboot: ifneeded
early:
day_of_week: Monday
count_of_week: 3
hours: 20:00 - 22:00
max_runs: 4
reboot: always
reboot: ifneeded
primary:
day_of_week: Friday
count_of_week: 3
hours: 22:00 - 00:00
max_runs: 4
reboot: always
reboot: ifneeded
secondary:
day_of_week: Saturday
count_of_week: 3
hours: 22:00 - 00:00
max_runs: 4
reboot: always
reboot: ifneeded
late:
day_of_week: Saturday
count_of_week: 4
hours: 22:00 - 00:00
max_runs: 4
reboot: always
reboot: ifneeded

patching_as_code::blocklist: []
patching_as_code::allowlist: []
Expand Down
2 changes: 1 addition & 1 deletion functions/is_patchday.pp
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ function patching_as_code::is_patchday(
$som_weekday = Integer($startofmonth.strftime('%u'))

# Calculate first occurence of same weekday
if $day_number - $som_weekday <= 0 {
if $day_number - $som_weekday < 0 {
$firstocc = 1 + 7 + $day_number - $som_weekday
} else {
$firstocc = 1 + $day_number - $som_weekday
Expand Down
24 changes: 24 additions & 0 deletions lib/patching_as_code/pending_reboot.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
function Get-PendingReboot {
#Copied from http://ilovepowershell.com/2015/09/10/how-to-check-if-a-server-needs-a-reboot/
#Adapted from https://gist.github.com/altrive/5329377
#Based on <http://gallery.technet.microsoft.com/scriptcenter/Get-PendingReboot-Query-bdb79542>

$rebootPending = $false

if (Get-ChildItem "HKLM:\Software\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending" -EA Ignore) { $rebootPending = $true }
if (Get-Item "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired" -EA Ignore) { $rebootPending = $true }
if (Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager" -Name PendingFileRenameOperations -EA Ignore) { $rebootPending = $true }
try {
$util = [wmiclass]"\\.\root\ccm\clientsdk:CCM_ClientUtilities"
$status = $util.DetermineIfRebootPending()
if (($null -ne $status) -and $status.RebootPending) {
$rebootPending = $true
}
}
catch { }

# return result
$rebootPending
}

Get-PendingReboot
27 changes: 27 additions & 0 deletions lib/patching_as_code/pending_reboot.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
if [ -f '/usr/bin/needs-restarting' ]
then
case $(facter os.release.major) in
7|8)
/usr/bin/needs-restarting -r 2>/dev/null 1>/dev/null
if [ $? -eq 0 ]
then
echo "true"
fi
;;
6)
/usr/bin/needs-restarting 2>/dev/null 1>/dev/null
if [ $? -gt 0 ]
then
echo "true"
fi
;;
esac
fi

if [ $(facter osfamily) = 'Debian' ] || [ $(facter osfamily) = 'Suse' ]
then
if [ -f '/var/run/reboot-required' ]
then
echo "true"
fi
fi
36 changes: 10 additions & 26 deletions lib/puppet/type/patch_package.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,6 @@
desc 'Puppet schedule to link package resource to'
end

newparam(:cache_clean, array_matching: :all) do
desc 'Name of Exec resource to clean cache'

munge do |values|
values = [values] unless values.is_a? Array
values
end
end

newparam(:triggers, array_matching: :all) do
desc 'Resources to notify after updating the package'

Expand All @@ -32,7 +23,7 @@

# All parameters are required
validate do
[:name, :patch_window, :cache_clean, :triggers].each do |param|
[:name, :patch_window, :triggers].each do |param|
raise Puppet::Error, "Required parameter missing: #{param}" unless @parameters[param]
end
end
Expand All @@ -41,14 +32,6 @@
# If package is found, update resource one-time for patching
# If package is not found, create a one-time package resource
def pre_run_check
# Validate :cache_clean
cache_clean = parameter(:cache_clean)
cache_clean.value.each do |res|
retrieve_resource_reference(res)
rescue ArgumentError => e
raise Puppet::Error, "Parameter cache_clean failed: #{e} at #{@file}:#{@line}"
end

# Validate :triggers
triggers = parameter(:triggers)
triggers.value.each do |res|
Expand All @@ -71,20 +54,21 @@ def pre_run_check
Puppet.send('notice', "#{package_res} (managed) will be updated by Patching_as_code")
catalog.resource(package_res)['ensure'] = 'latest'
catalog.resource(package_res)['schedule'] = self[:patch_window]
catalog.resource(package_res)['require'] = Array(catalog.resource(package_res)['require']) + cache_clean.value
catalog.resource(package_res)['before'] = Array(catalog.resource(package_res)['before']) + ['Anchor[patching_as_code::patchday::end]']
catalog.resource(package_res)['require'] = Array(catalog.resource(package_res)['require']) + ['Anchor[patching_as_code::patchday::start]']
catalog.resource(package_res)['notify'] = Array(catalog.resource(package_res)['notify']) + triggers.value
else
Puppet.send('notice', "#{package_res} (managed) will not be updated by Patching_as_code, due to the package enforcing a specific version")
end
else
Puppet.send('notice', "#{package_res} (unmanaged) will be updated by Patching_as_code")
catalog.add_resource(Puppet::Type.type('package').new(
title: package,
ensure: 'latest',
schedule: self[:patch_window],
require: cache_clean.value,
notify: triggers.value,
))
catalog.create_resource('package',
title: package,
ensure: 'latest',
schedule: self[:patch_window],
before: 'Anchor[patching_as_code::patchday::end]',
require: 'Anchor[patching_as_code::patchday::start]',
notify: triggers.value)
end
end

Expand Down
128 changes: 128 additions & 0 deletions lib/puppet/type/reboot_if_pending.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# rubocop:disable Style/FrozenStringLiteralComment
# rubocop:enable Style/FrozenStringLiteralComment
Puppet::Type.newtype(:reboot_if_pending) do
@doc = 'Perform a clean reboot if it was pending before this agent run'

newparam(:name) do
isnamevar
desc 'Name of this resource (has no function)'
end

newparam(:patch_window) do
desc 'Puppet schedule to link the reboot resource to'
end

newparam(:os) do
desc 'OS type from kernel fact'
end

# All parameters are required
validate do
[:name, :patch_window, :os].each do |param|
raise Puppet::Error, "Required parameter missing: #{param}" unless @parameters[param]
end
end

# Add a reboot resource to the catalog if a pending reboot is detected
def pre_run_check
# Check for pending reboots
pending_reboot = false
kernel = parameter(:os).value.downcase
case kernel
when 'windows'
sysroot = ENV['SystemRoot']
powershell = "#{sysroot}\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"
# get the script path relative to the Puppet Type
checker_script = File.join(
__dir__,
'..',
'..',
'patching_as_code',
'pending_reboot.ps1',
)
pending_reboot = Puppet::Util::Execution.execute("#{powershell} -ExecutionPolicy Unrestricted -File #{checker_script}").chomp.to_s.downcase == 'true'
when 'linux'
# get the script path relative to the Puppet Type
checker_script = File.join(
__dir__,
'..',
'..',
'patching_as_code',
'pending_reboot.sh',
)
pending_reboot = Puppet::Util::Execution.execute("/bin/sh #{checker_script}").chomp.to_s.downcase == 'true'
else
raise Puppet::Error, "Patching as Code - Unsupported Operating System type: #{kernel}"
end
return unless pending_reboot

Puppet.send('notice', 'Patching as Code - Pending OS reboot detected, node will reboot at start of patch window today')
## Reorganize dependencies for pre-patch, post-patch and pre-reboot exec resources:
pre_patch_resources = []
post_patch_resources = []
pre_reboot_resources = []
catalog.resources.each do |res|
next unless res.type.to_s == 'exec'
next unless res['tag'].is_a? Array
next unless (res['tag'] & ['patching_as_code_pre_patching', 'patching_as_code_post_patching', 'patching_as_code_pre_reboot']).any?

if res['tag'].include?('patching_as_code_pre_patching')
pre_patch_resources << res
elsif res['tag'].include?('patching_as_code_post_patching')
post_patch_resources << res
elsif res['tag'].include?('patching_as_code_pre_reboot')
pre_reboot_resources << res
end
end
## pre-patch resources should gain Reboot[Patching as Code - Pending OS reboot] for require
pre_patch_resources.each do |res|
catalog.resource(res.to_s)['require'] = Array(catalog.resource(res.to_s)['require']) << 'Reboot[Patching as Code - Pending OS reboot]'
end
## post-patch resources should lose existing before dependencies
post_patch_resources.each do |res|
catalog.resource(res.to_s)['before'] = []
end
## pre-reboot resources should lose existing dependencies
pre_reboot_resources.each do |res|
catalog.resource(res.to_s)['require'] = []
catalog.resource(res.to_s)['before'] = []
end

catalog.add_resource(Puppet::Type.type('reboot').new(
title: 'Patching as Code - Pending OS reboot',
apply: 'immediately',
schedule: parameter(:patch_window).value,
before: "Class[patching_as_code::#{kernel}::patchday]",
require: pre_reboot_resources,
))

catalog.add_resource(Puppet::Type.type('notify').new(
title: 'Patching as Code - Performing Pending OS reboot before patching...',
schedule: parameter(:patch_window).value,
notify: 'Reboot[Patching as Code - Pending OS reboot]',
before: "Class[patching_as_code::#{kernel}::patchday]",
require: pre_reboot_resources,
))
end

def retrieve_resource_reference(res)
case res
when Puppet::Type # rubocop:disable Lint/EmptyWhen
when Puppet::Resource # rubocop:disable Lint/EmptyWhen
when String
begin
Puppet::Resource.new(res)
rescue ArgumentError
raise ArgumentError, "#{res} is not a valid resource reference"
end
else
raise ArgumentError, "#{res} is not a valid resource reference"
end

resource = catalog.resource(res.to_s)

raise ArgumentError, "#{res} is not in the catalog" unless resource

resource
end
end
Loading

0 comments on commit 93602b9

Please sign in to comment.