Compiler deployments fail when puppet_metrics_dashboard::profile::master::install classified.

[16c7x]

Describe the Bug

If puppet_metrics_dashboard::profile::master::install class is included in the PE Master node group the deployments of any compilers will fail. Using the new puppet infrastructure provision compiler deployment process triggers puppet runs during the orchestration tasks. These puppet runs happen before the pe-puppetserver process has been installed so puppet_metrics_dashboard::profile::master::install fails when it tries to notify that service.

Expected Behavior

Somehow it needs to wait until the compiler deployment process has ben completed before trying to notify the service.

Steps to Reproduce

Steps to reproduce the behavior:

1. Add puppet_metrics_dashboard::profile::master::install class to the PE Master node group.
2. Run puppet infrastructure provision compiler to deploy a compiler.

Environment

PE 2019.8.4

Additional Context

Add any other context about the problem here.

Agent_cert_regen: Downloading certs for compiler4.platform9.puppet.net
Starting: command /opt/puppetlabs/bin/puppet ssl download_cert on compiler4.platform9.puppet.net
Finished: command /opt/puppetlabs/bin/puppet ssl download_cert with 0 failures in 2.41 sec
Starting: task enterprise_tasks::run_puppet on compiler4.platform9.puppet.net
Finished: task enterprise_tasks::run_puppet with 1 failures in 8 min, 35 sec
Errors in task enterprise_tasks::run_puppet:
  compiler4.platform9.puppet.net:
    Message: Running puppet failed on host with certname compiler4.platform9.puppet.net
    Kind: puppetlabs.installpe/run-puppet-failed
    Details:
      "\e[0;32mInfo: Using configured environment 'production'\e[0m\n" +
      "\e[0;32mInfo: Retrieving pluginfacts\e[0m\n" +
      "\e[0;32mInfo: Retrieving plugin\e[0m\n" +
      "\e[0;32mInfo: Retrieving locales\e[0m\n" +
      "\e[0;32mInfo: Loading facts\e[0m\n" +
      "\e[1;31mError: Could not retrieve catalog from remote server: Error 500
      on SERVER: Server Error: Could not find resource 'Service[puppetserver]'
      in parameter 'notify' (file:
      /etc/puppetlabs/code/environments/production/modules/puppet_metrics_dashboard/manifests/profile/master/install.pp,
      line: 18) on node compiler4.platform9.puppet.net\e[0m\n" +
      "\e[1;33mWarning: Not using cache on failed catalog\e[0m\n" +
      "\e[1;31mError: Could not retrieve catalog; skipping run\e[0m\n"
Starting: task enterprise_tasks::enable_agent on compiler4.platform9.puppet.net
Finished: task enterprise_tasks::enable_agent with 0 failures in 4.45 sec
Agent_cert_regen: Restarting puppet service...
Starting: task enterprise_tasks::enable_agent_service on compiler4.platform9.puppet.net
Finished: task enterprise_tasks::enable_agent_service with 0 failures in 4.72 sec
Agent_cert_regen: ERROR: Failed to regenerate agent certificate on node compiler4.platform9.puppet.net
Agent_cert_regen: bolt/run-failure:Plan aborted: run_task 'enterprise_tasks::run_puppet' failed on 1 target
Agent_cert_regen: puppetlabs.installpe/run-puppet-failed Running puppet failed on host with certname compiler4.platform9.puppet.net: Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'Service[puppetserver]' in parameter 'notify' (file: /etc/puppetlabs/code/environments/production/modules/puppet_metrics_dashboard/manifests/profile/master/install.pp, line: 18) on node compiler4.platform9.puppet.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Starting: task enterprise_tasks::enable_agent on peauto12.platform9.puppet.net
Finished: task enterprise_tasks::enable_agent with 0 failures in 4.09 sec
Finished: plan enterprise_tasks::provision_compiler in 11 min, 14 sec
Overall job status: failure...
Error: Error during orchestrated plan run of enterprise_tasks::provision_compiler.
Error: An error has occurred while running orchestrated plan.
	Message: Error regenerating agent certificates on 1 node(s).
	Kind: bolt/plan-failure
	Details:
		class: Bolt::PlanFailure
[root@peauto12 ~]#