Er 661 new module email (#747)

* wip: add complete registration reminder email

* wip: update spec

* add start training email notification

* add training emails condition to email recipients

* update recipient selector spec and user scope

* rubocop

* wip: add nudge email for new module

* wip: new module alert email

* add nudge emails for new module

* update recipient selector and spec

* fix syntax error

* remove new module nudge email

* refactor nudge mails

* update dashboard spec update time

* refactor nudge mails

* wip: new module nudge email

* add new module release nudge emails

* add sandbox contentful env

* revert changes to app_config

* check new modules for duplicate names for contentful sandbox

* add email send to test release webhook

* add new modules check to change method in hook controller

* remove new modules check on change method

* add new modules check to change method in hook controller

* remove unnecessary mail jobs from hook controller

* check new modules for repeated names for sandbox

* update spec and remove package-lock.json

* rubocop and revert yarn.lock changes

* revert schema changes

* use cms training modules

* use course progress service for continue module nudge emails

* make modules_in_progress a method on user model

* update interval time on spec

* fix typo

* rubocop

* rubocop

* use user course method for modules progress

* revert dashboard interval change

* use que-locks to prevent duplicate concurrent mailjobs

* add que-locks gem

* revert change to que version

* change que version

* protect against duplicate mail jobs in queue

* rubocop

* add comment to training module record migration

* add yard annotation

* add job helper to guard against duplicate queued jobs

* refactor nudge mail service and spec

* refactor new module check

* wip: create base job to guard against duplicate jobs

* refactor spec

* refactor scheduled que jobs

* update email preferences logic on account page

* refactor nudge mail jobs

* change contentful env to sandbox

* add changes to make testing easier

* rubocop

* fix merge conflicts

* fix merge conflicts

* make completed method public in course progress

* fix bug for user account page

* remove typo from course progress

* change new module mail job from run to enqueue for testing

* in the event of no modulerelease records, populate with training module data

* rubocop

* change contentful env to test

* remove draft new module check for testing

* remove duplicate check for testing

* update timestamps

* trigger new module email even if no released module records for testing

* rubocop

* rubocop

* update spec for testing

* comment out new module mail spec for testing

* comment out new module mail spec for testing

* skip new module spec for testing

* comment out new module mail spec for testing

* add draft check back to new module job

* rubocop

* rubocop

* send email to more users for testing and add sentry output

* send latest module in new mod mailer to all users for testing

* rubocop

* add filtered recipients back

* add filtered recipients back

* select first module for nudge email

* remove create new module release record, for testing

* send mail to all users

* send new module email to all users when there is a release

* rubocop

* trigger email taken mail when release for testing notify

* remove commented test code

* remove application job super to simplify testing

* ruboocop

* log to sentry when job runs

* return recipients to contentful response for debugging

* more sentry logging

* send email directly in release hook

* more sentry logging

* call notify properly

* fix email sending for nudge mail jobs

* update base job and spec

* rubocop

* update mail job specs

* add check for empty ModuleRelease table

* rubocop

* update for qa

* fix merge conflicts

* fix merge conflicts

* scenario 2 qa

* scenario 3 qa

* fix typo

* update continuetraining mail job

* fix typo in notify mailer

* add live scope to modules for email job

* fix continue training mail job

* revert changes made for qa testing

* refactor mail jobs

* use with progress context in mail specs

* add nudge mail recipients to user overview data export

* remove unused matcher

* add mail job shared context

* Rework mail prompt shared context to assert without using log output and
that specific users are messaged

* refactor: use dynamic user scope to select recipients base mail job

* fix user overview data model to reference correct recipients scopes

* remove unused error from mail job

* rubocop

* remove rogue comments

* remove completed todo

* remove missed comments

* add comments to document mail job spec setup

* remove missed comment

* rubocop

---------

Co-authored-by: Peter Hamilton <[email protected]>

.rubocop_todo.yml

@@ -6,6 +6,18 @@
 # Note that changes in the inspected code, or installation of new
 # versions of RuboCop, may require this file to be generated again.
 
+# Offense count: 1
+# This cop supports safe autocorrection (--autocorrect).
+# Configuration parameters: AllowInHeredoc.
+Layout/TrailingWhitespace:
+  Exclude:
+    - 'app/jobs/application_job.rb'
+
+# Offense count: 1
+Lint/MixedRegexpCaptureTypes:
+  Exclude:
+    - 'lib/seed_course_entries.rb'
+
 # Offense count: 1
 Lint/ShadowedException:
   Exclude:
@@ -39,8 +51,8 @@ Rails/CreateTableWithTimestamps:
 # Include: app/**/*.rb, config/**/*.rb, db/**/*.rb, lib/**/*.rb
 Rails/Output:
   Exclude:
+    - 'app/jobs/application_job.rb'
     - 'app/jobs/content_check_job.rb'
-    - 'app/jobs/dashboard_job.rb'
     - 'app/jobs/fill_page_views_job.rb'
     - 'app/models/concerns/to_csv.rb'
     - 'app/services/content_integrity.rb'

app/controllers/application_controller.rb

@@ -38,8 +38,10 @@ def clear_flash
   def prepare_cms
     # ensure correct API for each request
     ContentfulModel.use_preview_api = Rails.application.preview?
+
     # memoise the latest release timestamp
     Training::Module.reset_cache_key!
+
     :done
   end
 

app/controllers/hook_controller.rb

@@ -10,12 +10,13 @@ class HookController < ApplicationController
   #     - Release (execute)
   #
   def release
-    Release.create!(
+    new_release = Release.create!(
       name: payload.dig('sys', 'id'),
       time: payload.dig('sys', 'completedAt'),
       properties: payload,
     )
 
+    NewModuleMailJob.enqueue(new_release.id)
     render json: { status: 'content release received' }, status: :ok
   end
 

app/jobs/application_job.rb

@@ -0,0 +1,44 @@
+class ApplicationJob < Que::Job
+  class DuplicateJobError < StandardError
+  end
+
+  # We are checking for duplicate jobs to prevent undesirable mulitple actions in the event of blocked or slow jobs
+  #
+  # @return [void]
+  def run(*)
+    start_time = Time.zone.now
+    log "#{self.class.name} running"
+
+    if duplicate_job_queued?
+      raise DuplicateJobError, "#{self.class.name} already queued"
+    elsif block_given?
+      yield
+    end
+
+    log "#{self.class.name} finished in #{(Time.zone.now - start_time).round(2)} seconds"
+  end
+
+private
+
+  # @return [Boolean]
+  def duplicate_job_queued?
+    Que.job_stats.any? { |job| job[:job_class] == self.class.name && job[:count] > 1 }
+  end
+
+  # @param error [Error]
+  # @return [void]
+  def handle_error(error)
+    log("#{self.class.name} failed with '#{error.message}'")
+  end
+
+  # @return [String]
+  def log(message)
+    Sentry.capture_message(message, level: :info) if Rails.application.live?
+
+    if ENV['RAILS_LOG_TO_STDOUT'].present?
+      Rails.logger.info(message)
+    elsif ENV['VERBOSE'].present?
+      puts message
+    end
+  end
+end

app/jobs/complete_registration_mail_job.rb

@@ -0,0 +1,7 @@
+class CompleteRegistrationMailJob < MailJob
+  def run
+    super do
+      self.class.recipients.each(&:send_complete_registration_notification)
+    end
+  end
+end

app/jobs/content_check_job.rb

@@ -1,18 +1,12 @@
 # :nocov:
-class ContentCheckJob < Que::Job
+class ContentCheckJob < ApplicationJob
   # @return [Boolean]
   def run(*)
     Training::Module.cache.clear
     log "#{self.class.name}: Running in '#{env}' via '#{api}'"
     valid?
   end
 
-  def handle_error(error)
-    message = "#{self.class.name}: Failed with '#{error.message}'"
-    log(message)
-    Sentry.capture_message(message)
-  end
-
 private
 
   # @return [Boolean] are all modules valid
@@ -23,9 +17,7 @@ def valid?
       log Training::Module.cache.size # should be larger than 0
 
       unless check.valid?
-        message = "ContentCheckJob: #{mod.name} in '#{env}' via '#{api}' is not valid"
-        log(message)
-        Sentry.capture_message(message)
+        log "ContentCheckJob: #{mod.name} in '#{env}' via '#{api}' is not valid"
       end
 
       check.valid?
@@ -41,15 +33,5 @@ def env
   def api
     ContentfulModel.use_preview_api ? 'preview' : 'delivery'
   end
-
-  # @param message [String]
-  # @return [String, nil]
-  def log(message)
-    if ENV['RAILS_LOG_TO_STDOUT'].present?
-      Rails.logger.info(message)
-    elsif ENV['VERBOSE'].present?
-      puts message
-    end
-  end
 end
 # :nocov:

app/jobs/continue_training_mail_job.rb

@@ -0,0 +1,12 @@
+class ContinueTrainingMailJob < MailJob
+  def run
+    super do
+      self.class.recipients.each do |recipient|
+        recipient.modules_in_progress.each do |mod_name|
+          mod = Training::Module.by_name(mod_name)
+          recipient.send_continue_training_notification(mod)
+        end
+      end
+    end
+  end
+end

app/jobs/dashboard_job.rb

@@ -1,5 +1,5 @@
 # :nocov:
-class DashboardJob < Que::Job
+class DashboardJob < ApplicationJob
   # Jobs will default to priority 100 and run immediately
   # a lower number is more important
   #
@@ -9,15 +9,11 @@ class DashboardJob < Que::Job
 
   # @param upload [Boolean] defaults to true in production or if $DASHBOARD_UPDATE exists
   def run(upload: Rails.application.dashboard?)
-    log "#{self.class.name}: Running upload=#{upload}"
+    super do
+      log "DashboardJob: Running upload=#{upload}"
 
-    Dashboard.new(path: build_dir).call(upload: upload, clean: true)
-  end
-
-  def handle_error(error)
-    message = "#{self.class.name}: Failed with '#{error.message}'"
-    log(message)
-    Sentry.capture_message(message) if Rails.application.live?
+      Dashboard.new(path: build_dir).call(upload: upload, clean: true)
+    end
   end
 
 private
@@ -26,15 +22,5 @@ def handle_error(error)
   def build_dir
     Rails.root.join('tmp')
   end
-
-  # @param message [String]
-  # @return [String, nil]
-  def log(message)
-    if ENV['RAILS_LOG_TO_STDOUT'].present?
-      Rails.logger.info(message)
-    elsif ENV['VERBOSE'].present?
-      puts message
-    end
-  end
 end
 # :nocov:

app/jobs/fill_page_views_job.rb

@@ -1,5 +1,5 @@
 # :nocov:
-class FillPageViewsJob < Que::Job
+class FillPageViewsJob < ApplicationJob
   self.queue = 'default'
   self.priority = 1
   # self.run_at = proc { 1.minute.from_now }
@@ -11,23 +11,5 @@ def run(*)
     log "#{self.class.name}: Running"
     FillPageViews.new.call
   end
-
-  def handle_error(error)
-    message = "#{self.class.name}: Failed with '#{error.message}'"
-    log(message)
-    Sentry.capture_message(message) if Rails.application.live?
-  end
-
-private
-
-  # @param message [String]
-  # @return [String, nil]
-  def log(message)
-    if ENV['RAILS_LOG_TO_STDOUT'].present?
-      Rails.logger.info(message)
-    elsif ENV['VERBOSE'].present?
-      puts message
-    end
-  end
 end
 # :nocov:

app/jobs/mail_job.rb

@@ -0,0 +1,13 @@
+class MailJob < ApplicationJob
+  # @return [Array<User>]
+  def self.recipients
+    scope_name = "#{name.underscore}_recipients"
+    User.send(scope_name)
+  end
+
+  def run(*)
+    super
+
+    log("#{self.class.name}: #{self.class.recipients.count} recipients")
+  end
+end

app/jobs/new_module_mail_job.rb

@@ -0,0 +1,42 @@
+class NewModuleMailJob < MailJob
+  # @param release_id [Integer]
+  def run(release_id)
+    super do
+      return unless new_module_published?
+
+      self.class.recipients.each do |recipient|
+        recipient.send_new_module_notification(latest_module)
+      end
+
+      newest_release = Release.find(release_id)
+
+      record_module_release(latest_module, newest_release)
+    end
+  end
+
+private
+
+  # @return [Training::Module]
+  def latest_module
+    Training::Module.live.last
+  end
+
+  # @return [Boolean]
+  def new_module_published?
+    return false unless ModuleRelease.exists?
+
+    ModuleRelease.ordered.last.module_position < latest_module.position
+  end
+
+  # @param mod [Training::Module]
+  # @param release [Release]
+  # @return [ModuleRelease]
+  def record_module_release(mod, release)
+    ModuleRelease.create!(
+      release_id: release.id,
+      module_position: mod.position,
+      name: mod.name,
+      first_published_at: release.time,
+    )
+  end
+end

app/jobs/start_training_mail_job.rb

@@ -0,0 +1,7 @@
+class StartTrainingMailJob < MailJob
+  def run
+    super do
+      self.class.recipients.each(&:send_start_training_notification)
+    end
+  end
+end

app/mailers/notify_mailer.rb

@@ -8,6 +8,10 @@ class NotifyMailer < GovukNotifyRails::Mailer
   PASSWORD_CHANGED_TEMPLATE_ID = 'f77e1eba-3fa8-45ae-9cec-a4cc54633395'.freeze
   RESET_PASSWORD_TEMPLATE_ID = 'ad77aab8-d903-4f77-b074-a16c2658ca79'.freeze
   UNLOCK_TEMPLATE_ID = 'e18e8419-cfcc-4fcb-abdb-84f932f3cf55'.freeze
+  COMPLETE_REGISTRATION_TEMPLATE_ID = 'b960eb6a-d183-484b-ac3b-93ae01b3cee1'.freeze
+  START_TRAINING_TEMPLATE_ID = 'b3c2e4ff-da06-4672-8941-b2f50d37eadc'.freeze
+  CONTINUE_TRAINING_TEMPLATE_ID = '83dd3dc6-c5de-4e32-a6b4-25c76e805d87'.freeze
+  NEW_MODULE_TEMPLATE_ID = '2352b6ce-a098-47f0-870a-286308b9798f'.freeze
 
   include Devise::Controllers::UrlHelpers
 
@@ -106,4 +110,47 @@ def unlock_instructions(record, token, _opts = {})
     )
     mail(to: record.email)
   end
+
+  # @param [User] record
+  def complete_registration(record)
+    set_template(COMPLETE_REGISTRATION_TEMPLATE_ID)
+    set_personalisation(
+      url: root_url,
+    )
+    mail(to: record.email)
+  end
+
+  # @param [User] record
+  def start_training(record)
+    set_template(START_TRAINING_TEMPLATE_ID)
+    set_personalisation(
+      url: root_url,
+    )
+    mail(to: record.email)
+  end
+
+  # @param [User] record
+  # @param [Training::Module] mod
+  def continue_training(record, mod)
+    set_template(CONTINUE_TRAINING_TEMPLATE_ID)
+    set_personalisation(
+      mod_number: mod.position,
+      mod_name: mod.name,
+      url: root_url,
+    )
+    mail(to: record.email)
+  end
+
+  # @param [User] record
+  # @param [Training::Module] mod
+  def new_module(record, mod)
+    set_template(NEW_MODULE_TEMPLATE_ID)
+    set_personalisation(
+      mod_number: mod.position,
+      mod_name: mod.name,
+      mod_criteria: mod.criteria,
+      url: root_url,
+    )
+    mail(to: record.email)
+  end
 end

app/models/ahoy/visit.rb

@@ -6,5 +6,7 @@ class Ahoy::Visit < ApplicationRecord
   has_many :events, class_name: 'Ahoy::Event'
   belongs_to :user, optional: true
 
+  scope :month_old, -> { where(started_at: 4.weeks.ago.beginning_of_day..4.weeks.ago.end_of_day) }
+  scope :last_4_weeks, -> { where(started_at: 4.weeks.ago.end_of_day..Time.zone.now) }
   scope :dashboard, -> { where(started_at: Time.zone.now.beginning_of_month..Time.zone.now) }
 end

app/models/data/user_overview.rb

@@ -26,6 +26,10 @@ def column_names
           'With Notes Percentage',
           'Without Notes',
           'Without Notes Percentage',
+          'Complete Registration Mail Recipients',
+          'Start Training Mail Recipients',
+          'Continue Training Mail Recipients',
+          'New Module Mail Recipients',
         ]
       end
 
@@ -52,6 +56,10 @@ def dashboard
           with_notes_percentage: with_notes_percentage,
           without_notes: without_notes_count,
           without_notes_percentage: 1 - with_notes_percentage,
+          complete_registration_mail_recipients: CompleteRegistrationMailJob.recipients.count,
+          start_training_mail_recipients: StartTrainingMailJob.recipients.count,
+          continue_training_mail_recipients: ContinueTrainingMailJob.recipients.count,
+          new_module_mail_recipients: NewModuleMailJob.recipients.count,
         }]
       end