Hooks do not run if mail delivery fails

[svalo]

Hello,
I'm using this systemd unit for certspotter:

# /etc/systemd/system/certspotter.service
[Unit]
Description=Certificate Transparency Log Monitor
Documentation=man:certspotter(8)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=certspotter
DynamicUser=true
LockPersonality=true
Environment=CERTSPOTTER_CONFIG_DIR=/etc/certspotter CERTSPOTTER_STATE_DIR=/var/cache/certspotter [email protected]
ExecStart=/usr/bin/certspotter -verbose -start_at_end -stdout
ConfigurationDirectory=certspotter
CacheDirectory=certspotter
# not strict, because we want to allow some flexibility to hooks
ProtectSystem=full

[Install]
WantedBy=multi-user.target

I have a local mailserver which correctly deliver mails.

I defined a hook which correctly works

Whats happens

If certspotter encounters errors while delivering the mail such as the following

Nov 06 12:22:25 example certspotter[672]: 2024/11/06 12:22:25 finished downloading entries from https://ct.googleapis.com/logs/us1/argon2025h2/
Nov 06 12:22:25 example certspotter[672]: 2024/11/06 12:22:25 saving state in defer for https://ct.googleapis.com/logs/us1/argon2025h2/
Nov 06 12:22:25 example postfix/postdrop[994]: warning: mail_queue_enter: create file maildrop/514281.994: Permission denied
Nov 06 12:22:26 example postfix/postdrop[996]: warning: mail_queue_enter: create file maildrop/24434.996: Permission denied

then hooks do not run.

If I solve the mail delivery issue, the hook fires correctly.

What I expected

Mail delivery fails, other hooks are fired

System details:

$ lsb_release -a
LSB Version:    n/a
Distributor ID: Arch
Description:    Arch Linux
Release:        rolling
Codename:       n/a

$ certspotter --version
certspotter version v0.18.0?

systemd version: 256.7-1

Side note

The issue delivering the email is caused probably by systemd, if I add the directive ReadWritePaths=/var/spool/postfix/maildrop and add the user to the maildrop group mail delivery works . I have yet to investigate if only the change to systemd is enough to let mail flow. Just adding the user to the group is not enough as maildrop fails with an error regarding the filesystem being readonly.

[svalo]

I think the line

certspotter/monitor/notify.go

Lines 40 to 41 in 0ba0a1f

	if err := sendEmail(ctx, s.Email, notif); err != nil {
	return err

is the responsible of the behavior, would you be in favor of emitting an error but continuing processing hook and hookdir?

The same behavior is used also by execScript and execScriptDir, woul you be in favor of changing it also there?

[AGWA]

When a notification fails, certspotter is supposed to terminate and write an error to stderr (which should be captured by the systemd journal). Are you saying that certspotter continues to run when sending an email fails?

It might be reasonable to try all of the notification methods even if one fails. However, if any of them fail, I do believe certspotter needs to terminate. This allows the operator to correct the problem and restart certspotter, which will cause the failed notifications to be delivered. Continuing on failure means those notifications are essentially lost forever.

[svalo]

When a notification fails, certspotter is supposed to terminate and write an error to stderr (which should be captured by the systemd journal).

And indeed it does, I noticed about the failure in the journal. Certspotter continued to try delivering the email.

Are you saying that certspotter continues to run when sending an email fails?

Yes, I'd like that. If email sending fails, I'd like to get an alert anyway using a different channels

It might be reasonable to try all of the notification methods even if one fails. However, if any of them fail, I do believe certspotter needs to terminate. This allows the operator to correct the problem and restart certspotter, which will cause the failed notifications to be delivered. Continuing on failure means those notifications are essentially lost forever.

I see, indeed losing notifications forever is not desirable. Maybe attempting all notification channels and then returning an error could be a good compromise.

One thing I do not understand is what you mean by 'terminate', even if notifications fail I see certspotter running and attempting to deliver the email but not terminating. What am I missing?