Migrate away from ImprovMX for mailing lists

[jfly]

We currently use ImprovMX to handle mail sent to @nixos.org (see relevant dns entries).

• We only use ImprovMX for mail forwarding (teams like infra@, marketing@, etc). Today, nobody sends mail from @nixos.org, and nobody has any inboxes.
• You need a web account with ImprovMX to see and to update these mail forwards. The Nix community can't see/audit any of this.
• There are various limits (number of forwards, perhaps the number of emails an address can forward to?). See https://improvmx.com/pricing/. I don't know if we're currently paying for ImprovMX. I think I heard that we've run into some of these limits.

The plan

A few weeks ago, @Mic92 asked me to look into self hosting this instead. He recommended Simple NixOS Mailserver (SNM). I've played with it a bit, and it does seem like a good fit here.

1. Install SNM on umbriel.
   • The configuration docs here are great: https://nixos-mailserver.readthedocs.io/en/latest/setup-guide.html.
   • Leave mailserver.loginAccounts empty, and disable pop/imap.
   • Port the existing mailing lists from ImprovMX to mailserver.forwards
     • @Mic92 has posted a dump [REMOVED] (accurate as of 2024-09-30).
2. Verify this server can successfully send mail (target: 10/10 on https://www.mail-tester.com/). Either by temporarily adding a login account, or speaking directly to postfix via the cli.
3. TBD (see below): configure a metrics/alerting solution. Intentionally break emailing somehow and confirm that the infra team gets notified about stale mail stuck in queues.
4. Talk to t-online and outlook to tell them we exist
5. Wait until the Nix Steering Committee Election is done: https://nixos.org/blog/announcements/2024/sc-election-2024/.
6. Rollout the change (ETA: early December 2024)
   • Check that listsWithSecretFiles is up to date
   • Switchover the MX records from ImprovMX to umbriel.nixos.org.
   • After the MX record change has propagated everywhere (check with https://www.whatsmydns.net/), verify that email forwards still work. If not, switch the MX records back.
   • Cleanup: shut down our ImprovMX account, or do whatever we can to reduce confusion about this

Open questions/TBD:

1. Monitoring
   • Ideally, the infra team would get alerted if emails have been sitting in a postfix queue for a long time. Are there any best practices for this? We use Prometheus, perhaps https://github.com/kumina/postfix_exporter is a good pick? It's packaged in nixpkgs =)
   • @jfly chatted with @Mic92, and we're going to start with "blackbox" monitoring, which runs on pluto. Dumping some links from our discussion:
     • smtp_starttls cannot work well prometheus/blackbox_exporter#913
     • https://github.com/prometheus/blackbox_exporter/blob/53e78c2b3535ecedfd072327885eeba2e9e51ea2/example.yml#L124
     • probe_ssl_earliest_cert_expiry
     • https://search.nixos.org/options?channel=24.05&show=services.prometheus.exporters.blackbox.enable&from=0&size=50&sort=relevance&type=packages&query=blackbox
     • http://build01.nix-community.org:9273/metrics
   • Anything else?
2. Backups
   • Arguably not necessary. This service is pretty much stateless (except for the mail stuck in queues, which perhaps we can live with?)

Alternatives considered

• I don't know if there's been any serious discussion about paying someone (ImprovMX or something else) to handle this for us. Since declarative management and audit-ability are important to us, it would either have to be a provider that has a Terraform provider, or we could build one ourselves.
• @Mic92, can you shed any light on this?

[SuperSandro2000]

I just want to make awareness that you probably need to write a mail to t-online and outlook (none 356) to whitelist your IP otherwise mails cannot be delivered.

[mweinelt]

After the leak of the existing email mappings I would be interested in discussing the privacy aspect of the email mappings. What other organization publishes those? The current set of addresses were not given to us by its recipients with the intent to make them public.

[jfly]

I just want to make awareness that you probably need to write a mail to t-online and outlook (none 356) to whitelist your IP otherwise mails cannot be delivered.

I hear you on this. I've never run a mailserver before, and honestly have no idea what our deliverability is going to be like. I believe the current set of emails is quite tiny, and may not even include any t-online or outlook. My personal opinion on this is that we should make sure we've solved the monitoring story: if we get notified for email stuck in queues, then we can tackle these allowlists as necessary, or we can give up and pay someone to handle this for us.

After the leak of the email mappings I would be interested in discussing the privacy aspect of the email mappings.

Sorry about that. I asked one person about this, but should have talked to more people before posting.

Ideas:

1. We could encrypt the email addresses. This would be hard to code review.
2. We could seek consent from all the relevant people. I don't know how hard this would be. I don't have the list anymore, but it didn't seem like an insurmountable number.
3. Do this behind some self-hosted (or paid) webapp with a login. That's basically what we do today with ImprovMX.

[Mic92]

I just want to make awareness that you probably need to write a mail to t-online and outlook (none 356) to whitelist your IP otherwise mails cannot be delivered.

For T-Online at least this is just one email after setting up reverse DNS and everything up correctly.

Overall I also don't expect the NixOS foundation to have to handle large volume of email. The vote was the first time, we had to do this actually.

[Mic92]

1. We could encrypt the email addresses. This would be hard to code review.
2. We could seek consent from all the relevant people. I don't know how hard this would be. I don't have the list anymore, but it didn't seem like an insurmountable number.
3. Do this behind some self-hosted (or paid) webapp with a login. That's basically what we do today with ImprovMX.

@zimbatm started to ask existing users of email addresses about that.

[Mic92]

I hear you on this. I've never run a mailserver before, and honestly have no idea what our deliverability is going to be like. I believe the current set of emails is quite tiny, and may not even include any t-online or outlook. My personal opinion on this is that we should make sure we've solved the monitoring story: if we get notified for email stuck in queues, then we can tackle these allowlists as necessary, or we can give up and pay someone to handle this for us.

Some DMARC and reading the mail logs in case there are delivery problems. I didn't had any big issues with emails for the NixOS wiki and that looks more like bulk messages compared to what I expect to be sent from nixos.org.

[zimbatm]

@jfly Is it possible to move the email addresses into sops-encoded secrets, or is this part only configurable with plain Nix code?

[SuperSandro2000]

For T-Online at least this is just one email after setting up reverse DNS and everything up correctly.

And you need to have a proper imprint on the TLD of the rDNS entry and contact means via I think telephone and e-mail that is not going over the mail server.

I have recently done it and it took me a few back and forths but it is doable.

[jfly]

@jfly Is it possible to move the email addresses into sops-encoded secrets, or is this part only configurable with plain Nix code?

It currently requires plain Nix code:

• The relevant SNM code sets nixpkgs's services.postfix.virtual.
• nixpkgs massages that into services.postfix.config. This code is setting Postfix's virtual_alias_maps setting.

Adding support for encrypted emails seems like it might actually not be too hard:

• We could adjust the nixpkgs service to allow for multiple virtual_alias_maps (currently it supports exactly 0 or 1), and then we could add a new entry to that array to point at a virtual alias map generated with a sops-nix template.
  • I think the nixpkgs change I have in mind will look weird. We might need a more generic solution that has a satisfying answer to this question: "why does virtual_alias_maps get this special escape hatch but not other maps like alias_maps?"
• Adding a new entry is a little tricky because you actually need to run postmap to "compile" these mappings, but I think the existing services.postfix.mapFiles option is flexible enough to do this for us without changes.

tl;dr:

• It's possible, but requires changes to nixpkgs, and perhaps SNM, depending on how we want to expose this.
• I'm willing to do this work, but would prefer to wait until we know if it's necessary first.
  • If it makes sense to do this work, I could use a brainstorm partner on the nixpkgs change.