Fix varnish monitoring 2.0

[PhilTaken]

the default monitoring check relied on varnish to run on a specific address.
since this address can be changed, the sensu check needs to be adjusted to correctly infer the host and port that varnish is running on

This pr is based on #996 but fixes an issue where an incorrect address was inferred when multiple addresses were passed to varnish.

FC-36891

@flyingcircusio/release-managers

Release process

Impact:

Changelog:

PR release workflow (internal)

• PR has internal ticket
• internal issue ID (PL-…) part of branch name
• internal issue ID mentioned in PR description text
• ticket is on Platform agile board
• ticket state set to Pull request ready
• if ticket is more urgent than within the next few days, directly contact a member of the Platform team

Design notes

• Provide a feature toggle if the change might need to be adjusted/reverted quickly depending on context. Consider whether the default should be on or off. Example: rate limiting.
• All customer-facing features and (NixOS) options need to be discoverable from documentation. Add or update relevant documentation such that hosted and guided customers can understand it as well.

Security implications

• Security requirements defined? (WHERE)
  • the varnish monitors the correct address when not using the default
• Security requirements tested? (EVIDENCE)

case where a single address is set:

phil@test01 ~ $ sensu-client-show-config | jq '.checks.varnish_http'
{
  "command": "check_http -H 172.22.57.135 -p 8080 -c 10 -w 3 -t 20 -e HTTP",
  "interval": 60,
  "notification": "varnish port 8080 HTTP response",
  "standalone": true,
  "timeout": null,
  "ttl": null,
  "warnIsCritical": false
}

phil@test01 ~ $ cat /etc/local/nixos/varnish_http.nix
{ lib, ... }:
{
  services.varnish.http_address = lib.mkForce "172.22.57.135:8080";
}

phil@test01 ~ $

case where multiple addresses are set:

phil@test01 ~ $ sensu-client-show-config | jq '.checks.varnish_http'
{
  "command": "check_http -H 127.0.0.1 -p 8070 -c 10 -w 3 -t 20 -e HTTP",
  "interval": 60,
  "notification": "varnish port 8070 HTTP response",
  "standalone": true,
  "timeout": null,
  "ttl": null,
  "warnIsCritical": false
}

phil@test01 ~ $ cat /etc/local/nixos/varnish_http.nix
{ lib, ... }:
{
  services.varnish.http_address = lib.mkForce "*:8070 -a 127.0.0.1:8081";
}

phil@test01 ~ $

and the default when no address is set:

phil@test01 ~ $ sudo nix repl
[sudo] password for phil:
Welcome to Nix 2.18.1. Type :? for help.

nix-repl> :l <nixpkgs/nixos>
Added 6 variables.

nix-repl> config.services.varnish.http_address
"172.22.57.135:8008 -a [2a02:248:101:63::19db]:8008 -a 127.0.0.1:8008 -a [::1]:8008"

nix-repl>

phil@test01 ~ $ cat /etc/local/nixos/varnish_http.nix
{ lib, ... }:
{
}

phil@test01 ~ $ sensu-client-show-config | jq '.checks.varnish_http'
{
  "command": "check_http -H 172.22.57.135 -p 8008 -c 10 -w 3 -t 20 -e HTTP",
  "interval": 60,
  "notification": "varnish port 8008 HTTP response",
  "standalone": true,
  "timeout": null,
  "ttl": null,
  "warnIsCritical": false
}

phil@test01 ~ $

[osnyx]

While the code in general looks plausible, there are some general questions and decisions we need to take before investigating the details makes sense.

Happy to discuss this with you if needed or I misunderstood something.

[osnyx]

Adding a proper option description is good. We might want to document the approach of how to add multiple listen addresses here as well.

[osnyx]

The question here is what we want to achieve with our monitoring:

• monitor the varnish server process and whether it works -> it is sufficient to reach varnish via any of its multiple listen addresses, in this implementation we always take the first one
• monitor the listen address configuration as well: is varnish actually reachable via all configured IP-port configurations? Did we mis-type something, did one of the IPs somehow change, etc…
  • in that case, we need to run the http check against all the configured listen addresses, not just the first one.

[osnyx]

rebased onto latest dev branch to fix a test failure in docs.

[osnyx]

We also noticed that the docs regarding monitoring are outdated. It'd be really neat if you updated that part to reflect the current state. Can be a brief overview of what monitoring is happening, no need for deep details.

https://doc.flyingcircus.io/roles/fc-24.05-production/webproxy.html