CosmosFullNode sentry creates privval services.

[DavidNix]

Like p2p services, 1 per pod.

This is so remote signers can connect to a stable address.

We probably need to deprecate the readiness probe strategy. Readiness has to be turned off for sentries to be added to the service.

It's a chicken or egg problem. Comet tries to connect immediately and blocks thus failing readiness probes. The remote signer must connect to the sentry pod but can't if the sentry is not ready. (We have other safeguards for rollouts which now check the state of Comet.) So removing the probes should have little impact.

It's worth investigating how StatefulSets give each pod a stable network id.

The service is likely still necessary given the user may need to configure it. E.g. We used internal lb services on GKE.

[DavidNix]

We really want to explore using the pod's stable DNS name given #360 instead of creating a service for each pod. But advantage to a service is you can configure it more, such as internal LB IPs.