Opt metrics-server into ambient

[mjnagel]

Following completion of #1248, use the implemented toggle to opt the metrics-server layer into ambient mode.

Ensure that all metrics-server functionality still works and any end-to-end testing works as expected. Also re-evaluate any "istio workarounds" if present (peerauthentications and headless services in particular) and see if they can be removed. Also ensure that all interactions with other core services (if applicable) continue to function as expected.

Definition of done: Metrics Server runs in ambient mode within core, and functions as expected.