Typical kubernetes workload metrics as telemetry input to enable dashboarding and alerting

[a-thaler]

Description
The MetricPipeline supports already an input type runtime which emits metrics around the container and pod resource consumption. What is missing are further typical metrics:

• from the apiserver about configured resource limits
• from the apiserver around the state of workloads
• from the kubelet statistics of the volumes
• from the kubelet statistics of the nodes
  mainly the typical metrics resulting from the kubletstatsreceiver and the k8sclusterreceiver

Having these metrics available, basic troubleshooting for kubernetes workload including alerting can be fullfiled.

Goal
Provide a way to collect a typical set of metrics for basic workload troubleshooting (comparable to the metrics used by the dashboards provided by the kube-prometheus-stack)

Criterias

• Typical metrics are collectable which are needed to troubleshoot
  • Pod compute resource
  • Node resource usage
  • Volume resource usage
  • Health of workloads (deployment stuck for example)
• Namespace specific metrics can be enabled per namespace (probably independent from non-namespaces resources)
• Node and Volume related metrics can be enabled optional to workload related metrics

Actions

• Preparations
  • Build understanding of the available receivers and make an API proposal (Metric API proposal to cover typical Kubernetes metrics #1001)
  • Come up with a concept on how to run the k8sclusterreceiver which does not fit into the current architectural setup (PoC for operating the k8sclusterreceiver #1003)
• Implementation & Rollout
  • Have the existing input configurable for pod and container (Option to disable pod and/or container metrics #1183)
  • Enrich the container and pod sub-inputs with the metrics from the k8sclusterreceiver (Enhance runtime input with k8sclusterreciver metrics #1184)
  • Add new inputs for nodes being disabled by default for new pipelines (Enhance runtime input with selectors for nodes #1300)
  • Add manual Cloud Logging Dashboard for node metrics (docs: Add Cloud Logging dashboards for new metric features #1494)
  • Add new inputs for volumes being disabled by default for new pipelines (Enhance runtime input with selectors for PVC volumes #1301)
  • Add manual Cloud Logging Dashboard for volume metrics (docs: Add new cloud logging dashboard for k8s volumes #1528)
  • Add new inputs for daemonset/deployment/statefuleset/job being disabled by default #1521
  • Add manual Cloud Logging Dashboard for workload metrics (feat: Add cloud logging dashboard for workloads #1577)
  • Switch defaults to "on" for basic selectors, so that new pipelines expose them by default while existing ones stay where they are

Reasons
The current feature set is a good start but are missing apiserver related details like limits to get a complete picture for troubleshooting and defining relevant alerts. Furthermore typical workload health related metrics are missing from the apiserver. Also volumes and node statistics are important in daily operations.

Attachments

Release Notes

[a-thaler]

Feature will be fully rolled out with version 1.27.0. Afterwards, the defaults get changed so that the sub-selectors are enabled by default for new clusters