beyla.ebpf - New Pods Missing Resource Attribute Labels

[matt-gp]

What's wrong?

Hi all,

I'm trying to get the k8sattributes processor working with beyla but am having issues where traces from pods lose their annotations and metadata once the pod has been rotated.

What I've worked out is that once the pods are restarted beyla.ebpf stops adding k8s metadata to the resource attributes and then the k8sattribute processor can't match up the spans to pods.

I've been testing this using Kind and Alloy is running as a daemonset.

No errors are reported in the logs and Alloy has enough permissions.

Just seems that as soon as some pods are rotated that alloy forgets to add the metadata etc.

I'm hoping this is just an issue with my configuration but any help would be appreciated.

Steps to reproduce

• Run a kind cluster
• Run a random deployment with some annotations.
• Create alloy daemonset using the below config.
• Once the alloy daemonset is running, rollout restart the deployment.

System information

linux/arm64/kubernetes:v1.29.2

Software version

grafana/alloy:v1.4.3

Configuration

beyla.ebpf "default" {
  output {
    traces = [otelcol.processor.k8sattributes.default.input]
  }
  
  attributes {
    kubernetes {
      enable = "true"
      cluster_name = "test"
    }
  }

  discovery {
    services {
      kubernetes {
        namespace = ".*"
      }
    }
  }

  metrics {
    features = ["application","application_span","application_service_graph","application_process"]
  }
}

otelcol.processor.k8sattributes "default" {

  extract {
    annotation {
      from     = "pod"
      key      = "example.com/team"
      tag_name = "team"
    }
  }

  pod_association {

    source {
      from = "resource_attribute"
      name = "k8s.pod.name"
    }

    source {
      from = "resource_attribute"
      name = "k8s.namespace.name"
    }
  }

  output {
    traces = [otelcol.processor.groupbyattrs.default.input]
  }
}

otelcol.processor.groupbyattrs "default" {
  keys = ["team"]
  output {
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  metadata_keys = ["team"]
  output {
    traces = [otelcol.exporter.debug.default.input]
  }
}

otelcol.exporter.debug "default" {
  verbosity = "detailed"
  use_internal_logger = false
}

Logs

Before the deployment is restarted

{"ts":"2024-10-31T22:54:22.627883417Z","level":"debug","msg":"evaluating pod identifier","component_path":"/process_traces.traces.default","component_id":"otelcol.processor.k8sattributes.default","value":[{"Source":{"From":"resource_attribute","Name":"k8s.pod.name"},"Value":"mimir-6db8548bd4-bpmsv"},{"Source":{"From":"resource_attribute","Name":"k8s.namespace.name"},"Value":"monitoring"},{"Source":{"From":"","Name":""},"Value":""},{"Source":{"From":"","Name":""},"Value":""}]}

Resource attributes:
     -> service.name: Str(loki)
     -> telemetry.sdk.language: Str(go)
     -> telemetry.sdk.name: Str(beyla)
     -> host.name: Str(loki-5868469c7b-csx8n)
     -> host.id: Str(88742f305b054aa0870f91054c288cdb)
     -> service.namespace: Str(monitoring)
     -> k8s.node.name: Str(kind-worker3)
     -> k8s.pod.uid: Str(5d1e946b-db06-44dd-8cd0-fc6c359583bb)
     -> k8s.pod.start_time: Str(2024-11-10 21:57:29 +0000 UTC)
     -> k8s.cluster.name: Str(test)
     -> k8s.replicaset.name: Str(loki-5868469c7b)
     -> k8s.deployment.name: Str(loki)
     -> k8s.namespace.name: Str(monitoring)
     -> k8s.pod.name: Str(loki-5868469c7b-csx8n)
     -> service.instance.id: Str(alloy-bps5z-88188)
     -> otel.library.name: Str(github.com/grafana/beyla)
     -> team: Str(loki)

After rotation

{"ts":"2024-10-31T22:54:22.62843Z","level":"debug","msg":"evaluating pod identifier","component_path":"/process_traces.traces.default","component_id":"otelcol.processor.k8sattributes.default","value":[{"Source":{"From":"","Name":""},"Value":""},{"Source":{"From":"","Name":""},"Value":""},{"Source":{"From":"","Name":""},"Value":""},{"Source":{"From":"","Name":""},"Value":""}]}

Resource attributes:
     -> service.name: Str(loki)
     -> telemetry.sdk.language: Str(go)
     -> telemetry.sdk.name: Str(beyla)
     -> host.name: Str(alloy-dptnw)
     -> host.id: Str(1ec4a9bedd034cb495e9fbc5c2f97bfa)
     -> service.instance.id: Str(alloy-dptnw-93093)
     -> otel.library.name: Str(github.com/grafana/beyla)