v2 - Not able to extend declare block with custom logic

[dmitrii-didenko]

Hey everyone!

We have tried to upgrade helm chart to v2 and faced the issue with impossibility of configuring additional custom logic.

We are using experimental feature loki.secretfilter for obfuscation of logs. In helm chart v1 we pasted the following code to make it work:

loki.source.kubernetes "ingress_nginx_logs" {
  targets    = discovery.relabel.filtered_pod_logs.output <<<<< REUSING output from existing chain
  forward_to = [loki.process.ingress_nginx_logs.receiver]
}

loki.process "ingress_nginx_logs" {
  stage.match {
    selector = "{app!=\"ingress-nginx\"}"
    action = "drop"
  }

  forward_to = [loki.secretfilter.secret_filter.receiver]
}

loki.secretfilter "secret_filter" {
  forward_to = [loki.write.logs_service.receiver] <<<<
}

So basically we just reused existing discovery.relabel.filtered_pod_logs.output in our custom logic.
In v2, it seams like its impossible. The new declare block is closed for any inputs/outputs and we can customise it only by adding new extraDiscoveryRules and extraLogProcessingStages.
We can not add new loki.process there or hook into output flow to introduce additional chain:

declare "pod_logs" {
  argument "logs_destinations" {
    comment = "Must be a list of log destinations where collected logs should be forwarded to"
  }
...
  discovery.relabel "filtered_pods" {
    targets = discovery.kubernetes.pods.targets
...

loki.source.kubernetes "ingress_nginx_logs" { <<<<< NOT POSSIBLE
  targets    = discovery.relabel.filtered_pods.output 
  forward_to = [loki.process.ingress_nginx_logs.receiver]
}

Also we can not use output from declare block to supply the new processing chain somehow:

declare "pod_logs" {
  argument "logs_destinations" {
    comment = "Must be a list of log destinations where collected logs should be forwarded to"
  }
}

loki.source.kubernetes "ingress_nginx_logs" { 
  targets    = declare. pod_logs.output.filtered_pods <<<<< NOT POSSIBLE
  forward_to = [loki.process.ingress_nginx_logs.receiver]
}

Could you please review this and let me know you thoughts on that matter?

[petewall]

Let me just make sure I understand the real request. You want to take the podLogs feature (gather logs from all pods) and:

1. Filter down to just the nginx pods (drop all non-nginx pods)
2. Send log data to the loki.secretfilter component
3. Deliver to the logs destination as usual?

If that sounds right, let me know and I can work on some options for you.

[dmitrii-didenko]

Absolutely correct, thank you for reviewing this

[dmitrii-didenko]

@petewall hey, do you have any new info about this? Do I understand correctly that it's required to change helm chart to support this case? thanks

[petewall]

Yeah, we don't have the ability to add the secretfilter just yet. I think it's a great idea though, so I'd love to get that included. I'll see if I can do that next week sometime.