Fluentbit cannot detect filename from all namespaces

[Angel0r]

Hello everyone!

I am using Fluentbit Helm Chart in Kubernetes and send logs to Grafana/Loki endpoint. I can see almmost everything but for some reason I cannot see the filename in other namespace other than the namespace where Fluentbit is installed. My configuration looks like bellow:

      customParsers: |
        [PARSER]
            Name docker_no_time
            Format json
            Time_Keep Off
            Time_Key time
            Time_Format %Y-%m-%dT%H:%M:%S.%L
      filters: |
        [FILTER]
            Name                   kubernetes
            Match                  kube.*
            Buffer_Size            1MB
            Merge_Log              On
            Keep_Log               Off
            K8S-Logging.Parser     On
            K8S-Logging.Exclude    On
      inputs: |
        [INPUT]
            Name                  tail
            Path                  /var/log/containers/*.log
            multiline.parser      docker, cri
            Tag                   kube.*
            Mem_Buf_Limit         5MB
            Skip_Long_Lines       On

        [INPUT]
            Name                  systemd
            Tag                   host.*
            Systemd_Filter        _SYSTEMD_UNIT=kubelet.service
            Read_From_Tail        On
      outputs: |
        [OUTPUT]
            Name                   loki
            Match                  *
            Host                   vm-loki.k8s.com
            Port                   443
            Http_User              ${username}
            Http_Passwd            ${password}
            TLS                    On
            tls.verify             On
            Labels                 job=fluent-bit,namespace=$kubernetes['namespace_name'],pod=$kubernetes['pod_name'],container=$kubernetes['container_name']
            Auto_Kubernetes_Labels on
            Compress               gzip
      service: |
        [SERVICE]
            Daemon                Off
            Flush                 {{ .Values.flush }}
            Log_Level             {{ .Values.logLevel }}
            Parsers_File          /fluent-bit/etc/parsers.conf
            Parsers_File          /fluent-bit/etc/conf/custom_parsers.conf
            HTTP_Server           On
            HTTP_Listen           0.0.0.0
            HTTP_Port             {{ .Values.metricsPort }}
            Health_Check          On

In Grafana when I search on Fluentbit namespace I can see the filename field
like filename /var/log/pods/fluentbit-namespace-123123dfsf2ws-erwef213-1231.log. However, when I search on an other namespace this field doesn't exist.

I have configured serviceAccount, clusterRole, and clusterRoleBindings with clusterRole having resources: ["namespaces", "pods", "pods/log"].

What am I missing?

/Angelos