help request: opentelemetry input does not update the fluent-bit service

[Amirbekyan]

Describe the issue

Adding opentelemetry input/clusterinput resources neither triggers addition of 4318 port to existing fluent-bit service, nor creation of a new one. The fluentbit-config k8s secret is updated successfully:

fluent-bit.conf: |                                                                                                                                                                                                
  [Service]                                                                                                                                                                                                       
      Http_Server    true                                                                                                                                                                                         
      Parsers_File    /fluent-bit/etc/parsers.conf                                                                                                                                                                
  [Input]                                                                                                                                                                                                         
      Name    opentelemetry                                                                                                                                                                                       
      listen    0.0.0.0                                                                                                                                                                                           
      port    4318                                                                                                                                                                                                
      tag_key    traces
...

fluent-bit cotainers are successfully loading the configuration and starting listening on port 4318:

* Copyright (C) 2015-2024 The Fluent Bit Authors                                                                                                                                                                  
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd                                                                                                                                                  
* https://fluentbit.io                                                                                                                                                                                            
                                                                                                                                                                                                                  
______ _                  _    ______ _ _           _____  __                                                                                                                                                     
|  ___| |                | |   | ___ (_) |         |____ |/  |                                                                                                                                                    
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |                                                                                                                                                    
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |                                                                                                                                                    
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_                                                                                                                                                   
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/                                                                                                                                                   
                                                                                                                                                                                                                  
[2024/11/15 11:24:38] [ info] [fluent bit] version=3.1.7, commit=c6e902a43a, pid=12                                                                                                                               
[2024/11/15 11:24:38] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128                                                                                                      
[2024/11/15 11:24:38] [ info] [cmetrics] version=0.9.5                                                                                                                                                            
[2024/11/15 11:24:38] [ info] [ctraces ] version=0.5.5                                                                                                                                                            
[2024/11/15 11:24:38] [ info] [input:opentelemetry:opentelemetry.0] initializing                                                                                                                                  
[2024/11/15 11:24:38] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)                                                                                                       
[2024/11/15 11:24:38] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4318
...

but the pod and service ports configs are not updated, thus the port is not properly exposed

pod ⬇️

...
    name: fluent-bit                                                                                                                                                                                              
    ports:                                                                                                                                                                                                        
    - containerPort: 2020                                                                                                                                                                                         
      name: metrics                                                                                                                                                                                               
      protocol: TCP                                                                                                                                                                                               
    resources:         
...

service ⬇️

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit
...
spec:
  ports:
  - name: metrics
    port: 2020
    protocol: TCP
    targetPort: 2020
  selector:
    app.kubernetes.io/name: fluent-bit
...

How did you install fluent operator?

i've used this chart https://github.com/fluent/fluent-operator/tree/master/charts/fluent-operator

Additional context

No response

[wenchajun]

You can check the logs of the fluent-operator to see if everything is normal.

[Amirbekyan]

here are the logs from the fluent-operator pod:

fluent-operator 2024-11-19T11:45:50Z    INFO    setup    starting manager
setenv + echo 'CONTAINER_ROOT_DIR=/var/log'
fluent-operator 2024-11-19T11:45:50Z    INFO    controller-runtime.metrics    Starting metrics server
fluent-operator 2024-11-19T11:45:50Z    INFO    controller-runtime.metrics    Serving metrics server    {"bindAddress": ":8080", "secure": false}
fluent-operator 2024-11-19T11:45:50Z    INFO    starting server    {"name": "health probe", "addr": ":8081"}
Stream closed EOF for fluent/fluent-operator-57cb87b7cc-2gkp5 (setenv)
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBit"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.Fluentd"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1.Secret"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterFluentBitConfig"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBitConfig"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterInput"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterFilter"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterOutput"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterParser"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterMultilineParser"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.Filter"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.Output"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.Parser"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.MultilineParser"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting Controller    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1.ServiceAccount"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1.DaemonSet"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1.StatefulSet"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1.Service"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting Controller    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.Fluentd"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1.Secret"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.ClusterFluentdConfig"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.FluentdConfig"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.Filter"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.ClusterFilter"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.Output"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "source": "kind source: *v1alpha1.ClusterOutput"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting Controller    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBit"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1.ServiceAccount"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1.DaemonSet"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting Controller    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1alpha2.Collector"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1.ServiceAccount"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1.StatefulSet"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1.Service"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting Controller    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector"}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting workers    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "worker count": 1}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting workers    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "worker count": 1}
fluent-operator 2024-11-19T11:45:56Z    INFO    Starting workers    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "worker count": 1}
fluent-operator 2024-11-19T11:45:56Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    Starting workers    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "worker count": 1}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluentbit-config", "secret": "fluentbit-config"}
fluent-operator 2024-11-19T11:45:57Z    INFO    Starting workers    {"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "worker count": 1}

[Amirbekyan]

@wenchajun ⬆️