No Logs Output Despite Enabling Logging Configuration

[qchenzi]

Hi everyone,

I'm encountering an issue with the logging configuration in my Milvus setup. Despite setting the log level to "info" and enabling the logging configuration, I am not seeing any logs being output.

Here are the details of my configuration:

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: my-release
  labels:
    app: milvus
spec:
  mode: cluster
  config: {}
  components:
    image: "image.midea.com/datamars/milvusdb/milvus:v2.4.4"
    enableRollingUpdate: true
    imageUpdateMode: rollingUpgrade
    proxy:
      replicas: 1
      serviceType: ClusterIP
    dataNode:
      replicas: 1
    indexNode:
      replicas: 1
    queryNode:
      replicas: 1
    mixCoord:
      replicas: 1
  dependencies:
    etcd:
      inCluster:
        values:
          replicaCount: 3
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 1
              memory: 2Gi
          persistence:
            size: 20Gi
        deletionPolicy: Delete
        pvcDeletion: true
    storage:
      inCluster:
        values:
          replicaCount: 4
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 1
              memory: 2Gi
          persistence:
              size: 20Gi
        deletionPolicy: Delete
        pvcDeletion: true
    msgStreamType: kafka
    kafka:
      inCluster:
        values:
          defaultReplicationFactor: 3
          offsetsTopicReplicationFactor: 3
          replicaCount: 3
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 1
              memory: 2Gi
          persistence:
            size: 20Gi
          zookeeper:
            replicaCount: 3
            resources:
              limits:
                cpu: 1
                memory: 2Gi
              requests:
                cpu: 1
                memory: 2Gi
            persistence:
              size: 20Gi
        deletionPolicy: Delete
        pvcDeletion: true

```yaml

 Is there something I might be missing in the configuration, or are there any known issues related to logging in this setup?
Thank you for your help!

[haorenfsa]

Hi @qchenzi.
First from the Milvus CR given, it seems that you didn't set the log level through CR.

I'm not sure how you changed the configurations. But if you somehow changed the log level after Milvus is already stated, there's another issue with dynamic configuration update. which is not fix until the recent v2.4.5.

The configuration you set is also not correct. Below is an example to alter milvus log configuration.

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: my-release
  labels:
    app: milvus
spec:
  config: 
    log:
      level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
      file:
        rootPath:  # root dir path to put logs, default "" means no log file will print. please adjust in embedded Milvus: /tmp/milvus/logs
        maxSize: 300 # MB
        maxAge: 10 # Maximum time for log retention in day.
        maxBackups: 20
      format: text # text or json
      stdout: true # Stdout enable or not
## other fields

For more fields, plz check https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml

[qchenzi]

Thank your assistance @haorenfsa , I will give it a try.

I have one more question: Does Milvus support setting the log directory to a PVC mount? If so, could you please provide some guidance on how to configure it?

Thanks again!

[haorenfsa]

Does Milvus support setting the log directory to a PVC mount?
Yes

First add volumes & volumemounts to your pod under the field spec.components. for more info check the CRD doc.

Then change the spec.config.log.file.rootPath to your pvc mount path.
The full CR structure would look like below:

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: my-release
  labels:
    app: milvus
spec:
  components:
    volumes: []
    volumeMounts: []
  config: 
    log:
      level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
      file:
        rootPath:  # root dir path to put logs, default "" means no log file will print.

[qchenzi]

Hi @haorenfsa
Thank you for the detailed instructions. I will follow these steps to set up the log directory with a PVC mount.

And I noticed that components like DataNode and IndexNode are deployed using Deployments, while dependencies like etcd and Kafka are deployed using StatefulSets. Is this because DataNode and IndexNode are stateless services?

[haorenfsa]

@qchenzi The nodes of milvus' states are auto registered into dependent etcd. So it's fine to use deployment.