How to Migrate Jaeger from Agent to Collector (OTLP) as DaemonSet?

[eilaytal]

Hi all,

I’m using Jaeger with the agent running as a DaemonSet to collect traces from each node and forward them to the collector. Here's my current config:

agent:
  enabled: true
  strategy: DaemonSet
  service:
    binaryPort: 6832
    compactPort: 6831
    samplingPort: 5778
  env:
    - name: COLLECTOR_SERVICE_HOST
      value: "jaeger-collector.jaeger.svc.cluster.local"
    - name: COLLECTOR_SERVICE_PORT_GRPC
      value: "14250"

collector:
  enabled: true
  service:
    grpc:
      port: 14250
    http:
      port: 14268

storage:
  type: elasticsearch  # This points to my OpenSearch backend

Since the agent may be deprecated, I’m looking to migrate to a collector-only setup (using OTLP) as a DaemonSet.

• How can I update my config to achieve this migration?
• Are there any potential issues to be aware of?

Thanks in advance!

[dosubot[bot]]

Hey there, @eilaytal! I'm here to assist you with any bugs, questions, or contributions you have. Let's tackle this migration together!

To migrate from a Jaeger Agent to a Collector-only setup using OTLP as a DaemonSet, you'll need to update your configuration and be aware of some potential issues.

Configuration Update

1. Remove the Agent Configuration: Since you're moving to a Collector-only setup, you can remove the agent configuration from your setup.

2. Configure the Collector to Use OTLP: Update your collector configuration to receive data via the OTLP protocol. Here's an example configuration:

   collector:
     enabled: true
     strategy: DaemonSet
     service:
       grpc:
         port: 4317  # Default gRPC port for OTLP
       http:
         port: 4318  # Default HTTP port for OTLP
     env:
       - name: SPAN_STORAGE_TYPE
         value: "elasticsearch"  # Ensure this matches your OpenSearch setup

3. Update Application Configuration: Ensure your applications are configured to send traces directly to the Jaeger Collector using the OTLP protocol. This typically involves setting the OTLP endpoint in your application's tracing configuration.

Potential Issues

1. Configuration Complexity: Without the Jaeger Agent, each application needs to be configured with the Collector's endpoint, which can add complexity, especially in multi-region setups.

2. Lack of Local Proxy: The Jaeger Agent acts as a local proxy, simplifying configuration. Without it, each application must be configured individually to communicate with the Collector.

3. Traffic Management: The Agent helps manage traffic to the Collectors by load balancing and reducing the number of open connections. Without it, there might be too many open connections for each Collector to handle, especially in environments with many hosts and applications.

4. Metadata Enrichment: The Agent can enrich tracing data with infrastructure-specific metadata. This functionality might be lost or require additional setup when using a Collector-only setup.

5. Additional Functionality: Features like strong authentication and multi-tenancy provided by the Agent might need alternative solutions in a Collector-only setup.

If these functionalities are required, consider deploying an OpenTelemetry Collector as a host agent ^[1][2].

By addressing these configuration changes and potential issues, you can successfully migrate to a Collector-only setup using OTLP as a DaemonSet.

To continue talking to Dosu, mention @dosu.

[eilaytal]

---

@dosu I've updated my configuration as follows:

agent:
  enabled: false

collector:
  enabled: true
  strategy: DaemonSet
  image:
    repository: jaegertracing/jaeger-collector
  service:
    type: ClusterIP
    otlp:
      grpc:
        port: 4317
        name: otlp-grpc
      http:
        port: 4318
        name: otlp-http

However, the collector is not deploying as a DaemonSet. Could you provide any insights on what I might be missing? I appreciate your help!

[dosubot[bot]]

If your Jaeger Collector is not deploying as a DaemonSet, here are a few things to check:

1. Kubernetes DaemonSet Configuration: Ensure that your Kubernetes manifest specifies the kind: DaemonSet for the Jaeger Collector. Here's a basic example:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: jaeger-collector
     namespace: jaeger
   spec:
     selector:
       matchLabels:
         app: jaeger-collector
     template:
       metadata:
         labels:
           app: jaeger-collector
       spec:
         containers:
         - name: jaeger-collector
           image: jaegertracing/jaeger-collector
           ports:
           - containerPort: 4317
             name: otlp-grpc
           - containerPort: 4318
             name: otlp-http

2. Check Logs and Events: Use kubectl logs and kubectl describe to check for any errors or events related to the DaemonSet. This can provide insights into why the DaemonSet might not be deploying correctly.

3. Resource Quotas and Limits: Ensure that there are no resource quotas or limits in your namespace that might prevent the DaemonSet from being scheduled on all nodes.

4. Node Selectors and Tolerations: Verify if there are any node selectors or tolerations required for the DaemonSet to be scheduled on the nodes.

5. Network Policies: Check if there are any network policies that might be preventing the Collector from communicating with other services or nodes.

By reviewing these aspects, you should be able to identify and resolve the issue preventing the Jaeger Collector from deploying as a DaemonSet.

To continue talking to Dosu, mention @dosu.

[eilaytal]

[dosubot[bot]]

If the Jaeger Collector is still not deploying as a DaemonSet, consider the following additional checks:

1. Kubernetes API Version: Ensure that the apiVersion in your DaemonSet manifest is correct. For most Kubernetes clusters, it should be apps/v1.

2. Namespace: Double-check that the DaemonSet is being created in the correct namespace. If your manifest specifies a namespace, make sure it matches the one you're using.

3. Cluster Role and Role Binding: Ensure that the necessary permissions are set up for the DaemonSet to be deployed. This might involve setting up a ClusterRole and ClusterRoleBinding if your setup requires it.

4. DaemonSet Selector: Verify that the selector in your DaemonSet spec matches the labels in the pod template. Any mismatch here can prevent the DaemonSet from managing the pods.

5. Pod Template: Ensure that the pod template within the DaemonSet spec is correctly configured, including the image, ports, and any necessary environment variables.

6. Kubernetes Version Compatibility: Check if there are any compatibility issues with the Kubernetes version you are using. Some features or configurations might differ between versions.

7. Check for Existing Resources: Ensure there are no existing resources with the same name that might conflict with the DaemonSet deployment.

If these checks do not resolve the issue, reviewing the output of kubectl describe daemonset <name> -n <namespace> and kubectl get events -n <namespace> can provide more detailed error messages or events that might indicate the root cause.

To continue talking to Dosu, mention @dosu.