[Bug]: OTEL sender GRPC not compatible with OTLP receiver on collector

[tdarwin]

Getting the following errors in my otel collector when trying to send OTEL traffic with loadgen:

2024-11-13 08:35:42 2024-11-13T14:35:42.504Z    info    zapgrpc/zapgrpc.go:176  [transport] [server-transport 0x40021d1500] Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x00\\xf5\\x01\\x00\\x00\\xf1\\x03\\x03n\\x9d\\xf9\\xfa\\x19EL,<Gq\\xba\\x19\""    {"grpc_log": true}
2024-11-13 08:35:42 2024-11-13T14:35:42.505Z    info    zapgrpc/zapgrpc.go:176  [core] [Server #2]grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x00\\xf5\\x01\\x00\\x00\\xf1\\x03\\x03n\\x9d\\xf9\\xfa\\x19EL,<Gq\\xba\\x19\""  {"grpc_log": true}
2024-11-13 08:35:42 2024-11-13T14:35:42.509Z    info    zapgrpc/zapgrpc.go:176  [transport] [server-transport 0x40021d1680] Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x00\\xf5\\x01\\x00\\x00\\xf1\\x03\\x03cMR\\xd7\\xeb\\xc6@\\x00]iR8\\xdd\""        {"grpc_log": true}
2024-11-13 08:35:42 2024-11-13T14:35:42.509Z    info    zapgrpc/zapgrpc.go:176  [core] [Server #2]grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x00\\xf5\\x01\\x00\\x00\\xf1\\x03\\x03cMR\\xd7\\xeb\\xc6@\\x00]iR8\\xdd\""      {"grpc_log": true}

It looks like traces are not accepted by the collector, and they're definitely not sent on to Honeycomb.

Versions

• Go: 1.23.1
• loadgen: built from main on 11/13/2024

Steps to reproduce

1. Create a collector docker container with the following basic config:
   otelcol-config.yaml:

   exporters:
     otlp:
       endpoint: api.honeycomb.io:443
       headers:
         x-honeycomb-team: ${HONEYCOMB_API_KEY}
     debug:
       verbosity: detailed
   
   processors:
     memory_limiter:
       check_interval: 5s
       limit_mib: 80
       spike_limit_mib: 25
     batch: {}
   
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: otel-collector:4317
         http:
           endpoint: otel-collector:4318
   
   service:
     telemetry:
       logs:
         level: debug
     pipelines:
       traces:
         receivers: [otlp]
         processors: [memory_limiter, batch]
         exporters: [otlp]

docker-compose.yaml:

  services:
    otel-collector:
      image: otel/opentelemetry-collector-contrib
      hostname: otel-collector
      restart: always
      environment:
        - HONEYCOMB_API_KEY=<MY_INGEST_KEY>
      command: ["--config=/etc/otel-collector-config.yaml"]
      volumes:
        - ./collector_configs/otelcol-config.yaml:/etc/otel-collector-config.yaml
      ports:
        - "4317:4317" # OTLP gRPC receiver
        - "4318:4318" # OTLP HTTP receiver
      networks:
        - honeycomb
  
  networks:
    honeycomb:
      driver: bridge

2. Use a current build of loadgen to attempt to send traffic to the collector container:

  export HONEYCOMB_API_KEY=<MY_INGEST_KEY>
  ./loadgen --host="localhost:4317" --sender=otel --nspans=50 --extra=10 --tracetime=500ms --tps=10 --runtime=5s --dataset=otel-loadgen

3. Observe errors in the collector container

Additional context

This is also using the latest image of collector-contrib

[tdarwin]

Okay, this requires the --insecure option, and then it works... My bad. Closing the issue.