"validate" command does not detect "connector x used as receiver in logs pipeline but not used in any supported exporter pipeline" misconfiguration

[ringerc]

Describe the bug

The validate sub-command of the OpenTelemetry Collector executable does not detect a connector use misconfiguration where a connector is used as a receiver but not in any exporters.

Steps to reproduce

Save the configuration from the below section as config.yaml, then run:

$ docker run -i --rm --mount=type=bind,source=${PWD}/config.yaml,destination=/etc/otel-collector/config.yaml otel-collector validate --config=/etc/otel-collector/config.yaml
$ echo $?
0

This will execute without errors. But if you try to start the collector with this configuration it will fail to start with a configuration error relating to channel configuration:

$ docker run -it --mount=type=bind,source=${PWD}/config.yaml,destination=/etc/otel-collector/config.yaml otel-collector --config=/etc/otel-collector/config.yaml 
2023-11-13T21:48:54.278Z	info	[email protected]/telemetry.go:84	Setting up own telemetry...
2023-11-13T21:48:54.279Z	info	[email protected]/telemetry.go:201	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
Error: failed to build pipelines: connector "forward/bugrepro" used as receiver in logs pipeline but not used in any supported exporter pipeline
2023/11/13 21:48:54 collector server run finished with error: failed to build pipelines: connector "forward/bugrepro" used as receiver in logs pipeline but not used in any supported exporter pipeline

The validate sub-command fails to detect this misconfiguration.

What did you expect to see?

Validation failure with error like

Error: failed to build pipelines: connector "forward/test" used as receiver in "logs/out" pipeline but not used in any supported exporter pipeline

Note that the specific logs pipeline is mentioned in the "expect to see" error I'd want to see from validate. The current error message only mentions "logs" pipeline, it does not name the actual pipeline instance affected, like logs/out in my example config.

What did you see instead?

Validate success.

Runtime execution fails with above error.

See repro section.

What version did you use?

$ docker run -i --mount=type=bind,source=${PWD}/config.yaml,destination=/etc/otel-collector/config.yaml otel-collector --version                                    
otelcol-contrib version 0.87.0

What config did you use?

This configuration is deliberately invalid to illustrate the validator issue. It is greatly simplified from my real deployment to provide a minimal example.

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  debug:
connectors:
  forward/bugrepro: {}
service:
  pipelines:
    # The 'validate' subcommand fails to notice that there is no
    # exporter reference to the channel, but a runtime check will
    # cause the collector to fail to start up.
    #
    #logs/in:
    #  receivers: [otlp]
    #  processors: []
    #  exporters: [forward/bugrepro]
    logs/out:
     receivers: [forward/bugrepro]
     processors: []
     exporters: [debug]

In the deployment I was using two --config arguments, but I'd used a logs pipeline entry in both (not logs/in or logs/out or whatever), so the logs entry in the second config clobbered the one in the first config. So while the channel was referenced by both an exporter and importer in the config on-disk, it was not in-memory. The collector lacks any support for dumping/reporting its effective configuration after include-file and multiple config merging, so this was harder to diagnose than you'd expect.

Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 23.04
Release:	23.04
Codename:	lunar
$  docker --version
Docker version 24.0.5, build ced0996
$ uname -prmo
6.2.0-36-generic x86_64 x86_64 GNU/Linux

Additional context

• Define new component type "connector" #6577

[mx-psi]

Thanks for filing this issue @ringerc, I think this is a duplicate of #8721, so I would suggest closing this in favor of that one, but your comment about stating the specific pipeline seems worth commenting over there or having a separate issue.

[djaglowski]

I agree with @mx-psi that this is a duplicate. I'm closing this one but please reopen if there's something distinct about this issue which we've missed.

[ringerc]

Agreed. I missed that issue when searching, thanks for finding it.