Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate Connector Usage #8721

Open
arielvalentin opened this issue Oct 22, 2023 · 5 comments · May be fixed by #12488
Open

Validate Connector Usage #8721

arielvalentin opened this issue Oct 22, 2023 · 5 comments · May be fixed by #12488
Labels
area:config area:connector bug Something isn't working

Comments

@arielvalentin
Copy link

Is your feature request related to a problem? Please describe.

I expected the configuration validator to find pipeline issues related to invalid usage of connector components, however the collector fails to start due to invalid usage and exits.

Here is an example that defines a count connector and uses it as a receiver in a metrics pipeline:

---
receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  logging:
connectors:
  count:

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      exporters: [logging]
    metrics/out:
      receivers: [count]
      exporters: [logging]

The config.yaml passes validation:

otel-validations(:|✔) %
🤘 docker run -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.87.0 validate --config=/etc/otelcol-contrib/config.yaml

otel-validations(:|✔) %
🤘 echo $?
0

However upon using this configuration, the collector reports an error and exits:

otel-validations(:|✔) %
🤘 docker run -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.87.0
2023-10-22T21:05:36.612Z	info	[email protected]/telemetry.go:84	Setting up own telemetry...
2023-10-22T21:05:36.612Z	info	[email protected]/telemetry.go:201	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
Error: failed to build pipelines: connector "count" used as receiver in metrics pipeline but not used in any supported exporter pipeline
2023/10/22 21:05:36 collector server run finished with error: failed to build pipelines: connector "count" used as receiver in metrics pipeline but not used in any supported exporter pipeline

otel-validations(:|✔) %
🤘 echo $?
1

 
Describe the solution you'd like

I expect the validator to return an error
@mx-psi
Copy link
Member

mx-psi commented Oct 23, 2023

cc @djaglowski

@djaglowski
Copy link
Member

This problem is not specific to connectors. The validate command currently does not consider supported data types for any class of component.

That said, I agree there are additional configuration errors, such as this one, which we could surface within the command if we would try to instantiate (but not start) a service.

@mx-psi mx-psi mentioned this issue Nov 14, 2023
Closed
@djaglowski djaglowski mentioned this issue Nov 14, 2023
Merged
@ringerc
Copy link

ringerc commented Nov 15, 2023

I raised this in #8866 after missing this issue in a search (somehow).

As I noted in that issue, it'd also be extremely helpful if the connector error mentioned the specific pipeline name in its complaint, e.g logs/out instead of just logs, like this:

Error: failed to build pipelines: connector "forward/test" used as receiver in "logs/out" pipeline but not used in any supported exporter pipeline

Currently it doesn't name the actual pipeline, so in complex configurations it can be tricky to tell what exactly it's complaining about.

@robincw-gr
Copy link

robincw-gr commented Dec 13, 2024 •
edited
Loading

Any eyes on this?
Shouldn't this have the label bug, because the validate command should definitely fail if the config prevents the collector from starting. Fixing is not an enhancement

@mx-psi mx-psi added bug Something isn't working and removed enhancement New feature or request labels Dec 17, 2024
@mx-psi
Copy link
Member

mx-psi commented Dec 17, 2024

@robincw-gr I don't think there is anyone working on this at the moment. In general there is no way to statically validate all things that would prevent the Collector from starting (think for example about a server using a port that is not available or some permissions issue). But I agree this should be considered a bug since it is statically verifiable.

@sudiptob2 sudiptob2 mentioned this issue Feb 17, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 24, 2025
@sudiptob2 @bogdandrutu
Add pipeline ID to the error message for unused connectors. (#12410)
18e18b2 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
As mentioned in this
#8721 (comment),
the error message for unused connectors currently lacks specific
pipeline names, making debugging more difficult.

This PR enhances the error message by including pipeline names in the
`[signal/name]` format, consistent with how they appear in
`config.yaml`. This provides a better context for identifying
misconfigurations.

<!-- Issue number if applicable -->
#### Link to tracking issue
Related to #8721 

<!--Describe what testing was performed and which tests were added.-->
#### Testing
A few scenarios and example output are given below. I will do additional
testing and add unit tests if necessary.

**1. Used as a receiver but not used as an exporter with 1 signal**
<details>
  <summary><strong>config.yaml</strong></summary>

```yaml
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  debug:
connectors:
  forward:
service:
  pipelines:
    logs/in:
      receivers: [otlp]
      processors: []
      exporters: [debug]
    logs/out:
      receivers: [forward]
      processors: []
      exporters: [debug]
```
</details>

Main Branch Output:
```
Error: failed to build pipelines: connector "forward" used as receiver in logs pipeline but not used in any supported exporter pipeline
```
Proposed Output:
```
Error: failed to build pipelines: connector "forward" used as receiver in [logs/out] pipeline but not used in any supported exporter pipeline
```

**2. Plain**
<details>
  <summary><strong>config.yaml</strong></summary>

```yaml
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  debug:
connectors:
  forward:
service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ ]
      exporters: [ forward ]
    metrics:
      receivers: [ forward ]
      processors: [ ]
      exporters: [ debug ]
```
</details>

Main Branch Output:
```
Error: failed to build pipelines: connector "forward" used as exporter in traces pipeline but not used in any supported receiver pipeline
```
Proposed Output:
```
Error: failed to build pipelines: connector "forward" used as exporter in [traces] pipeline but not used in any supported receiver pipeline
```
**3. Multiple pipeline**
<details>
  <summary><strong>config.yaml</strong></summary>

```yaml
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  debug:
connectors:
  forward:
service:
  pipelines:
    logs/in:
      receivers: [otlp]
      processors: []
      exporters: [forward]
    logs/in2:
      receivers: [ otlp ]
      processors: [ ]
      exporters: [ forward ]
    logs/out:
      receivers: [otlp]
      processors: []
      exporters: [debug]
    traces:
      receivers: [ otlp ]
      processors: [ ]
      exporters: [ forward ]
    metrics:
      receivers: [ forward ]
      processors: [ ]
      exporters: [ debug ]
```
</details>

Main Branch Output:
```
Error: failed to build pipelines: connector "forward" used as exporter in logs pipeline but not used in any supported receiver pipeline
```
Proposed Output:
```
Error: failed to build pipelines: connector "forward" used as exporter in [logs/in2 logs/in] pipeline but not used in any supported receiver pipeline

```

---------

Co-authored-by: Bogdan Drutu <[email protected]>
@sudiptob2 sudiptob2 linked a pull request Feb 25, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:config area:connector bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enhance config validation by Instantiating service during dryrun. sudiptob2/opentelemetry-collector
5 participants
@arielvalentin @ringerc @djaglowski @mx-psi @robincw-gr