[exporter/prometheus] Content-Type header for Prometheus 3.0.0

[jcpunk]

Component(s)

exporter/prometheus

Is your feature request related to a problem? Please describe.

Prometheus 3.0.0 has the option (by default) to be far more strict about what content it scrapes. This can cause unexpected scrape failures for the prometheus exporter.

Describe the solution you'd like

Set the header "to the correct value". It should probably be aware if "enable_open_metrics" is set.

Describe alternatives you've considered

If a scrape target is not providing the correct Content-Type header the fallback protocol can be specified using the fallback_scrape_protocol parameter.

Additional context

From : https://prometheus.io/docs/prometheus/3.0/migration/

Prometheus v3 is more strict concerning the Content-Type header received when scraping. Prometheus v2 would default to the standard Prometheus text protocol if the target being scraped did not specify a Content-Type header or if the header was unparsable or unrecognised. This could lead to incorrect data being parsed in the scrape. Prometheus v3 will now fail the scrape in such cases.

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

👍

I think it's always valid to add the header. Even if the backend is a Prometheus < 3.0, adding it has no downsides.

[ArthurSens]

/label -needs-triage help-wanted

[namco1992]

According to the client_golang used to handle the prometheus scraping, it negotiates the content type and sets the header properly:

https://github.com/prometheus/client_golang/blob/dcf944d6035ee45bcb9165ec081541da6b5b1ead/prometheus/promhttp/http.go#L187-L193

May I ask what else needs to be done here?

[dashpole]

@ArthurSens do we just need to wait for the client_golang dependency to be bumped?

[jcpunk]

do we just need to wait for the client_golang dependency to be bumped?

That was my understanding when I opened the ticket.

[ArthurSens]

I'm not sure actually, looking at git blame I can see that code is there for a long time already:

And the PR that moved the line adding content type header just moved that line from one place to another:
prometheus/client_golang@f08b10c#diff-8dc7897d4d9d5bf1c91a5fe1c931db556eabc1a6458dfce5ddd087b0bc5eec78

Without looking deeper into the code I can't tell what needs to be done, but it doesn't seem to be a client_golang bump that would fix this

[namco1992]

The current client_golang version used in the exporter is 1.20.5 which is the latest release https://github.com/prometheus/client_golang/releases/tag/v1.20.5:

opentelemetry-collector-contrib/exporter/prometheusexporter/go.mod

Line 10 in 7e3d003

github.com/prometheus/client_golang v1.20.5

CMIIW but we don't need to do any extra work to support the content-type header.