prometheus_scrape source should have request headers configurable

[alloran]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

For example, Hashicorp Vault requires Accept: prometheus/telemetry or Accept: application/openmetrics-text] headers to be present in request to return metrics in Prometheus format.

https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus

Attempted Solutions

No response

Proposal

prometheus_scrape source should have request headers configurable, as http_client already do.

References

No response

Version

0.35.0

[hhromic]

While I don't oppose the idea of configurable headers, I think that the prometheus_scrape source at least should mirror the behaviour of Prometheus' scrape system by default.

In particular, Prometheus defines the following list of scrape protocols:

PrometheusProto      ScrapeProtocol = "PrometheusProto"
PrometheusText0_0_4  ScrapeProtocol = "PrometheusText0.0.4"
OpenMetricsText0_0_1 ScrapeProtocol = "OpenMetricsText0.0.1"
OpenMetricsText1_0_0 ScrapeProtocol = "OpenMetricsText1.0.0"

Which map to the following Accept header values:

ScrapeProtocolsHeaders = map[ScrapeProtocol]string{
	PrometheusProto:      "application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited",
	PrometheusText0_0_4:  "text/plain;version=0.0.4",
	OpenMetricsText0_0_1: "application/openmetrics-text;version=0.0.1",
	OpenMetricsText1_0_0: "application/openmetrics-text;version=1.0.0",
}

And are sent to the scraped service in the following order:

DefaultScrapeProtocols = []ScrapeProtocol{
	OpenMetricsText1_0_0,
	OpenMetricsText0_0_1,
	PrometheusText0_0_4,
}

Which results in the following Accept header being sent by default by Prometheus:

Accept: application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=0.0.4;q=0.3,*/*;q=0.2

(Note that I couldn't find any references to prometheus/telemetry anywhere in the Prometheus source code).

If the same logic is implemented in the prometheus_scrape source, then Vector would work out of the box with Vault and probably other strict exporters like it does for Prometheus (e.g. due to the OpenMetrics accept values).

For reference, this is a real example of a Prometheus request to an exporter service:

GET /metrics HTTP/1.1
Host: myservice.local:9091
User-Agent: Prometheus/2.49.1
Accept: application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=0.0.4;q=0.3,*/*;q=0.2
Accept-Encoding: gzip
X-Prometheus-Scrape-Timeout-Seconds: 10

EDIT: Technically speaking, the prometheus_scrape source should actually implement all the advertised protocols properly. I'm not an expert on the differences between the OpenMetrics and the original Prometheus protocols. I understand they are very similar but not sure if exactly the same. So I guess care must be taken if implemented.

The same care should be taken if the Accept header is made freely configurable. The user cannot expect to customise Accept: application/json and the prometheus_scrape source to magically understand JSON metrics 😄. This is part of why I'm not so convinced of making headers fully configurable for this source.

[jszwedko]

Good thought @hhromic . It seems like the prometheus_scrape source should set the correct Accept header for the "known formats" it is capable of parsing.