prometheus_remote_write sinks stuck if one sink is unavailable

[hedard]

Hi!

Problem

I'm setting up a simple configuration where metrics from node exporter are pushed to victoriametrics. One is primary and the second one is backup. I have an issue that metrics are not pushed to backup if primary is down.
When I stop vict1, metrics stop going into vict2 too.
I expect these sinks to work separatelly and independent from each other.
I'm using tlast_over_time function to check incoming metrics on backup vm, but last TS for backup vm is when primary was up.
Debug log doesn't say much

Configuration

sources:
  local_node_exporter:
    endpoints:
    - http://127.0.0.1:9429/metrics
    instance_tag: instance
    scrape_interval_secs: 5
    scrape_timeout_secs: 4
    type: prometheus_scrape
transforms:
  local_node_exporter_transform:
    drop_on_abort: true
    drop_on_error: true
    inputs:
    - local_node_exporter
    source: 'metric_port = split!(.tags.instance, pattern: ":")[1]

      .tags.instance = "127.0.0.1:" + metric_port

      .tags.domain = "test_vict"'
    type: remap

sinks:
  vict1:
    endpoint: http://127.0.0.1:8431/api/v1/write
    healthcheck:
      enabled: false
    inputs:
    - local_node_exporter_transform
    type: prometheus_remote_write
  vict2:
    endpoint: http://127.0.0.1:8432/api/v1/write
    healthcheck:
      enabled: false
    inputs:
    - local_node_exporter_transform
    type: prometheus_remote_write

Version
I'm using docker image timberio/vector:0.39.0-alpine

Debug Output

2024-10-25T10:08:43.907375Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.00000004839025547411212
2024-10-25T10:08:48.907572Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.00000005487706745736119
2024-10-25T10:08:53.906891Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.000000039692361454338906
2024-10-25T10:08:58.907082Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.00000004662761927132699
2024-10-25T10:09:03.907504Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.000000037779958012585306
2024-10-25T10:09:05.269205Z DEBUG sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://127.0.0.1:8431/api/v1/write method=POST version=HTTP/1.1 headers={"x-prometheus-remote-write-version": "0.1.0", "content-type": "application/x-protobuf", "content-encoding": "snappy", "user-agent": "Vector/0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[11959 bytes]
2024-10-25T10:09:05.269297Z DEBUG sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: hyper::client::connect::http: connecting to 127.0.0.1:8431
2024-10-25T10:09:05.269635Z  WARN sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: vector::internal_events::http_client: Internal log [HTTP error.] has been suppressed 1 times.
2024-10-25T10:09:05.269654Z  WARN sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: tcp connect error: Connection refused (os error 111) error_type="request_failed" stage="processing" internal_log_rate_limit=true
2024-10-25T10:09:05.269721Z  WARN sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] has been suppressed 1 times.
2024-10-25T10:09:05.269740Z  WARN sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111) internal_log_rate_limit=true
2024-10-25T10:09:05.269759Z DEBUG sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}: vector::sinks::util::retries: Retrying request. delay_ms=781
2024-10-25T10:09:06.052105Z DEBUG sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://127.0.0.1:8431/api/v1/write method=POST version=HTTP/1.1 headers={"x-prometheus-remote-write-version": "0.1.0", "content-type": "application/x-protobuf", "content-encoding": "snappy", "user-agent": "Vector/0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[11959 bytes]
2024-10-25T10:09:06.052185Z DEBUG sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: hyper::client::connect::http: connecting to 127.0.0.1:8431
2024-10-25T10:09:06.052497Z  WARN sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}:http: vector::internal_events::http_client: Internal log [HTTP error.] is being suppressed to avoid flooding.
2024-10-25T10:09:06.052555Z  WARN sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.
2024-10-25T10:09:06.052572Z DEBUG sink{component_kind="sink" component_id=vict1 component_type=prometheus_remote_write}:request{request_id=1}: vector::sinks::util::retries: Retrying request. delay_ms=18577
2024-10-25T10:09:08.907050Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.000000028800265357621418
2024-10-25T10:09:13.907465Z DEBUG sink{component_kind="sink" component_id=vict2 component_type=prometheus_remote_write}: vector::utilization: utilization=0.0000000401369387910745

[jszwedko]

Hi @hedard !

This is expected behavior (there are more details on https://vector.dev/docs/about/under-the-hood/architecture/buffering-model/). To avoid one sink blocking, you can configure the buffer to instead drop data rather than blocking by adding:

buffer:
  on_full: drop_newest

on the sink you don't want to block if down. This, of course, means that data going to that sink will be dropped.