Metadata containes an unknown parameter _type

[shahar-I-levy]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When i try to ship logs to opensearch, i get a 400 status code with the errors:

2024-08-06T20:36:39.376628Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=1}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_400" response=Response { status: 400, version: HTTP/1.1, headers: {"content-type": "application/json; charset=UTF-8"}, body: b"{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}" }
2024-08-06T20:36:39.376725Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}" internal_log_rate_limit=true
2024-08-06T20:36:39.376750Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2024-08-06T20:36:39.376772Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=81 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

Configuration

This is my config:

sources:
  docker_logs:
    type: docker_logs
    include_containers:
      - nginx

transforms:
  add_container_name:
    type: remap
    inputs:
      - docker_logs
    source: |
      .container_name = .docker.container_name

sinks:
  # console:
  #   type: console
  #   inputs:
  #     - add_container_name
  #   encoding:
  #     codec: json
  #   target: stdout
  opensearch:
    type: elasticsearch
    inputs:
      - add_container_name
    endpoints: ["http://opensearch:9200"]
    bulk:
      action: create
      index: "nginx-logs"
    healthcheck:
      enabled: true
    request:
      batch:
        size: 1000
      compression: gzip
      timeout_secs: 10

Version

0.41.0 / timberio/vector:nightly-debian 86e411c44a0c

Debug Output

2024-08-06T20:54:11.799815Z DEBUG source{component_kind="source" component_id=docker_logs component_type=docker_logs}: vector::internal_events::docker_logs: Received one container event. container_id=855716a5d13e34f7f0d264f45b57a86d4913578c9c6e5d8fb002f27a7443aee3 action=start
2024-08-06T20:54:11.799869Z DEBUG source{component_kind="source" component_id=docker_logs component_type=docker_logs}: bollard::docker: unix://2f7661722f72756e2f646f636b65722e736f636b/containers/855716a5d13e34f7f0d264f45b57a86d4913578c9c6e5d8fb002f27a7443aee3/logs?follow=true&stdout=true&stderr=true&since=1722977650&timestamps=true&tail=
2024-08-06T20:54:11.799879Z  INFO source{component_kind="source" component_id=docker_logs component_type=docker_logs}: vector::internal_events::docker_logs: Started watching for container logs. container_id=855716a5d13e34f7f0d264f45b57a86d4913578c9c6e5d8fb002f27a7443aee3
2024-08-06T20:54:12.597188Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}: tower::balance::p2c::service: updating from discover
2024-08-06T20:54:12.597249Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}: tower::buffer::worker: service.ready=true processing request
2024-08-06T20:54:12.597423Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://opensearch:9200/_bulk?timeout=10s method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "accept-encoding": "zstd,gzip,deflate,br", "user-agent": "Vector/0.41.0 (aarch64-unknown-linux-gnu 210ff09 2024-08-06 04:01:43.030775167)"} body=[93566 bytes]
2024-08-06T20:54:12.597548Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}:http: hyper::client::pool: reuse idle connection for ("http", opensearch:9200)
2024-08-06T20:54:12.598386Z DEBUG hyper::proto::h1::io: flushed 46336 bytes
2024-08-06T20:54:12.598622Z DEBUG hyper::proto::h1::io: flushed 47480 bytes
2024-08-06T20:54:12.601867Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-08-06T20:54:12.601912Z DEBUG hyper::proto::h1::conn: incoming body is content-length (152 bytes)
2024-08-06T20:54:12.601932Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-08-06T20:54:12.601979Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}:http: hyper::client::pool: pooling idle connection for ("http", opensearch:9200)
2024-08-06T20:54:12.602006Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}:http: vector::internal_events::http_client: HTTP response. status=400 Bad Request version=HTTP/1.1 headers={"content-type": "application/json; charset=UTF-8", "content-encoding": "gzip", "content-length": "152"} body=[152 bytes]
2024-08-06T20:54:12.602162Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_400" response=Response { status: 400, version: HTTP/1.1, headers: {"content-type": "application/json; charset=UTF-8"}, body: b"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" }
2024-08-06T20:54:12.602251Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] is being suppressed to avoid flooding.
2024-08-06T20:54:12.602289Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
2024-08-06T20:54:12.602313Z ERROR sink{component_kind="sink" component_id=opensearch component_type=elasticsearch}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

@shahar-I-levy I would try setting the api_version field on the sink: https://vector.dev/docs/reference/configuration/sinks/elasticsearch/#api_version

[jszwedko]

For OpenSearch I think you want to use v8.

[jszwedko]

Closing due to lack of response and because I'm pretty confident in my response, but feel free to drop a comment if it still isn't working for you!