Elasticsearch custom index pattern with dates

[monwolf]

Hi,

I'm trying to setup vector with Opensearch replacing my current fluentd setup. I have this script to process the logs:

      timest = "-%Y.%m.%d"
      if logtype == "my_log" {
          timest = "-%Y.%m"
      }
      if includes(["content1", "content2"], svc ) {
        .index, err = "ae-microservices-" + .cluster + "-" + .environment + "-" + logtype + "-" + svc + timest
      } else {
        .index, err = "ae-microservices-" + .cluster + "-" + .environment + "-" + logtype + timest
      }

With this sink:

sinks:
  elastic:
    type: elasticsearch
    inputs:
      - normalize_logs
    api_version: auto
    compression: gzip
    endpoints:
      - https://elastic.example.com
    mode: bulk
    auth:
      strategy: basic
      user: vector
      password: xxxxxx
    bulk:
      index: "{{ .index }}"

But I'm getting the error:

2023-04-24T08:18:28.222602Z ERROR sink{component_kind="sink" component_id=elastic component_type=elasticsearch component_name=elastic}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: invalid_index_name_exception, reason: Invalid index name [microservices-pre-dev-my_log-%Y.%m], must be lowercase" internal_log_rate_limit=true

I have mixed index patterns YYYY.mm or YYYY.mm.dd depending on the content of the log.

Is there any way to achieve it with vector?

[bruceg]

The index name handling can only interpolate either a field name (for example .index here) or a timestamp format (the %Y within the field), and not both at the same time. You would need to do the timestamp interpolation in VRL with something like timest = format_timestamp(.timestamp, timest).

[monwolf]

Thanks! it worked fine :)