Timestamp Mismatch when Ingesting logs to Loki

[shiblisec]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I have been gathering data from multiple sources and then using vector to transform the data, I have remaped the timestamp fields for all my data sources, so now only one key identifier exisits timestamp for all my data sources. When I am ingesting the transformed logs into loki using the following sink configuration.

  loki:
     type: loki
     inputs: ["source1", "source2", "source3"]
     endpoint: http://localhost:3100
     encoding:
       codec: json
     labels:
      "source": "vector"

I am getting the following results on Grafana, as we can observe the Grafana is showing a different timestamp while the actual timestamp on the log mentions a different time.

Configuration

No response

Version

vector 0.40.0 (x86_64-apple-darwin 1167aa9 2024-07-29 15:08:44.028365803)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @shiblisec ,

Can you share the remap configuration? My guess is that the your timestamp field hasn't been parsed into an actual timestamp and is instead a string timestamp. If that is the case, you'd want to use the parse_timestamp VRL function.

[shiblisec]

Hi @jszwedko ,

Yep, that was the problem the timestamp was string instead of the timestamp datatype, did the type formatting and its working fine now.

Closing this as resolved.

[LinTechSo]

Hi
same issue here
assuming below log line

    type: demo_logs
    format: shuffle
    interval: 1
    sequence: true
    lines:
      - '{
          "level": "INFO",
          "timestamp": 1732008699728000000,
          "version_code": 223,
          "version_name": "5.2.3"
          }'

and this is my remap for example:

      if exists(.timestamp) {
          if is_integer(.timestamp) {
              .log.timestamp = to_unix_timestamp(from_unix_timestamp!(.timestamp, unit: "nanoseconds"), unit: "nanoseconds")
          } else if is_float(.timestamp) {
              .timestamp = to_unix_timestamp(from_unix_timestamp!(to_int!(.timestamp), unit: "nanoseconds"), unit: "nanoseconds")
          } else if is_string(.timestamp) {
              .timestamp = to_unix_timestamp(parse_timestamp!(.timestamp, "%+"),unit: "nanoseconds")
          } else {
              .timestamp = now()
          }
      }

still Grafana is showing a different timestamp while the actual timestamp on the log mentions a different time.
can you help me figure this out

[jszwedko]

I think you don't want to use to_unix_timestamp. Instead, just use the result of from_unix_timestamp or parse_timestamp since the loki sink expects timestamps to be of type timestamp rather than type integer.