Unclear about timestamps

[trickster]

I am trying to use file sink, where do these values come from? I tried with timezone="America/Los_Angeles at top in my pipeline. But it doesn't seem to honor it, default to UTC I think. At the time of writing, LA time is 11:30PM Jun 30, but the file is created with July 1 using the below

[sinks.local_file]
  type = "file"
  path = "/data/Vector-%Y-%m-%d.json"
  compression = "none"
  encoding.codec = "ndjson"

Minimal reproducible example

# cat vector_sample.toml
[sources.in]
  type = "stdin"

[sinks.out]
  inputs = ["in"]
  type = "console"
  encoding.codec = "json"


# date  # LA time
Wed Jun 30 23:44:35 PDT 2021
#
# vector --config ./vector_sample.toml
Jun 30 23:44:38.940  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info"
Jun 30 23:44:38.940  INFO vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root.
Jun 30 23:44:38.940  INFO vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root.
Jun 30 23:44:38.941  INFO vector::app: Loading configs. path=[("vector_sample.toml", None)]
Jun 30 23:44:38.947  INFO vector::topology: Running healthchecks.
Jun 30 23:44:38.948  INFO vector::sources::stdin: Capturing STDIN.
Jun 30 23:44:38.948  INFO vector::topology: Starting source. name="in"
Jun 30 23:44:38.948  INFO vector::topology::builder: Healthcheck: Passed.
Jun 30 23:44:38.948  INFO vector::topology: Starting sink. name="out"
Jun 30 23:44:38.948  INFO vector: Vector has started. version="0.14.0" arch="x86_64" build_id="5f3a319 2021-06-03"
Jun 30 23:44:38.948  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
123
{"host":"xxx","message":"123","source_type":"stdin","timestamp":"2021-07-01T06:44:40.795367048Z"}
^CJun 30 23:44:42.021  INFO vector: Vector has stopped.
Jun 30 23:44:42.021  INFO source{component_kind="source" component_name=in component_type=stdin}: vector::sources::stdin: Finished sending.

[jszwedko]

Hi @trickster ,

Currently Vector converts all timestamps internally to UTC. timezone can be set in the config file, but this only controls how timestamps are parsed that are lacking a timezone.

I think what you want here is covered by #3333 . That would ensure the input timezone is recorded appropriately, which would then be used later when formatting the filename. I added a note there to make sure that part is handled correctly as well.

[trickster]

Hi @jszwedko

I found the documentation that has this information, thank you so much. I found out that by default every thing is UTC. However I found a workaround for this

[log_schema]
  timestamp_key = "timestamp_2"

[transforms.step1]
  type = "remap"
  inputs = ["step0"]
  source = '''
  .timestamp_2 = to_timestamp(to_int!(.timestamp) -/+ SECONDS) # original data has `timestamp` field which is UTC in unix epoch
  '''

The above changes will change the below file to the respective timezone (defined by SECONDS)

[sinks.local_file]
  type = "file"
  inputs = ["step1"]
  path = "/data-%Y-%m-%d.json.gz"
  compression = "none"
  encoding.codec = "ndjson"