Using journald source while running vector in docker

[pySilver]

Is that possible? Looks like not :(

$ cat vector.toml
[sources.in]
  type = "journald" # required

[sinks.out]
  # General
  type = "console" # required
  inputs = ["in"] # required

  # Encoding
  encoding.codec = "json" # required

output:

$ docker run --rm --user=root \
  -v $PWD/vector.toml:/etc/vector/vector.toml:ro \
  -p 9000:9000 \
  timberio/vector:latest-alpine
Jun 19 16:26:50.518  INFO vector: Log level "info" is enabled.
Jun 19 16:26:50.518  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Jun 19 16:26:50.522  INFO vector: Vector is starting. version="0.9.1" git_version="v0.9.2" released="Tue, 02 Jun 2020 21:45:49 +0000" arch="x86_64"
Jun 19 16:26:50.522 ERROR vector::topology: Configuration error: Source "in": data_dir "/var/lib/vector/" does not exist

So the only way is to have some vector running on host that simply ingest journald and listens on 9000 so another more complex, dynamically configured vector can use it as a source. Is there any drawbacks?

Btw, elstic's Journalbeat works fine with this. So another approach can be use both of them..

[bruceg]

The journald source writes a checkpoint file recording the last record it handled in order to prevent duplicating records. By default, vector is configured to write those files in /var/lib/vector. If that directory does not exist, vector will not create it. Simply ensure that the directory exists and is writable to vector and this will work. I know of no reason vector's journald source will not work within docker, as long as journalctl has access to the journal files.

[pySilver]

@bruceg thanks for explanation. Just to be clear /var/lib/vector is the path that should be writable by vector that runs in docker. That way Vector will be able to read journald logs from host, correct?

[bruceg]

Partly right. /var/lib/vector must be writable so that vector's journald source can write checkpoints there. This is true both within and without docker. This may be configured with the data_dir option, either globally or within the journald source. This has nothing to do with its ability to read logs, only the checkpoints to avoid re-reading the same logs the next time it is started.

[binarylogic]

@bruceg are there action items to make it obvious users need to do this? Are we logging this information? Is this a check in the ‘validate’ sub command? I’d like to open issues for anything like that.

[bruceg]

The error message could be expanded to include a note about it being required for checkpoints. This is a little tricky because it's a shared routine between the file and journald sources that just returns a Result. This directory is set up during the config build phase, so I think validate should check it.

[pySilver]

@binarylogic I've tried once again and it does not work. Here is the output and configurations:

Jun 22 15:10:33.468  INFO vector: Log level "info" is enabled.
Jun 22 15:10:33.469  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Jun 22 15:10:33.473  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-256-g5e00483" released="Sat, 20 Jun 2020 14:49:03 +0000" arch="x86_64"
Jun 22 15:10:33.482  INFO vector::sources::docker: Listening docker events
Jun 22 15:10:33.538 ERROR vector::topology: Configuration error: Source "journald": journalctl failed to execute: No such file or directory (os error 2)

vector.toml

[sources.journald]
  type = "journald"

[sinks.loki-journald]
  type = "loki"
  inputs = ["journald"]
  endpoint = "http://loki.service.consul:3100"
  healthcheck = true
  remove_timestamp = true
  encoding.codec = "json"
  labels.input = "journald"

shared volumes:

volumes = [
          "secrets/vector.toml:/etc/vector/vector.toml:ro",
          "/opt/vector:/var/lib/vector",
          "/var/run/docker.sock:/var/run/docker.sock",
          "/:/hostfs:ro",
        ]

[bruceg]

The journald source uses the journalctl program as its interface to retrieve logs from journald. As such, it must be installed in your docker image for this source to work. Please see the journald source documentation for more details.

[pySilver]

This is why I've opened this ticker in a first place :) Since vector offers official docker image it is expected that all features are available no matter how you run it. Some sign in docs that journald won't work with official Vector docker image would suffice for now. I'd be happy to help to make it work, but unfortunately I'm not familiar with golang (yet).

Also this is why I've mentioned journalbeat from elastic stack. They got it covered – one can run journalbeat in container and read journald logs from host & push to elasticsearch.

[bruceg]

Thanks for the note. @Hoverbear is working on new packaging and will include journalctl in the next stable release (ref #2824).

[Hoverbear]

Checking this for #3022

[Hoverbear]

Please note alpine does not package systemd. Once we add this. You'll need to use the debian images for this.

[Hoverbear]

#3129 adds journald to the debian image. -- Sorry I didn't get this earlier. I thought I had it covered but I had the wrong image changed.

[Hoverbear]

New nightly packages and the upcoming 0.10.0 releases of the debian image have this in it now. :)

hoverbear@bearbuntu:/git/timberio/vector$ cat vector.toml 
[sources.in]
  type = "journald"

[sinks.out]
  type = "console"
  inputs = ["in"]
  encoding.codec = "json"

hoverbear@bearbuntu:/git/timberio/vector$ docker run --rm --user=root   -v $PWD/vector.toml:/etc/vector/vector.toml:ro   -p 9000:9000   --volume $(pwd)/var:/var/lib/vector timberio/vector:latest-debian
Jul 20 22:27:18.440  INFO vector: Log level "info" is enabled.
Jul 20 22:27:18.441  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Jul 20 22:27:18.443  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-388-g89db41f" released="Mon, 20 Jul 2020 19:34:22 +0000" arch="x86_64"
Jul 20 22:27:18.443  INFO vector::topology: Running healthchecks.
Jul 20 22:27:18.443  INFO vector::topology: Starting source "in"
Jul 20 22:27:18.443  INFO vector::topology: Starting sink "out"
Jul 20 22:27:18.443  INFO vector::topology::builder: Healthcheck: Passed.
Jul 20 22:27:18.444  INFO source{name=in type=journald}: vector::sources::journald: Starting journald server.

^CJul 20 22:27:22.096  INFO vector: Shutting down.