Issue with "Exec" source

[ilcylic]

I'm trying to read some NATS data, but the system that's generating it wants authentication, so I'm using the "Exec" source to run a program that writes NATS records to STDOUT.

My relevant Vector config sections are:

[sources.exec_relayer_nats]
  type = "exec"
  mode = "streaming"
  framing.method = "newline_delimited"
  framing.newline_delimited.max_length = 102400
  include_stderr = true
  decoding.codec = "json"
  command = [ "/usr/bin/nats", "sub", "--creds", "/etc/vector/relayer_nats.creds", "-s", "nats://westcoast.nats.backend.aurora.dev:4222i", "-r", "writelog" ]

[transforms.parsed_relayer_nats]
  inputs = ["exec_relayer_nats"]
  type = "remap"
  source = '''
  ., err = parse_json(.message)
  if !exists(.source) {
      .source = "nats"
  }
  '''

And the log messages I'm getting that describe the error are:

2022-01-17T21:39:07.313653Z DEBUG source{component_kind="source" component_id=exec_relayer_nats component_type=exec component_name=exec_relayer_nats}: vector::sources::exec: Restarting streaming process.
2022-01-17T21:39:07.313676Z DEBUG source{component_kind="source" component_id=exec_relayer_nats component_type=exec component_name=exec_relayer_nats}: vector::sources::exec: Starting command run.
2022-01-17T21:39:07.313843Z DEBUG vector::sources::exec: Start capturing stdout command output.
2022-01-17T21:39:07.313854Z DEBUG vector::sources::exec: Start capturing stderr command output.
2022-01-17T21:39:07.329609Z TRACE vector::codecs::framing::character_delimited: Decoding the frame. bytes_proccesed=113
2022-01-17T21:39:07.329627Z  WARN vector::internal_events::decoder: Internal log [Failed deserializing frame.] is being rate limited.
2022-01-17T21:39:07.329633Z DEBUG vector::sources::exec: Finished capturing stderr command output.
2022-01-17T21:39:07.330183Z DEBUG vector::sources::exec: Finished capturing stdout command output.
2022-01-17T21:39:07.330204Z TRACE source{component_kind="source" component_id=exec_relayer_nats component_type=exec component_name=exec_relayer_nats}: vector::internal_events::exec: Executed command. command=/usr/bin/nats sub --creds /etc/vector/relayer_nats.creds -s nats://westcoast.nats.backend.aurora.dev:4222i -r writelog exit_status=1 elapsed_millis=16
2022-01-17T21:39:07.330220Z DEBUG source{component_kind="source" component_id=exec_relayer_nats component_type=exec component_name=exec_relayer_nats}: vector::sources::exec: Finished command run.
2022-01-17T21:39:07.330228Z  WARN source{component_kind="source" component_id=exec_relayer_nats component_type=exec component_name=exec_relayer_nats}: vector::sources::exec: Streaming process ended before shutdown.

The command runs on the CLI, and I can run its output through awk and the like. I'm not sure what I'm doing worng, unless I should be using the "Stdin" source type, but I wasn't sure how to configure that.

Alternatively, if there's a way to use the NATS source module directly with authentication, I'd be happy to use that instead.

Thanks!

[ilcylic]

Oops, I found the problem. Typo in my connection string. If I knew how to remove a question I would delete this.

[spencergilbert]

Nice! You'll be interested in #10688 which should be adding authentication to both the source and sink for NATS

[ilcylic]

Yes, I am in fact very very interested in that. :D Thanks for the heads up!