How to call API and consume the output using exec

[satscreate]

Any example script for exec source, which calls an API every N minute and collects the response and transform.?
Tried,

sources:
  my_source_id:
    type: exec
    command: ["curl","-k","https://gorest.co.in/public/v2/users"]
sinks:
  stdout:
    type: console
    inputs:
      - my_source_id
    target: stdout
    encoding:
      codec: json

Any examples?

[jszwedko]

Hi @satscreate ,

If you want to call it on an interval, you should use mode.scheduled with scheduled.exec_interval_secs like:

sources:
  my_source_id:
    type: exec
    command: ["curl","-k","https://gorest.co.in/public/v2/users"]
    mode: scheduled
    scheduled:
      exec_interval_secs: 600 # five minutes

Otherwise, what you have seems reasonable to me.

[satscreate]

How can i pass the user credentials in the form of Basic Auth header form, using function encode_base64() in command?

[jszwedko]

@satscreate I'd recommend writing a shell script that Vector can run that uses base64 to encode the credentials. Or pre-encode them and add them here directly by adding "-H", "Authorization ...." to the command. It isn't possible to use the encode_base64 VRL function in this context.

[satscreate]

Thanks @jszwedko , much more interested in if it does support keep & serve creds on the fly from docker secrets inside exec command!!!

[satscreate]

But why not

@satscreate I'd recommend writing a shell script that Vector can run that uses base64 to encode the credentials. Or pre-encode them and add them here directly by adding "-H", "Authorization ...." to the command. It isn't possible to use the encode_base64 VRL function in this context.

Hi @jszwedko, But why not ,
command: ["curl","-k","https://gorest.co.in/public/v2/users","--header","Authorization: Basic ","cat /run/secrets/my_external_secret"]

also i have my secret inside the docker instance where vector is loaded happily. If this "cat /run/secrets/my_external_secret" statement considered as just text representation then how come "curl" considered as a command? Much appreciated for your response..

[jszwedko]

Hi @satscreate !

You should be able to do something like that like:

command: ["bash", "-c", "curl -k https://gorest.co.in/public/v2/users --header \"Authorization: Basic $(cat /run/secrets/my_external_secret)\""]