Skip empty lines

[k-bx]

I have a rule where I want to log slow queries from my nginx into a log file. It looks like this:

sources:
  nginx_access:
    type: file
    include:
      - /var/log/nginx/access.log

transforms:
  nginx_access_t:
    inputs:
      - nginx_access
    type: "remap"
    source: |
            . = parse_regex(.message, r'(?P<remote_addr>[^ ]*) - - \[(?P<time_local>[^\]]*)\] "(?P<request>[^"]*)" (?P<status>[^ ]*) (?P<body_bytes_sent>[^ ]*) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)" (?P<request_time>[^ ]*) (?P<upstream_response_time>[^ ]*)') ?? {}
            .time_local = parse_timestamp(.time_local, "%d/%b/%Y:%H:%M:%S %z") ?? now()
            .request = parse_regex(.request, r'(?P<request_method>[^ ]*) (?P<request_uri>[^ ]*) (?P<request_protocol>[^ ]*)') ?? {}
            .status = parse_int(.status) ?? 0
            .body_bytes_sent = parse_int(.body_bytes_sent) ?? 0
            .request_time = parse_float(.request_time) ?? 0
            .upstream_response_time = parse_float(.upstream_response_time) ?? 0

  nginx_access_t_02_keep_slow_queries:
    inputs:
      - nginx_access_t
    type: "filter"
    condition: .upstream_response_time > 0.5 ?? true

  nginx_access_t_03_remove_empty_messages:
    inputs:
      - nginx_access_t_02_keep_slow_queries
    type: "filter"
    condition: .message != ""

sinks:
  out:
    type: file
    inputs:
      - nginx_access_t
    path: /var/log/nginx-access-slow.log
    encoding:
      codec: "text"

for nginx configured with:

log_format timed_combined '$remote_addr - $remote_user [$time_local] '
	'"$request" $status $body_bytes_sent '
	'"$http_referer" "$http_user_agent" '
	'$request_time $upstream_response_time $pipe';

access_log /var/log/nginx/access.log timed_combined;

Now, this works, however what I see is that vector keeps putting empty lines into my /var/log/ log file. How do I tell it to stop doing that?

[jorgehermo9]

Your sink input must be nginx_access_t_03_remove_empty_messages

sinks:
  out:
    type: file
    inputs:
      - nginx_access_t_03_remove_empty_messages
    path: /var/log/nginx-access-slow.log
    encoding:
      codec: "text"

Note that transforms does not mutate its input, so in order to use the transforms' output, you should specify their output id as a sink input

Also, you seem to manually parse nginx logs, are you aware of the vrl parse_nginx_log function? https://vector.dev/docs/reference/vrl/functions/#parse_nginx_log

Maybe it does not work with your custom log formar,so I would recommend if possible to use the default nginx log format and the native vrl function

Also, I think that you are seeing empty lines because the text encoder uses the message field of the logs, which is not present in the events that are reaching your sink, as you are replacing the original object in the first line of your remap with another object without that field. You should specify a message field if you want to just display the message in the text encoder. If you want to have the parsed nginx log event, then using encoding.codec=json would do the job

@k-bx

[k-bx]

Hey, thanks. Solved by codec=json indeed as I didn't realize that codec=text will only output the .message part.

The reason I'm not using parse_nginx_log is because default nginx logs are useless for me as they don't contain the timing info. Please see the "nginx configured with" part of the message.

Anyways, thanks for looking into this. I now have fully working slow log solution for nginx!