Problems with nested fields

[ottramst]

Hello,

Thanks for the awesome project!

I have a slight problem with using VRL with nested fields (mainly renaming).

Configuration

  sources:
    var_log_kubelet:
      type: file
      include:
        - /var/log/kubelet/kubelet.ERROR
        - /var/log/kubelet/kubelet.WARNING
  transforms:
    kubelet_parse:
      type: remap
      inputs:
        - var_log_kubelet
      source: |
        . = parse_klog!(.message)
        .@timestamp = del(.timestamp)
    aws_ec2_metadata:
      type: aws_ec2_metadata
      inputs:
        - kubelet_parse
      fields:
        - ami-id
        - availability-zone
        - instance-id
        - local-hostname
        - local-ipv4
        - public-hostname
        - public-ipv4
        - region
    ecs_transform:
      type: remap
      inputs:
        - aws_ec2_metadata
      source: |
        .cloud.region = del(.region)
        .cloud.availability_zone = del(.availability-zone)
  sinks:
    console:
      type: console
      inputs:
        - ecs_transform
      target: stdout
      encoding: json

Please note the transforms.ecs_transform.source configuration.
When the source contains only cloud.region = del(.region) then everything works and the field is renamed successfully.

Problems start when I start adding other renames like .cloud.availability_zone = del(.availability-zone), which starts producing errors:

{"timestamp":"2021-12-07T15:11:30.771645Z","level":"INFO","message":"Log level is enabled.","level":"\"vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info\"","target":"vector::app"}
{"timestamp":"2021-12-07T15:11:30.771754Z","level":"INFO","message":"Loading configs.","paths":"[\"/etc/vector\"]","target":"vector::app"}
{"timestamp":"2021-12-07T15:11:30.797327Z","level":"ERROR","message":"Configuration error.","error":"Transform \"ecs_transform\": \n\u001b[0m\u001b[1m\u001b[38;5;9merror[E701]\u001b[0m\u001b[1m: call to undefined variable\u001b[0m\n  \u001b[0m\u001b[34m┌─\u001b[0m :2:46\n  \u001b[0m\u001b[34m│\u001b[0m\n\u001b[0m\u001b[34m2\u001b[0m \u001b[0m\u001b[34m│\u001b[0m .cloud.availability_zone = del(.availability-\u001b[0m\u001b[31mzone\u001b[0m)\n  \u001b[0m\u001b[34m│\u001b[0m                                              \u001b[0m\u001b[31m^^^^\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m                                              \u001b[0m\u001b[31m│\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m                                              \u001b[0m\u001b[31mundefined variable\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m                                              \u001b[0m\u001b[34mdid you mean \"true\"?\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m\n  \u001b[0m\u001b[34m=\u001b[0m see language documentation at https://vrl.dev\n\n\u001b[0m\u001b[1m\u001b[38;5;9merror[E630]\u001b[0m\u001b[1m: fallible argument\u001b[0m\n  \u001b[0m\u001b[34m┌─\u001b[0m :2:32\n  \u001b[0m\u001b[34m│\u001b[0m\n\u001b[0m\u001b[34m2\u001b[0m \u001b[0m\u001b[34m│\u001b[0m .cloud.availability_zone = del(\u001b[0m\u001b[31m.availability-zone\u001b[0m)\n  \u001b[0m\u001b[34m│\u001b[0m                                \u001b[0m\u001b[31m^^^^^^^^^^^^^^^^^^\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m                                \u001b[0m\u001b[31m│\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m                                \u001b[0m\u001b[31mthis expression can fail\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m                                \u001b[0m\u001b[34mhandle the error before passing it in as an argument\u001b[0m\n  \u001b[0m\u001b[34m│\u001b[0m\n  \u001b[0m\u001b[34m=\u001b[0m see documentation about error handling at https://errors.vrl.dev/#handling\n  \u001b[0m\u001b[34m=\u001b[0m see language documentation at https://vrl.dev\n","target":"vector::topology"}

Also can confirm that if the source features only .cloud.availability_zone = del(.availability-zone) I get the same exact error.

Not sure where to go from here, any help and/or guidance will be greatly appreciated.

Thank you!

[jszwedko]

Hi @ottramst !

I think you need to quote the access the the availability-zone field since it has a special character (-). This would look like ."availability-zone". This is needed because VRL also uses - for subtraction and so otherwise interprets what you have as .availability - zone where zone is expected to be a defined variable.

https://vector.dev/docs/reference/vrl/expressions/#path has docs about the allowed characters in a field name. Any other ones require quoting the field.

Let me know if this doesn't help!

[ottramst]

This was exactly it. Thank you!