removing multiple fields

[Wilderone]

Hi!
I want to remove few fields by some mask, but can't understand how to achieve this
For example, i have multiple fields like:

kubernetes.pod_annotations.checksum/result-backend-secret
kubernetes.pod_annotations.checksum/webserver-secret-key

so first i was tried to use del function with mask
del(."kubernetes.pod_annotations.checksum.*")
but no luck, seems that it's impossible to use some regex pattern for that, neither to use variable as argument for del function, so my next step is to remove fields with remove function in for_each loop like this

for_each(.) -> |key, val| {
            if match(key, r'^kubernetes\.pod_annotations\.checksum*') {
              ., err = remove(value: ., path: [key])
              if err != null {
                log(err)
              }
            }

But still got messages with this fields
This looks like common usecase, but i wasn't able to find some leads in documentation, can anyone help me with this?

[spencergilbert]

Hi @Wilderone, sorry we missed this question!

You actually got pretty close, this is what I ended up using:

for_each(.kubernetes.pod_annotations) -> |key, _val| {
  if starts_with(key, "checksum") {
    . = remove!(., path: ["kubernetes", "pod_annotations", key])
  }
}

The event has a top level key of kubernetes which contains pod_annotations an object of string values. Paths have a few invalid characters so the leaf keys you're working with are actually "checksum/...".

I removed the error handling for brevity, and switched to starts_with() since we just needed to match substrings, not anything complex. Finally, when using remove I rebuilt the complete path we needed to delete as we were operating on the very edge in the iteration.

Ex:

$ .
{ "kubernetes": { "pod_annotations": { "checksum/result-backend-secret": "foo", "checksum/webserver-secret-key": "bar", "safe/key": "value" } } }

$ for_each(.kubernetes.pod_annotations) -> |key, _val| {
  if starts_with(key, "checksum") {
    . = remove!(., path: ["kubernetes", "pod_annotations", key])
  }
}
null

$ .
{ "kubernetes": { "pod_annotations": { "safe/key": "value" } } }

I was hoping map_keys would allow me to do this more easily (and not pull in the val either) but it doesn't look like we support deleting there, I dropped a question internally to see if that's something we were interested in adding as a future improvement.

Please let me know if this helped, and sorry again for the delayed reply!

[Wilderone]

@spencergilbert Thanks a lot for this detailed answer, it helps and let me understand VRL better!

UPD: Looks like kubernetes source gave me some different type for this path, because of your example return errors:

error[E121]: type mismatch in closure parameter
the closure tied to this function call expects a different input value
expression has an inferred type of any where an array or object was expected

UPD2: solved by casting annotations to object type

for_each(object!(.kubernetes.pod_annotations)) -> |key, _val| {
  if starts_with(key, "checksum") {
    . = remove!(., path: ["kubernetes", "pod_annotations", key])
  }
}