path problem, how can i solve this ?

[StefanSa]

Hi there,
have this:

[transforms.ecs_url_mapping_atp]
inputs = ["ecs_src_mapping_atp"]
type   = "remap"
source = '''
if exists(.sophos.xg.url) {
   .url.original = del(.sophos.xg.url)
}
. = parse_url!(.url.original)
.url.domain = del(.host)
.url.path = del(.path)
.url.scheme = del(.scheme)
.url.fragment = del(.fragment)
.url.password = del(.password)
.url.username = del(.username)
.url.port = del(.xg.port)
.url.query = del(.query)

[transforms.geo_mapping_src_atp]
type = "geoip"
inputs = ["ecs_url_mapping_atp"]
database = "/etc/vector/GeoIP/GeoLite2-City.mmdb"
source = "source.ip"
target = "source.geo"

With this construct, vector aborts with the error message that it cannot find the field for .source.ip (geoip)
and that the values of parce_url are make complete the json value defect.

i have understood that this is a path problem, for example, if i set the path as in the master pipeline,
. = set!(., path: ["sophos", "xg"], data: parse_url!(.url.original))
the previous events with this path will be deleted / missing.
However the value of the parse_url is correct and also the json result.
The question is, what is the correct value of the path for parce_url function?

[StefanSa]

Ok. took another path like the one in the master pipeline, so it seems to work.
Apparently a path must be passed here.
The question is how to do it right.

[spencergilbert]

Can you share an example event thats coming into that remap component, as well as what you'd expect the event to look like after?

One thing that I notice immediately about the script is this:

if exists(.sophos.xg.url) {
   .url.original = del(.sophos.xg.url)
}
# Everything below assumes the above `if` statement is true,
# which suggests everything below should be included in the
# above block. Otherwise you could be calling these functions
# on fields that don't exist and erroring.
. = parse_url!(.url.original)
.url.domain = del(.host)
.url.path = del(.path)
.url.scheme = del(.scheme)
.url.fragment = del(.fragment)
.url.password = del(.password)
.url.username = del(.username)
.url.port = del(.xg.port)
.url.query = del(.query)

That is to say, the following seems more correct and robust:

if exists(.sophos.xg.url) {
  .url.original = del(.sophos.xg.url)
  . = parse_url!(.url.original)
  .url.domain = del(.host)
  .url.path = del(.path)
  .url.scheme = del(.scheme)
  .url.fragment = del(.fragment)
  .url.password = del(.password)
  .url.username = del(.username)
  .url.port = del(.xg.port)
  .url.query = del(.query)
}

[StefanSa]

Hi @spencergilbert Spencer,

Incoming value
.sophos.xg.url = http://sophostest.com/callhom e/index.html

Expected result:

  "url": {
    "domain": "sophostest.com",
    "original": "http://sophostest.com/callhom e/index.html",
    "path": "/callhome/index.html",
    "scheme": "http"
  }

FYI, as mentioned.
If i take any path here, but not the one from the master pipeline, everything is OK.

. = set!(., path: ["s", "g"], data: parse_url!(.url.original))
.url.domain = del(.s.g.host)
.url.path = del(.s.g.path)
.url.scheme = del(.s.g.scheme)
.url.fragment = del(.s.g.fragment)
.url.password = del(.s.g.password)
.url.username = del(.s.g.username)
.url.port = del(.s.g.port)
.url.query = del(.s.g.query)

[spencergilbert]

Is it possible the path is actually ."sophos.xg".url and not .sophos.xg.url?

{
  "sophos.xg": {
    "url": ""
  }
}

vs

{
  "sophos.xg.url": ""
}

[StefanSa]

@spencergilbert
no Spencer, i have modified the path you provided to follow the elastic ecs scheme.
I currently use this (without problems).
. = set!(., path: ["sophos", "xg"], data: parse_key_value!(.message))

[spencergilbert]

OH - @StefanSa perhaps I see the problem, . = parse_url!(.url.original) will replace the entire event with the parsed URL. You may be wanting to do this instead:

if exists(.sophos.xg.url) {
  .url.original = del(.sophos.xg.url)
  . |= parse_url!(.url.original) # `|=` is the "assignment operator"
  # ref: https://vector.dev/docs/reference/vrl/expressions/#assignment-example-object-merge-assignment
  .url.domain = del(.host)
  .url.path = del(.path)
  .url.scheme = del(.scheme)
  .url.fragment = del(.fragment)
  .url.password = del(.password)
  .url.username = del(.username)
  .url.port = del(.xg.port)
  .url.query = del(.query)
}

[StefanSa]

@spencergilbert Spencer
here is an example with successful parsing with Vector :)
example.json.zip

[spencergilbert]

This looks really nice! I'd love to see your full config once you have it finished, I spent some time a year ago trying to setup a config to transform to ECS and I eventually gave up 😆

[StefanSa]

@spencergilbert
👍 Thank you, perfect.
something about the background.
i am the initial maintainer of some firewall modules from elastic filebeat, e.g. sophos-xg and juniper. I now want to realize these modules, plus others, with vector. The goal should be to get along without the ELK stack and to realize this using only vector and opensearch.