-
|
I want to use Vector to ingest some gzip files then use remap to parse them. We put some log files with json format into a gzip file to do the test. Here is my config: Finally, I got error messages: Could you help me to solve it please? Thank you so much for your help. |
Beta Was this translation helpful? Give feedback.
Answered by
jszwedko
Apr 25, 2022
Replies: 1 comment 9 replies
-
|
Hi @syzcch ! Looking at your error messages there, it seems like you might have compressed To verify Vector's behavior of reading individual files, I tested with: (flog is a log generator: https://github.com/seattlerb/flog) Running Vector with the below configuration: data_dir = "/tmp/vector"
[sources.gz_logs]
type = "file"
include = [ "/tmp/log.json.gz" ]
[transforms.parse_logs]
type = "remap"
inputs = ["gz_logs"]
source = '''
. = parse_json!(.message)
'''
[sinks.print]
type = "console"
inputs = ["parse_logs"]
encoding.codec = "json"And Vector was able to correctly read and parse the lines: |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
I added this example to our list of real-world use-cases for iteration in VRL, see: #12455 The final code that (once iteration lands in the next release) solves this issue is written as such: .input = map_values(array!(.input)) -> |input| {
# `input` is an object inside the top-level `.input` array.
input.items = map_values(array!(input.items)) -> |item| {
# `item` is an object inside the `.input.items array.
# "userAttributes": [
# {
# "__type": "String",
# "value": "Peter",
# "key": "Name"
# },
# {
# "__type": "String",
# "values": [
# {
# "city": "Tokyo",
# "country": "Japan"
# }
# ],
# "key": "Address"
# }
# ],
item.userAttributes = map_values(array!(item.userAttributes)) -> |attribute| {
if attribute.key == "Name" {
del(attribute.__type)
key = del(attribute.key)
value = del(attribute.value)
attribute = set!(attribute, [key], value)
} else if attribute.key == "Address" {
attribute.values = map_values(array!(attribute.values)) -> |address| {
del(address.city)
address
}
}
attribute
}
# "userId": [
# {
# "uId": "0000001",
# "userGroupId": 0
# }
# ]
item.userId = map_values(array!(item.userId)) -> |id| {
del(id.userGroupId)
id
}
item
}
input
} |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jszwedko and @JeanMertz, Thank you so much for your great help. I am waiting for the Vector 0.22 and then I will do the refactor to migrate log pipeline from Filebeat->Logstash->ES to Vector->ES. Really appreciate your help. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jszwedko, I saw that we published the latest version 0.22.2 and that's really an exciting news. So what about the iteration function in VRL? Already in this release or in next one? Thank you so much. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you can see a post regarding the newly added functions here: https://vector.dev/highlights/2022-05-18-vrl-iteration-support/ They are also documented at https://vrl.dev/functions |
Beta Was this translation helpful? Give feedback.
-
|
That's really a good news. Thank you team. |
Beta Was this translation helpful? Give feedback.
Answer selected by
jszwedko
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @syzcch !
Looking at your error messages there, it seems like you might have compressed
tararchives? Vector won't read those correctly, only individual compressed files.To verify Vector's behavior of reading individual files, I tested with:
(flog is a log generator: https://github.com/seattlerb/flog)
Running Vector with the below configuration:
And Vector was abl…