feat(new sink): add greptime log sink

[paomian]

• update greptimedb-ingester version
• add a new greptimedb_logs sink
• Changed the original greptime code structure to make it clearer

Details of greptime_logs

The greptime_logs sink is a new sink that writes logs to greptimedb.
It builds requests to send data using Vector's public http component.

Config

[sources.my_source_id]
type = "file"
data_dir = "./vector_data"
include = [ "access.log" ]


[sinks.sink_greptime_logs]
type = "greptimedb_logs"
## The endpoint to send the logs to.
endpoint = "http://localhost:4000"
## The table to insert into.
table = "logs"
## The database to insert into.
dbname = "public"
## pipeline name to use
## Pipeline is a processor that is used to process and remap the data before it is stored in the database.
pipeline_name = "test"
## The authentication
username = "username"
password = "password"
## The compression to use.
compression = "gzip"
inputs = [ "my_source_id" ]

[jszwedko]

Thanks @paomian ! We'll give this a review shortly.

[StephenWakely]

@paomian Just checking in terms of maintanance going forward. Are you a part of the GreptimeDB team and are they willing to maintain, or would you be willing to maintain yourself?

[paomian]

@StephenWakely I am a part of the GreptimeDB team. We will maintain the code for this sink.

[StephenWakely]

Thank you! There's just some fairly minor changes that need to be made here.

The version needs to be updated for the integration test here. I just get errors on the version mentioned there.

We will need some changes for the documentation:

• create a new file based on this for the greptime_logs sink.
• a new file based on [this] for greptime_logs, changing it here to be relevant for logs.

Can you add a changelog entry. You can find details here.

[StephenWakely]

Ah, you just added a changelog as I was doing the review! :-) You will need to add an authors line as an external contributor. Also please mention the change about the rename from greptimedb to greptimedb_metrics.

[paomian]

Ah, you just added a changelog as I was doing the review! :-) You will need to add an authors line as an external contributor. Also please mention the change about the rename from greptimedb to greptimedb_metrics.

It may take some time. I'll push a new version at a later time.

[paomian]

Ah, you just added a changelog as I was doing the review! :-) You will need to add an authors line as an external contributor. Also please mention the change about the rename from greptimedb to greptimedb_metrics.

@StephenWakely Added documentation and testing

[StephenWakely]

A few extra nits. Can you combine the log and metric integration tests so to test greptimedb you can run cargo vdev int test greptimedb and it will run both the metric and log tests.

[aliciascott]

@paomian in addition to a few other comments it looks like there is a missing link in this description, thanks !

[StephenWakely]

I get this error when running the integration test:

〉cargo vdev int test greptimedb                                                                                                              chore/greptime_log
Error: failed to read /home/stephenwakely/src/vector/scripts/integration/greptimedb/test.yaml

Caused by:
    No such file or directory (os error 2)

We just need one folder scripts/integration/greptimedb rather than the two scripts/integration/greptimedb_metrics scripts/integration/greptimedb_logs.

Can you make sure the command cargo vdev int test greptimedb runs successfully.

[paomian]

I get this error when running the integration test:

〉cargo vdev int test greptimedb                                                                                                              chore/greptime_log
Error: failed to read /home/stephenwakely/src/vector/scripts/integration/greptimedb/test.yaml

Caused by:
    No such file or directory (os error 2)

We just need one folder scripts/integration/greptimedb rather than the two scripts/integration/greptimedb_metrics scripts/integration/greptimedb_logs.

Can you make sure the command cargo vdev int test greptimedb runs successfully.

merged

[StephenWakely]

Can you run make generate-component-docs and check in the resulting cue file changes. We need this to generate the docs on the website.

A list of checks that need to be run before the PR will pass CI can be found here. Can you run through them to make sure they all work..

[paomian]

Can you run make generate-component-docs and check in the resulting cue file changes. We need this to generate the docs on the website.

A list of checks that need to be run before the PR will pass CI can be found here. Can you run through them to make sure they all work..

I followed the steps here but got the error reported.

cargo vdev check licenses
Record for "derive_builder" is missing or changed.
Record for "derive_builder_core" is missing or changed.
Record for "derive_builder_macro" is missing or changed.
Record for "greptimedb-ingester" is missing or changed.
Error: Current "LICENSE-3rdparty.csv" is not up to date.
Run `cargo vdev build licenses` to regenerate the file
Error: command: cd "vector" && "dd-rust-license-tool" "check"
  failed with exit code: 1

I fixed the problem about greptimedb-ingester. I don't know what to do with the others.

[paomian]

Currently make check-component-docs and make docs is working fine.

[StephenWakely]

I'd like to ask. How do I get another reviewer to approval my pr so it can be merged?

I've pinged them.

[StephenWakely]

Just a simple change which I think should resolve the issue with the docs. Make sure to run make generate-component-docs after making this change.

[StephenWakely]

Can you run make generate-component-docs and check in the changes.

[paomian]

Can you run make generate-component-docs and check in the changes.

Execution has been completed. And the branch has been updated.

[brett0000FF]

Thanks! Approved with minor suggestions.

[StephenWakely]

@paomian I tried to just commit the doc changes suggested before realising that it was in a generated file and needs to be updated in the doc comments. Can you make those changes (don't forget the make generate-component-docs at the end) and we should be good to go!

[jszwedko]

Thanks for the docs updates! This seems good-to-go now. Thanks for this contribution!

[paomian]

Is there anything that I need to update? @aliciascott

Comments were resolved and we had another Docs team review

[github-actions]

Regression Detector Results

Run ID: cc72af77-51bb-4ab7-8565-498e163f7f92 Metrics dashboard

Baseline: ac4e194
Comparison: ec6c629

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

Significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

perf	experiment	goal	Δ mean %	Δ mean % CI	links
❌	otlp_http_to_blackhole	ingress throughput	-5.15	[-5.30, -5.00]
❌	socket_to_socket_blackhole	ingress throughput	-5.59	[-5.66, -5.53]
❌	syslog_log2metric_humio_metrics	ingress throughput	-6.26	[-6.41, -6.11]

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	file_to_blackhole	egress throughput	+1.10	[-5.87, +8.07]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	syslog_splunk_hec_logs	ingress throughput	+4.42	[+4.29, +4.54]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	+3.37	[+3.28, +3.46]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	+3.11	[+2.99, +3.23]
➖	http_elasticsearch	ingress throughput	+2.94	[+2.80, +3.09]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+2.61	[+2.46, +2.77]
➖	syslog_humio_logs	ingress throughput	+2.19	[+2.07, +2.31]
➖	fluent_elasticsearch	ingress throughput	+2.17	[+1.67, +2.66]
➖	syslog_loki	ingress throughput	+1.74	[+1.64, +1.85]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	+1.35	[+1.18, +1.51]
➖	file_to_blackhole	egress throughput	+1.10	[-5.87, +8.07]
➖	http_to_http_acks	ingress throughput	+0.67	[-0.65, +2.00]
➖	http_to_http_noack	ingress throughput	+0.16	[+0.08, +0.25]
➖	http_to_http_json	ingress throughput	+0.06	[+0.01, +0.11]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.10, +0.09]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.00	[-0.10, +0.10]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.02	[-0.10, +0.06]
➖	http_text_to_http_json	ingress throughput	-0.04	[-0.16, +0.09]
➖	otlp_grpc_to_blackhole	ingress throughput	-0.23	[-0.34, -0.11]
➖	splunk_hec_route_s3	ingress throughput	-0.25	[-0.59, +0.09]
➖	datadog_agent_remap_blackhole	ingress throughput	-0.32	[-0.47, -0.17]
➖	http_to_s3	ingress throughput	-0.49	[-0.76, -0.22]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-1.40	[-1.62, -1.19]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	-1.77	[-1.89, -1.64]
❌	otlp_http_to_blackhole	ingress throughput	-5.15	[-5.30, -5.00]
❌	socket_to_socket_blackhole	ingress throughput	-5.59	[-5.66, -5.53]
❌	syslog_log2metric_humio_metrics	ingress throughput	-6.26	[-6.41, -6.11]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[aliciascott]

Is there anything that I need to update? @aliciascott

No we are good based on @brett0000FF review :) I'll go ahead and update my review though in case it's holding you up :)

[paomian]

I haven't changed any code for these three tests, so don't know how to optimize their performance.

[jszwedko]

Apologies @paomian , I think those are likely to be a false positive. I marked this for merge again.

[github-actions]

Regression Detector Results

Run ID: 77a70d0e-cb5a-4127-a8dc-04d71ffd8ff6 Metrics dashboard

Baseline: f371bc2
Comparison: 35e82bd

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	links
❌	file_to_blackhole	egress throughput	-14.20	[-20.73, -7.68]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	http_text_to_http_json	ingress throughput	+3.90	[+3.77, +4.03]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	+2.09	[+1.96, +2.23]
➖	splunk_hec_route_s3	ingress throughput	+2.02	[+1.71, +2.33]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	+1.83	[+1.71, +1.95]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+1.60	[+1.46, +1.73]
➖	syslog_log2metric_humio_metrics	ingress throughput	+1.59	[+1.45, +1.73]
➖	socket_to_socket_blackhole	ingress throughput	+1.57	[+1.49, +1.64]
➖	otlp_http_to_blackhole	ingress throughput	+1.55	[+1.40, +1.70]
➖	http_to_http_acks	ingress throughput	+0.60	[-0.72, +1.92]
➖	http_to_s3	ingress throughput	+0.59	[+0.33, +0.86]
➖	syslog_splunk_hec_logs	ingress throughput	+0.47	[+0.37, +0.58]
➖	datadog_agent_remap_blackhole	ingress throughput	+0.37	[+0.27, +0.48]
➖	http_to_http_noack	ingress throughput	+0.24	[+0.14, +0.34]
➖	fluent_elasticsearch	ingress throughput	+0.07	[-0.42, +0.56]
➖	http_to_http_json	ingress throughput	+0.02	[-0.01, +0.05]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.01	[-0.09, +0.07]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.01	[-0.14, +0.11]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.02	[-0.12, +0.09]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	-0.02	[-0.12, +0.08]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	-0.16	[-0.34, +0.02]
➖	otlp_grpc_to_blackhole	ingress throughput	-0.24	[-0.35, -0.12]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-0.42	[-0.61, -0.23]
➖	http_elasticsearch	ingress throughput	-1.31	[-1.46, -1.15]
➖	syslog_loki	ingress throughput	-1.46	[-1.53, -1.39]
➖	syslog_humio_logs	ingress throughput	-1.74	[-1.87, -1.61]
❌	file_to_blackhole	egress throughput	-14.20	[-20.73, -7.68]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[jszwedko]

Thanks again for this contribution @paomian !