Skip to content

Commit

Permalink
chore: Update to fluentbit 3.1.8 (#1459)
Browse files Browse the repository at this point in the history
  • Loading branch information
@a-thaler
a-thaler committed Sep 19, 2024
1 parent 193bb01 commit b96289e
Show file tree
Hide file tree
Showing 4 changed files with 19 additions and 18 deletions.
31 changes: 16 additions & 15 deletions docs/contributor/benchmarks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -339,21 +339,22 @@ test results are printed out.

<div class="table-wrapper" markdown="block">

| Version/Test | Single Pipeline (ci-logs) | | | | | Multi Pipeline (ci-logs-m) | | | | | Single Pipeline Backpressure (ci-logs-b) | | | | | Multi Pipeline Backpressure (ci-logs-mb) | | | | |
|---------------------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|
| | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage |
| 2.2.1 | 5165 | 8541 | 68518 | 172, 190 | 1, 1 | 2009 | 2195 | 102932 | 332, 320 | 0.9, 0.9 | 5914 | 1498 | 79247 | 184, 176 | 0.9, 1 | 1979 | 489 | 83442 | 310, 322 | 0.9, 0.9 |
| 2.2.2 | 5159 | 7811 | 75545 | 171, 170 | 1, 1 | 1910 | 2516 | 103780 | 324, 324 | 0.9, 0.9 | 5857 | 1513 | 72494 | 189, 200 | 1, 1 | 1860 | 421 | 90852 | 314, 322 | 0.9, 0.9 |
| 2.2.2 (new setup) | 5445 | 9668 | 68453 | 248, 981 | 1, 1 | 6201 | 2747 | 89291 | 544, 720 | 1, 1 | 6009 | 1723 | 58982 | 650, 682 | 1, 1 | 6111 | 385 | 108909 | 686, 931 | 0.9, 0.9 |
| 3.0.3 | 9483 | 22042 | 53251 | 366, 681 | 1, 1 | 10737 | 8785 | 115232 | 953, 568 | 0.9, 0.9 | 10425 | 4610 | 80614 | 856, 704 | 0.9, 0.9 | 10955 | 1724 | 87530 | 503, 594 | 0.9, 0.9 |
| 3.0.4 | 4341 | 8296 | 35628 | 971, 726 | 0.1, 0.1 | 1201 | 544 | 103624 | 652, 815 | 0, 0 | 932 | 297 | 37663 | 615, 726 | 0.1, 0.1 | 1477 | 171 | 108885 | 530, 566 | 0, 0.1 |
| 3.0.7 (old metrics) | 4241 | 7782 | 47586 | 815,1021 | 0.7, 0.1 | 3809 | 1968 | 107529 | 837,965 | 0.4, 0 | 3472 | 1093 | 33818 | 792, 597 | 0, 0.1 | 2180 | 177 | 87052 | 708, 631 | 0, 0.1 |
| 3.0.7 (new metrics) | 4036 | 7173 | 31689 | 825,852 | 0.1, 0.1 | 2481 | 1852 | 104689 | 747,395 | 0.1, 0 | 1520 | 484 | 37907 | 561, 731 | 0.1, 0.1 | 807 | 58 | 94365 | 544, 211 | 0,0 |
| 3.0.7 (new) | 9514 | 30273 | 30263 | 105, 113 | 1, 1 | 9027 | 23850 | 1521511 | 186, 552 | 1, 0.7 | 7285 | 8357 | 1891569 | 662, 668 | 0.8, 0.8 | 5602 | 2619 | 5249308 | 680, 713 | 0.5, 0.5 |
| 3.1.3 | 8922 | 28278 | 34609 | 105,107 | 0.8, 0.9 | 4542 | 9605 | 2676743 | 601,528 | 0.4, 0.4 | 3764 | 4216 | 1896390 | 620, 636 | 0.4, 0.4 | 3336 | 1499 | 4892724 | 678, 698 | 0.3, 0.3 |
| 3.1.6 | 9423 | 30652 | 37562 | 105 | 0.9 | 4522 | 9517 | 2414417 | 631 | 0.4 | 4002 | 4945 | 1897270 | 693 | 0.5 | 3224 | 1680 | 4898631 | 634 | 0.3 |
| 3.1.3 (after optimization) | 9149 | 29694 | 37883 | 93, 98 | 1, 0.8 | 4344 | 8354 | 2893257 | 601, 587 | 0.4, 0.4 | 3715 | 4000 | 1947742 | 739, 692 | 0.4, 0.4 | 3200 | 1314 | 4950684 | 662, 693 | 0.3, 0.3 |
| 3.1.6 (after optimization) | 10183 | 33117 | 30358 | 98, 93 | 1, 1 | 4349 | 8935 | 2779129 | 539, 660 | 0.4, 0.4 | 5471 | 5840 | 1889836 | 646, 641 | 0.5, 0.6 | 3224 | 1621 | 4882502 | 686, 649 | 0.3, 0.3 |
| Version/Test | Single Pipeline (ci-logs) | | | | | Multi Pipeline (ci-logs-m) | | | | | Single Pipeline Backpressure (ci-logs-b) | | | | | Multi Pipeline Backpressure (ci-logs-mb) | | | | |
|--------------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|
| | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage |
| 2.2.1 | 5165 | 8541 | 68518 | 172,190, | 1.0,1.0, | 2009 | 2195 | 102932 | 332,320, | 0.9,0.9, | 5914 | 1498 | 79247 | 184,176, | 0.9,1.0, | 1979 | 489 | 83442 | 310,322, | 0.9,0.9, |
| 2.2.2 | 5159 | 7811 | 75545 | 171,170, | 1.0,1.0, | 1910 | 2516 | 103780 | 324,324, | 0.9,0.9, | 5857 | 1513 | 72494 | 189,200, | 1.0,1.0, | 1860 | 421 | 90852 | 314,322, | 0.9,0.9, |
| 2.2.2 (new setup) | 5445 | 9668 | 68453 | 248,981, | 1.0,1.0, | 6201 | 2747 | 89291 | 544,720, | 1.0,1.0, | 6009 | 1723 | 58982 | 650,682, | 1.0,1.0, | 6111 | 385 | 108909 | 686,931, | 0.9,0.9, |
| 3.0.3 | 9483 | 22042 | 53251 | 366, 681 | 1.0,1.0, | 10737| 8785 | 115232 | 953,568, | 0.9,0.9, | 10425| 4610 | 80614 | 856,704, | 0.9,0.9, | 10955| 1724 | 87530 | 503,594, | 0.9,0.9, |
| 3.0.4 | 4341 | 8296 | 35628 | 971,726, | 0.1,0.1, | 1201 | 544 | 103624 | 652,815, | 0.0,0.0, | 932 | 297 | 37663 | 615,726, | 0.1,0.1, | 1477 | 171 | 108885 | 530,566, | 0.0,0.1, |
| 3.0.7 (old metrics) | 4241 | 7782 | 47586 | 815,1021,| 0.7,0.1, | 3809 | 1968 | 107529 | 837,965, | 0.4,0.0, | 3472 | 1093 | 33818 | 792,597, | 0.0,0.1, | 2180 | 177 | 87052 | 708,631, | 0.0,0.1, |
| 3.0.7 (new metrics) | 4036 | 7173 | 31689 | 825,852, | 0.1,0.1, | 2481 | 1852 | 104689 | 747,395, | 0.1,0.0, | 1520 | 484 | 37907 | 561,731, | 0.1,0.1, | 807 | 58 | 94365 | 544,211, | 0.0,0.0, |
| 3.0.7 (new) | 9514 | 30273 | 30263 | 105,113, | 1.0,1.0, | 9027 | 23850 | 1521511 | 186,552, | 1.0,0.7, | 7285 | 8357 | 1891569 | 662,668, | 0.8,0.8, | 5602 | 2619 | 5249308 | 680,713, | 0.5,0.5, |
| 3.1.3 | 8922 | 28278 | 34609 | 105,107 | 0.8, 0.9 | 4542 | 9605 | 2676743 | 601,528, | 0.4,0.4, | 3764 | 4216 | 1896390 | 620,636, | 0.4,0.4, | 3336 | 1499 | 4892724 | 678,698, | 0.3,0.3, |
| 3.1.6 | 9423 | 30652 | 37562 | 105,105, | 0.9,0.9, | 4522 | 9517 | 2414417 | 631,631, | 0.4,0.4, | 4002 | 4945 | 1897270 | 693,693, | 0.5,0.5, | 3224 | 1680 | 4898631 | 634,634, | 0.3,0.3, |
| 3.1.3 (optimized) | 9149 | 29694 | 37883 | 093,098, | 1.0,0.8, | 4344 | 8354 | 2893257 | 601,587, | 0.4,0.4, | 3715 | 4000 | 1947742 | 739,692, | 0.4,0.4, | 3200 | 1314 | 4950684 | 662,693, | 0.3,0.3, |
| 3.1.6 (optimized) | 10183 | 33117 | 30358 | 098,093, | 1.0,1.0, | 4349 | 8935 | 2779129 | 539,660, | 0.4,0.4, | 5471 | 5840 | 1889836 | 646,641, | 0.5,0.6, | 3224 | 1621 | 4882502 | 686,649, | 0.3,0.3, |
| 3.1.8 | 8385 | 27172 | 43983 | 102,093, | 0.8,0.8, | 4539 | 10019 | 2582036 | 545,550, | 0.4,0.4, | 3977 | 5080 | 1905566 | 674,645, | 0.5,0.4, | 3278 | 1506 | 4956213 | 702,651, | 0.3,0.3, |
</div>

> **NOTE:** The test results can vary up to 1000KByte/sec in the input, and up to 3000KByte/sec in the output processing rate with single pipeline tests. The multi pipeline test results can be lower than the single pipeline test results, because the test setup is shared between multiple pipelines and this may create pressure on the shared resources.
Expand Down
2 changes: 1 addition & 1 deletion internal/resources/fluentbit/resources.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ func MakeMetricsService(name types.NamespacedName) *corev1.Service {
"prometheus.io/scrape": "true",
"prometheus.io/port": strconv.Itoa(ports.HTTP),
"prometheus.io/scheme": "http",
"prometheus.io/path": "/api/v1/metrics/prometheus",
"prometheus.io/path": "/api/v2/metrics/prometheus",
},
},
Spec: corev1.ServiceSpec{
Expand Down
2 changes: 1 addition & 1 deletion main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ var (

const (
defaultFluentBitExporterImage = "europe-docker.pkg.dev/kyma-project/prod/directory-size-exporter:v20240605-7743c77e"
defaultFluentBitImage = "europe-docker.pkg.dev/kyma-project/prod/external/fluent/fluent-bit:3.1.6"
defaultFluentBitImage = "europe-docker.pkg.dev/kyma-project/prod/external/fluent/fluent-bit:3.1.8"
defaultOtelImage = "europe-docker.pkg.dev/kyma-project/prod/kyma-otel-collector:0.108.1-main"
defaultSelfMonitorImage = "europe-docker.pkg.dev/kyma-project/prod/tpi/telemetry-self-monitor:2.53.2-cc4f64c"

Expand Down
2 changes: 1 addition & 1 deletion sec-scanners-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module-name: telemetry
protecode:
- europe-docker.pkg.dev/kyma-project/prod/telemetry-manager:main
- europe-docker.pkg.dev/kyma-project/prod/kyma-otel-collector:0.108.1-main
- europe-docker.pkg.dev/kyma-project/prod/external/fluent/fluent-bit:3.1.6
- europe-docker.pkg.dev/kyma-project/prod/external/fluent/fluent-bit:3.1.8
- europe-docker.pkg.dev/kyma-project/prod/directory-size-exporter:v20240605-7743c77e
- europe-docker.pkg.dev/kyma-project/prod/tpi/telemetry-self-monitor:2.53.2-cc4f64c
whitesource:
Expand Down

0 comments on commit b96289e

Please sign in to comment.