es_out: support Upstream Servers with configuration overriding #7608
Conversation
mabrarov
requested review from
PettitWesley,
edsiper,
niedbalski,
leonardo-albertovich,
fujimotos and
koleini
as code owners
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
812020d to
ba3382a
Compare
|
Hi reviewers, Is it possible to approve only workflow for this pull request, so that automated checks and build can start? Thank you. |
|
@mabrarov sure |
|
Hi @PettitWesley, It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:
I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes. Help of maintainers is appreciated. Thank you. |
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
ba3382a to
b7cd81b
Compare
|
Hi @PettitWesley, Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened. Thank you. |
|
Hi dear reviewers, Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review? Thank you. |
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
b7cd81b to
b81d3f7
Compare
|
Hi @PettitWesley and @edsiper, It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have. Thank you. |
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
b81d3f7 to
f6431c2
Compare
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
7158b1a to
6ca93fb
Compare
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
6ca93fb to
c961aa7
Compare
…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <[email protected]>
…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>
…llback contract Signed-off-by: Marat Abrarov <[email protected]>
…with Upstream node configuration For Elastic cloud authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: cloud_id. For AWS authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: http_proxy, no_proxy, tls*. Signed-off-by: Marat Abrarov <[email protected]>
Signed-off-by: Marat Abrarov <[email protected]>
…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>
mabrarov
force-pushed
the
feature/out_es_upstream_support_extended
branch
from
c961aa7 to
e3c8bab
Compare
Implementation of Upstream feature for the Elasticsearch output plugin.
This pull request is based on pull request #1560 and Forward output plugin.
It was tested in a local setup with:
Fluent Bit without Upstream feature connected to a single node of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.
Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.
fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es/fluent-bit.conf and (same in YAML format) fluent-bit-es/fluent-bit.yaml in mabrarov/elastic-stack repository.
Debug log is available at flb_es.log.
Fluent Bit with Upstream feature connected to all Elasticsearch data nodes of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.
Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.
fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es-cluster/fluent-bit.conf and (same in YAML format) fluent-bit-es-cluster/fluent-bit.yaml in mabrarov/elastic-stack repository.
Debug log is available at flb_es_upstream.log.
Testing
TEST_PRESET=valgrind SKIP_TESTS='flb-rt-out_td flb-it-network' ./run_code_analysis.shok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.