Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add details on load balance behaviour with Logstash output #1024

Merged
merged 1 commit into from
May 8, 2024
Merged

Add details on load balance behaviour with Logstash output #1024

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .../ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,18 @@ load balances published events onto all {ls} hosts. If `false`,
the output plugin sends all events to one host (determined at random) and
switches to another host if the selected one becomes unresponsive.

With `loadbalance` enabled:

* {agent} reads batches of events and sends each batch to one {ls} worker dynamically, based on a work-queue shared between the outputs.
* If a connection drops, {agent} takes the disconnected {ls} worker out of its pool.
* {agent} tries to reconnect. If it succeeds, it re-adds the {ls} worker to the pool.
* If one of the {ls} nodes is slow but "healthy", it sends a keep-alive signal until the full batch of data is processed. This prevents {agent} from sending further data until it receives an acknowledgement signal back from {ls}. {agent} keeps all events in memory until after that acknowledgement occurs.

Without `loadbalance` enabled:

* {agent} picks a random {ls} host and sends batches of events to it. Due to the random algorithm, the load on the {ls} nodes should be roughly equal.
* In case of any errors, {agent} picks another {ls} node, also at random. If a connection to a host fails, the host is retried only if there are errors on the new connection.

*Default:* `false`

Example:
Expand Down