[performance][Logstash] Change Netty's based plugins to use JavaHeap instead of Direct

[andsel]

Considerations

Netty plugins uses Direct memory to allocate buffers to process data, however when byte arrays resides on direct space, there isn't any mechanism to track down to the root that retains most of the memory, when leaks or an OOM error happens.
To make more evident in heap profiling dumps, would be ideal to leverage Java Heap allocation of such buffers.

After some testing, using just Java Heap buffers, under heavy load on Logstash, didn't shown any performance degradation.

Plugins Impacted

• Beats Input
• HTTP Input
• TCP Input

Related Issues

• Beat stops processing events after OOM but keeps running logstash-plugins/logstash-input-beats#309
• Default XX:MaxDirectMemorySize to 1/2 Xmx #12621

Proposal

Provide a global setting that can enable disable the selection to allocate Netty buffers from direct vs java heap. The setting should implicitly configure the Netty's properties that command the kinds of allocation (-Dio.netty.allocator.numDirectArenas=0
-Dio.netty.noPreferDirect=true).
Initially this behaviour would be disabled by default, so that a user can opt-in for such feature. In a later release the flag could be enabled by default.

Tasks

Preview Give feedback

1. Add global setting to that changes the behavior to use Heap memory - affects beats, http and tcp input #16051
   +0
   
2. Document JVM Heap sizing based on the changes above
3. Consider changing this setting to on by default