Skip to content

Number of file descriptors increasing infinitely #17

Open
@khotkevych

Description

@khotkevych

I run fluent agent with fluent-plugin-flume inside Docker container.
After a period of time some programs inside my container failed with next error:

Check failed: _s.ok() Bad status: Runtime error: Could not create thread: Resource temporarily unavailable (error 11)

I discovered that number of file descriptors inside Docker container increasing infinitely:

sudo lsof| grep flume | wc -l
469292

And after a while number of descriptors becomes bigger then a file descriptors limit.
Looks like the thrift connection is handled in a wrong way: file descriptors are still present after the end of transaction.

Fluend config:

<source>
  @type forward
  tag forward_1
  bind 0.0.0.0
  port 24224
  linger_timeout 0
</source>

<match forward_1>
      @type copy
      <store>
              @type flume
              timeout 15
              host localhost
             port 33333
      </store>
</match>

Fume config:

forward_1.sources  = source1
forward_1.channels = channel1
forward_1.sinks = sink1
#
forward_1.sources.source1.type = thrift
forward_1.sources.source1.bind = localhost
forward_1.sources.source1.port = 33333
forward_1.sources.source1.channels = channel1
#
forward_1.channels.channel1.type = memory
forward_1.channels.channel1.capacity = 10000
forward_1.channels.channel1.transactionCapacity = 1000
#
forward_1.sinks.sink1.type = org.apache.kudu.flume.sink.KuduSink
forward_1.sinks.sink1.channel = channel1
forward_1.sinks.sink1.masterAddresses = 10.1.0.1:7051
forward_1.sinks.sink1.tableName = shop_logs
forward_1.sinks.sink1.batchSize = 50
forward_1.sinks.sink1.producer = KuduJsonProducer

Docker version: 17.03
Fluentd version: 0.14.19
Flume version: 1.7.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions