Compatibility issues with fluentd plugins

[PranavBhatSF]

Usecase

We need to push kubernetes pod logs to Azure, S3 or Splunk based on the configuration.

What we have done

Built a custom image with Fluentd image as base and included the required plugins.
Dockerfile

FROM ghcr.io/kube-logging/fluentd:v1.16-4.9-full as base-fluentd

# add azure storage append blob plugin
RUN gem install fluent-plugin-azure-storage-append-blob -v 0.2.1
# add aws s3 plugin
RUN gem install fluent-plugin-s3 -v 1.7.2
# add splunk plugin
RUN gem install fluent-plugin-splunk-hec -v 1.3.2

Issue

Fluentd is pushing logs to S3 and Azure. But when we configure splunk as destination, fluentd-configcheck pod fails with the below error

2024-09-25 07:20:53 +0000 [info]: starting fluentd-1.16.3 as dry run mode ruby="3.2.4"
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:210:in `rescue in try_activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:203:in `try_activate'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:204:in `try_activate'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': cannot load such file -- fluent/plugin/out_splunk (LoadError)
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	.....

[pepov]

What happens if you try ghcr.io/kube-logging/fluentd:v1.16-4.9-filters instead of full?

[danialram]

Even with ghcr.io/kube-logging/fluentd:v1.16-4.9-filters we are seeing same issue.

[pepov]

We had this issue with the azure plugin and had to remove it because of the same reason. I'm not aware of a good resolution to this problem and would be good if someone could take a deeper look. To make things worse the underlying plugin is not maintained and the ruby library is already deprecated. We would need to look at a more future proof solution here that is based on the supported Rest API ideally.