Geoip plugin unexpectedly requires native extensions and cannot be found by FluentD

[Luthien-in-edhil]

Problem

Half a year ago we deployed FluentD with fluent-plugin-geoip and fluent-plugin-multi-format-parser without any issues in Kubernetes using this code in the Dockerfile:

FROM fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
LABEL Author="Europeana Foundation <[email protected]>"
USER root
#RUN buildDeps="build-essential libgeoip-dev ruby-dev  libmaxminddb-dev wget" \
&& apt-get update \
&& apt-get install \
-y --no-install-recommends \
$buildDeps \
&& fluent-gem install fluent-plugin-geoip fluent-plugin-multi-format-parser  #\
&& rm -rf /var/lib/apt/lists/* \
# removes GeoIP database so we re-download below
&& gem sources --clear-all \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem \
&& mkdir -p /usr/share/geoip \
&& wget -P /usr/share/geoip/ https://git.io/GeoLite2-City.mmdb

We build the Docker image in a Jenkins job that also deploys it on our Kubernetes cluster on IBM Cloud.

Last week I tried to make some changes to FluentD and found that I creating the Dockerfile failed with

...
Processing triggers for libc-bin (2.31-13+deb11u7) ...
Building native extensions. This could take a while...
Successfully installed geoip-c-0.9.1
Building native extensions. This could take a while...
[91mERROR:  Error installing fluent-plugin-geoip:
	ERROR: Failed to build gem native extension.

    current directory: /fluentd/vendor/bundle/ruby/3.1.0/gems/geoip2_c-0.3.4/ext/geoip2
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 extconf.rb
./bootstrap: 7: autoreconf: not found
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.
...
The command '/bin/sh -c buildDeps= ... {see the dockerfile} ... returned a non-zero code: 1

I initially found a solution in the FluentD docs - Plugin Management:

Gem and Native Extension
Some plugins depend on the native extension library. It means that you need to install development packages to build it e.g. gcc, make, autoconf, etc.

I found that the Docker image was built successfully after adding automake and libtool to the buildeps command:

RUN buildDeps="build-essential libgeoip-dev ruby-dev automake libtool libmaxminddb-dev wget" \
  && apt-get update \
  ...

However, I could not start the pod any longer. It fails with this error:

2023-11-22T13:39:36.718242237Z 2023-11-22 13:39:36 +0000 [error]: config error file=“/fluentd/etc/fluent.conf” 
error_class=Fluent::NotFoundPluginError error=“Unknown filter plugin ‘geoip’. 
Run ‘gem search -rd fluent-plugin’ to find plugins”

And that's where I'm stuck now. I have no idea how to fix this - following the suggestion in the error message I tried adding the gem search -rd fluent-plugin command to the Dockerfile, but that doesn't have any effect - and I'm afraid I'm not that fluent in Ruby (if you pardon the pun).

Steps to replicate

That would be a bit hard because you'd need to copy our infrastructure. I did not yet get around to trying to run this on a local Kubernetes cluster, but judging from the October 18 message the issue is not limited to our setup.

Expected Behavior

1. Based on previous experience I wouldn't expect fluent-plugin-geoip to suddenly become dependent on the native extension library
2. I would expect that fluent-gem install fluent-plugin-geoip actually installs the geoip plugin

...

Your environment

• Debian (from Docker image fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch). Jenkins logs shows http://deb.debian.org/debian bullseye
• Fluentd version: 1.16.3
• plugin version: v1.3.2
  • paste boot log of fluentd: n/a, FluentD won't boot