systemd: build: fix service startup failure when using IMDSv2

[charltonstanley]

PR #9845 added a service dependency sethostname.service for Amazon Linux 2, which uses Amazon's Instance Metadata Service (IMDS) to query and set the hostname. This works for instances using the legacy IMDSv1, however it fails when admins have enforced the usage of IMDSv2. This patch is a modification of @cpandya-we's comment/work on #10186. Closes #10186

Note: I used modified versions of the code samples @cosmo0920 submitted in #9845. Specifically, note the differences in the packer file, where IMDSv2 support was enabled.

Testing

• Example configuration file for the change

[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on

[INPUT]
    name dummy
    tag  dummy.local

    # Read interval (sec) Default: 1
    interval_sec 10

[FILTER]
    Name  record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record filter_type record_modifiler

[OUTPUT]
    name  stdout
    match *

• [] Debug log output from testing the change

Log output confirming both fluent-bit.service and sethostname.service service start successfully.

[ec2-user@ip-172-31-10-223 ~]$ systemctl status fluent-bit -l
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2025-04-16 18:14:35 UTC; 52s ago
     Docs: https://docs.fluentbit.io/manual/
 Main PID: 3229 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3229 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [ctraces ] version=0.6.2
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [input:dummy:dummy.0] initializing
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [sp] stream processor started
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [output:stdout:stdout.0] worker #0 started
Apr 16 18:14:45 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827284.897177091, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:14:55 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827294.896876191, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:05 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827304.897044922, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:15 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827314.896878894, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:25 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827324.896980969, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
[ec2-user@ip-172-31-10-223 ~]$ systemctl status sethostname -l
● sethostname.service - Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support
   Loaded: loaded (/usr/lib/systemd/system/sethostname.service; disabled; vendor preset: disabled)
   Active: active (exited) since Wed 2025-04-16 18:14:35 UTC; 1min 21s ago
  Process: 3218 ExecStartPost=/bin/bash -c rm -f /run/imds_token /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3212 ExecStart=/bin/bash -c /usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)" (code=exited, status=0/SUCCESS)
  Process: 3204 ExecStartPre=/bin/bash -c curl -s "http://169.254.169.254/latest/meta-data/hostname"    -H "X-aws-ec2-metadata-token: $(cat /run/imds_token)" > /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3174 ExecStartPre=/bin/bash -c curl -sX PUT "http://169.254.169.254/latest/api/token"    -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" > /run/imds_token (code=exited, status=0/SUCCESS)
 Main PID: 3212 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sethostname.service

Apr 16 18:14:35 ip-172-31-10-223.ec2.internal systemd[1]: Starting Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support...
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal systemd[1]: Started Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support.

Here is the packer template for creating AMI. Note the addition of the the imds_support property, as well as the additional metadata_options.

packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.8"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "amazonlinux2" {
  ami_name      = "fbit-amazon-linux2-amd64"
  instance_type = "t3.micro"
  region        = "us-east-1"
  source_ami_filter {
    filters = {
      name                = "amzn2-ami-kernel-5.10-hvm-2.*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
      architecture        = "x86_64"
    }
    most_recent = true
    owners      = ["amazon"]
  }
  ssh_username = "ec2-user"

### Added IMDSv2 support ###
  imds_support = "v2.0"
  metadata_options {
    http_endpoint = "enabled"
    http_tokens = "required"
    http_put_response_hop_limit = 2
  }
}

build {
  name = "az2-fbit-packer"
  sources = [
    "source.amazon-ebs.amazonlinux2"
  ]

  provisioner "file" {
    source      = "/path/to/github/fluent-bit/packaging/packages/amazonlinux/2/2025-04-16-10_05_01/fluent-bit-4.0.1-1.x86_64.rpm"
    destination = "/home/ec2-user/fluent-bit.rpm"
  }

  provisioner "file" {
    source      = "path/to/fluent-bit.conf"
    destination = "/home/ec2-user/fluent-bit.conf"
  }

  provisioner "shell" {
    inline = [
      "sudo yum install -y /home/ec2-user/fluent-bit.rpm",
      "sudo mv /home/ec2-user/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf",
      "sudo systemctl enable fluent-bit"
    ]
  }
}

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [N/A] Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[charltonstanley]

Hi there, any chance this could be merged in soon? We have paused updates on our servers because we don't want to have to write an automation script to disable the fluent-bit repository on each instance. (The related bug affects ~125 within my org.) Thanks!