server returned HTTP status 404 Not Found

[hunter-luo666]

Bug Report

Describe the bug

when we start fluent-bit , can not get prometheus metrics

root@ip-10-70-36-133:~# lsof -i:2021
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
fluent-bi 2164251 root 74u IPv4 19834897 0t0 TCP *:2021 (LISTEN)
fluent-bi 2164251 root 108u IPv4 19834995 0t0 TCP ip-10-70-36-133.ap-northeast-1.compute.internal:2021->ip-10-70-xx-xx.ap-northeast-1.compute.internal:43444 (ESTABLISHED)

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}

• Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

• Version used: Fluent Bit v3.1.3
• Configuration: please see below
• Environment name and version (e.g. Kubernetes? What version?): EC2 Linux 6.8.0-1010-aws Build: fix failed to generate the executable file #9 #10-Ubuntu SMP Thu Jun 13 17:36:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Server type and version:
• Operating System and version: Linux 6.8.0-1010-aws Build: fix failed to generate the executable file #9 #10-Ubuntu SMP Thu Jun 13 17:36:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Filters and plugins:

Additional context

install service by run curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
root@ip-10-70-36-133:~# cat /etc/fluent-bit/fluent-bit.conf
[SERVICE]
flush 1
daemon Off
log_level info
parsers_file parsers.conf
plugins_file plugins.conf
http_server On
http_listen 0.0.0.0
http_port 2020
Health_Check On
HC_Errors_Count 5
HC_Retry_Failure_Count 5
HC_Period 5
storage.metrics on

[INPUT]
name tail
path /data/apache-shardingsphere/logs/.log
path_key log_path
tag sharding-services
[INPUT]
name tail
path /var/log/.log
path_key log_path
tag sharding-system

[INPUT]
Name exec
Tag prod-sharding
Interval_Sec 30
command pg_isready -h 10.70.xxx.xxx -p3307 -U sharding -d sharding_db

[INPUT]
Name proc
Proc_Name java
Interval_Sec 30
Interval_NSec 0
Fd true
Mem true
[INPUT]
name process_exporter_metrics
tag process_metrics
process_include_pattern java
scrape_interval 10

[OUTPUT]
name prometheus_exporter
match process_metrics
host 0.0.0.0
port 2021
add_label app sharding

[OUTPUT]
name loki
match *
host loki.xxx.com
port 443
tls on
labels job=fluentbit, env=prod, namespace=sharding, app=sharding-xxxx

root@ip-10-70-xxx:~# systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; preset: enabled)
Active: active (running) since Sat 2024-09-07 05:08:23 UTC; 10min ago
Docs: https://docs.fluentbit.io/manual/
Main PID: 2164251 (fluent-bit)
Tasks: 10 (limit: 37850)
Memory: 90.6M (peak: 132.7M)
CPU: 10min 31.477s
CGroup: /system.slice/fluent-bit.service
└─2164251 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.0] inotify_fs_add(): inode=6293396 watch_fd=2 name=/data/apa>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3584 watch_fd=1 name=/var/log/alt>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=79203 watch_fd=2 name=/var/log/ap>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3597 watch_fd=3 name=/var/log/aut>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=79146 watch_fd=4 name=/var/log/cl>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=78496 watch_fd=5 name=/var/log/cl>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=21 watch_fd=6 name=/var/log/dpkg.>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=79335 watch_fd=7 name=/var/log/fo>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3083 watch_fd=8 name=/var/log/ker>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3599 watch_fd=9 name=/var/log/ubu>
lines 1-21/21 (END)

root@ip-10-70-36-133:# curl http://10.70.36.133:2021/metrics
root@ip-10-70-36-133:# curl http://10.70.36.133:2021/metrics -I
HTTP/1.1 404 Not Found
Server: Monkey/1.7.2
Date: Sat, 07 Sep 2024 05:20:34 GMT
Transfer-Encoding: chunked

root@ip-10-70-36-133:# curl http://localhost:2021/metrics
root@ip-10-70-36-133:# curl http://localhost:2021/metrics -I
HTTP/1.1 404 Not Found
Server: Monkey/1.7.2
Date: Sat, 07 Sep 2024 05:20:59 GMT
Transfer-Encoding: chunked

root@ip-10-70-36-133:~#

[hunter-luo666]

we have CurrEstab(ESTABLISHED) 43680 , because of it is sharding-proxy instance

[patrick-stephens]

Weird, seems to work fine with the node exporter example from the docs: https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-exporter

Tried with both 3.1.3 and 3.1.7:

docker run --rm -it -p 2021:2021 -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:3.1.7
...
curl localhost:2021/metrics                                                                                        
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="idle"} 8048.1099999999997
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="iowait"} 5.5099999999999998
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="irq"} 0
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="nice"} 0
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="softirq"} 120.58
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="steal"} 0

Can you use log_level debug and check if any warnings or errors reported for opening the port or similar?

What actual Ubuntu package did you install? I've no idea what EC stuff maps to.