Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTL Crashes if FTLCONF_webserver_api_password has a value set #2242

Open
uurf opened this issue Feb 21, 2025 · 54 comments
Open

FTL Crashes if FTLCONF_webserver_api_password has a value set #2242

uurf opened this issue Feb 21, 2025 · 54 comments

Comments

@uurf
Copy link

uurf commented Feb 21, 2025

Versions

  • Core version is v6.0.3 (Latest: v6.0.3)
  • Web version is v6.0.1 (Latest: v6.0.1)
  • FTL version is v6.0.1 (Latest: v6.0.1)

Platform

  • OS and version:
  • Platform: Device: Synology DS1817+
    DSM 7.2.2-72806 Update 3
    Docker Compose via Synology Container Manager 20.10.23-1437

Expected behavior

Having set a password using FTLCONF_webserver_api_password: '<password>' I expect the web service to load on https://<adress>:8181/admin and to be challenged for the password I set.

Actual behavior / bug

FTL Crashes.

Steps to reproduce

Steps to reproduce the behavior:

  1. build a docker container using this compose code:
name: pihole6
services:
    pihole:
        container_name: Pi-Hole
        security_opt:
            - no-new-privileges=false
        volumes:
            # For persisting custom dnsmasq config files. Most will not need this, and can be safely removed/commented out
            - /volume1/docker/pihole/dnsmasq.d:/etc/dnsmasq.d:rw
            # For persisting Pi-hole's databases and common configuration file
            - /volume1/docker/pihole/pihole:/etc/pihole:rw
     environment:
            # FTLCONF_webserver_api_password: '<password>'
            FTLCONF_webserver_port: '8181s'
            # Set the appropriate timezone for your location (https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), e.g:
            TZ: 'America/New York'
            
        cap_add:
            - SYS_NICE
        restart: on-failure:5
        network_mode: host
        image: pihole/pihole:latest
  1. observe in the container logs that pihole has started.
  2. load https://<adress>:8181/admin
  3. observe in the logs that FTL has crashed. Admin UI does not load

Debug Token

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

if I comment out # FTLCONF_webserver_api_password: '' it starts and the FTL ui is accessible where expected, but there's no password. Not even a randomly generated one.

I am able to set a password via pihole setpassword but it doesn't seem to be used - after setting it the admin UI still loads without a login challenge.

@uurf
Copy link
Author

uurf commented Feb 21, 2025

crash log

2025-02-21 17:36:34.485 UTC [56M] INFO: Reading certificate from /etc/pihole/tls.pem ...
2025-02-21 17:36:34.486 UTC [56M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
2025-02-21 17:36:34.486 UTC [56M] INFO: Web server ports:
2025-02-21 17:36:34.486 UTC [56M] INFO:   - 8181 (HTTPS, IPv4)
2025-02-21 17:36:34.487 UTC [56M] INFO: Restored 0 API sessions from the database
2025-02-21 17:36:34.721 UTC [56M] ERROR: getrandom() failed in generate_password()
2025-02-21 17:36:34.721 UTC [56M] ERROR: Failed to generate CLI password hash!
2025-02-21 17:36:34.722 UTC [56M] INFO: Blocking status is enabled
2025-02-21 17:36:34.823 UTC [56/T172] INFO: Compiled 0 allow and 13 deny regex for 5 clients in 18.6 msec
2025-02-21 17:36:38.498 UTC [56/T174] ERROR: Cannot receive UDP DNS reply: Timeout - no response from upstream DNS server
2025-02-21 17:36:38.498 UTC [56/T174] INFO: Tried to resolve PTR "1.1.168.192.in-addr.arpa" on 127.0.0.1#53 (UDP)
2025-02-21 17:36:40.498 UTC [56/T174] ERROR: Cannot receive UDP DNS reply: Timeout - no response from upstream DNS server
2025-02-21 17:36:40.498 UTC [56/T174] INFO: Tried to resolve PTR "234.1.168.192.in-addr.arpa" on 127.0.0.1#53 (UDP)
2025-02-21 17:36:42.499 UTC [56/T174] ERROR: Cannot receive UDP DNS reply: Timeout - no response from upstream DNS server
2025-02-21 17:36:42.499 UTC [56/T174] INFO: Tried to resolve PTR "198.1.168.192.in-addr.arpa" on 127.0.0.1#53 (UDP)
2025-02-21 17:43:06.616 UTC [56/T316] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-02-21 17:43:06.616 UTC [56/T316] INFO: ---------------------------->  FTL crashed!  <----------------------------
2025-02-21 17:43:06.617 UTC [56/T316] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-02-21 17:43:06.617 UTC [56/T316] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2025-02-21 17:43:06.617 UTC [56/T316] INFO: and include in your report already the following details:
2025-02-21 17:43:06.617 UTC [56/T316] INFO: FTL has been running for 402 seconds
2025-02-21 17:43:06.617 UTC [56/T316] INFO: FTL branch: master
2025-02-21 17:43:06.617 UTC [56/T316] INFO: FTL version: v6.0.1
2025-02-21 17:43:06.617 UTC [56/T316] INFO: FTL commit: 62904aef
2025-02-21 17:43:06.617 UTC [56/T316] INFO: FTL date: 2025-02-20 23:07:18 +0000
2025-02-21 17:43:06.617 UTC [56/T316] INFO: FTL user: started as pihole, ended as pihole
2025-02-21 17:43:06.617 UTC [56/T316] INFO: Compiled for linux/amd64 (compiled on CI) using cc (Alpine 14.2.0) 14.2.0
2025-02-21 17:43:06.617 UTC [56/T316] INFO: Process details: MID: 56
2025-02-21 17:43:06.617 UTC [56/T316] INFO:                  PID: 56
2025-02-21 17:43:06.618 UTC [56/T316] INFO:                  TID: 316
2025-02-21 17:43:06.618 UTC [56/T316] INFO:                  Name: civetweb-worker
2025-02-21 17:43:06.618 UTC [56/T316] INFO: Received signal: Segmentation fault
2025-02-21 17:43:06.618 UTC [56/T316] INFO:      at address: 0
2025-02-21 17:43:06.618 UTC [56/T316] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2025-02-21 17:43:06.618 UTC [56/T316] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2025-02-21 17:43:06.618 UTC [56/T316] INFO: ------ Listing content of directory /dev/shm ------
2025-02-21 17:43:06.618 UTC [56/T316] INFO: File Mode User:Group      Size  Filename
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rwxrwxrwx root:root       360   .
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rwxr-xr-x root:root       340   ..
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rw------- pihole:pihole   786K  FTL-56-recycler
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rw------- pihole:pihole    20K  FTL-56-dns-cache-lookup
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rw------- pihole:pihole    33K  FTL-56-domains-lookup
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rw------- pihole:pihole     4K  FTL-56-clients-lookup
2025-02-21 17:43:06.619 UTC [56/T316] INFO: rw------- pihole:pihole   569K  FTL-56-fifo-log
2025-02-21 17:43:06.620 UTC [56/T316] INFO: rw------- pihole:pihole     4K  FTL-56-per-client-regex
2025-02-21 17:43:06.620 UTC [56/T316] INFO: rw------- pihole:pihole   102K  FTL-56-dns-cache
2025-02-21 17:43:06.620 UTC [56/T316] INFO: rw------- pihole:pihole     8K  FTL-56-overTime
2025-02-21 17:43:06.620 UTC [56/T316] INFO: rw------- pihole:pihole    11M  FTL-56-queries
2025-02-21 17:43:06.620 UTC [56/T316] INFO: rw------- pihole:pihole    29K  FTL-56-upstreams
2025-02-21 17:43:06.621 UTC [56/T316] INFO: rw------- pihole:pihole   348K  FTL-56-clients
2025-02-21 17:43:06.621 UTC [56/T316] INFO: rw------- pihole:pihole   131K  FTL-56-domains
2025-02-21 17:43:06.621 UTC [56/T316] INFO: rw------- pihole:pihole   164K  FTL-56-strings
2025-02-21 17:43:06.622 UTC [56/T316] INFO: rw------- pihole:pihole   144   FTL-56-settings
2025-02-21 17:43:06.622 UTC [56/T316] INFO: rw------- pihole:pihole   328   FTL-56-counters
2025-02-21 17:43:06.622 UTC [56/T316] INFO: rw------- pihole:pihole    88   FTL-56-lock
2025-02-21 17:43:06.624 UTC [56/T316] INFO: ---------------------------------------------------
2025-02-21 17:43:06.624 UTC [56/T316] INFO: Please also include some lines from above the !!!!!!!!! header.
2025-02-21 17:43:06.624 UTC [56/T316] INFO: Thank you for helping us to improve our FTL engine!
2025-02-21 17:43:06.624 UTC [56/T316] INFO: Waiting for threads to join
2025-02-21 17:43:06.624 UTC [56/T316] INFO: Thread database (0) is idle, terminating it.
2025-02-21 17:43:06.625 UTC [56/T316] INFO: Thread housekeeper (1) is idle, terminating it.
2025-02-21 17:43:06.625 UTC [56/T316] INFO: Thread dns-client (2) is idle, terminating it.
2025-02-21 17:43:06.625 UTC [56/T316] INFO: Thread timer (3) is idle, terminating it.
2025-02-21 17:43:06.625 UTC [56/T316] INFO: All threads joined
2025-02-21 17:43:06.626 UTC [56/T316] INFO: PID file emptied
2025-02-21 17:43:06.627 UTC [56/T316] INFO: Stored 0 API sessions in the database
2025-02-21 17:43:09.242 UTC [56M] INFO: Finished final database update
2025-02-21 17:43:09.242 UTC [56M] INFO: Waiting for threads to join
2025-02-21 17:43:09.242 UTC [56M] INFO: Thread database (0) is idle, terminating it.
2025-02-21 17:43:09.243 UTC [56M] INFO: Thread housekeeper (1) is idle, terminating it.
2025-02-21 17:43:09.243 UTC [56M] INFO: Thread dns-client (2) is idle, terminating it.
2025-02-21 17:43:09.243 UTC [56M] INFO: Thread timer (3) is idle, terminating it.
2025-02-21 17:43:09.243 UTC [56M] INFO: All threads joined
2025-02-21 17:43:09.243 UTC [56M] INFO: PID file emptied
2025-02-21 17:43:09.709 UTC [56M] INFO: ########## FTL terminated after 6m 44s  (code 0)! ##########```

@yubiuser yubiuser transferred this issue from pi-hole/pi-hole Feb 21, 2025
@yubiuser
Copy link
Member

Might be fixed by #2240

@vitaliy-sk
Copy link

Same issue on Linux nas 3.10.108 #42962 SMP Mon May 29 14:36:16 CST 2023 x86_64 GNU/Linux synology_bromolow_3615xs

2025-02-21 23:36:06.864 GMT [58M] ERROR: getrandom() failed in generate_password()
2025-02-21 23:36:06.865 GMT [58M] ERROR: Failed to generate CLI password hash!

It's not crashing, but UI isn't asking for password event if it was set before migration to v6

@DL6ER
Copy link
Member

DL6ER commented Feb 22, 2025

Please try again after running

sudo pihole checkout ftl development

to get the latest bug fixing attempts for this problem. The issue here is that your VMs somehow do not provide cryptographically sound randomness and we need find out why.

@GDT1978
Copy link

GDT1978 commented Feb 23, 2025

I have the same issue with my installation in a Synology DS415+. If I configure my docker with a blank password I was able to open the admin page.

@vitaliy-sk
Copy link

@DL6ER thanks, tried, but it's not working on Docker

1ad31eb40277:/# sudo !!
sudo pihole checkout ftl development
Function not supported in Docker images

also tried pihole/pihole:nightly image but no luck

@yubiuser
Copy link
Member

If your are on docker, you need to build the image yourself.

https://github.com/pi-hole/docker-pi-hole?tab=readme-ov-file#building-the-image-locally

./build.sh -f development

@uurf
Copy link
Author

uurf commented Feb 24, 2025

While I think it's within my capabilities to build locally, I'm not sure how to use a local image with my pihole container project, and deploy it reliably within my environment.

@yubiuser
Copy link
Member

If you use the build script, it will create an image tagged pihole:local

You can simply change one line in your docker compose file to load this instead of the one from dockers repository.

services:
  pihole:
    container_name: pihole_local
    image: pihole:local

@uurf
Copy link
Author

uurf commented Feb 24, 2025

If you use the build script, it will create an image tagged pihole:local

You can simply change one line in your docker compose file to load this instead of the one from dockers repository.

services:
  pihole:
    container_name: pihole_local
    image: pihole:local

where does the image live? just put it in the docker/pihole directory?

@uurf
Copy link
Author

uurf commented Feb 24, 2025

I did grab the latest Docker tag this morning but it still crashes FTL if I set a password in the docker container script, doesn't generate its own password if I don't and runs no password in that situation. Looks like the latest docker image is still behind Core version.

Core version is v6.0.3 (Latest: v6.0.4)
Web version is v6.0.1 (Latest: v6.0.1)
FTL version is v6.0.2 (Latest: v6.0.2)```

@yubiuser
Copy link
Member

where does the image live?

It lives in your local host's repo rigth on your own machine.

@PromoFaux
Copy link
Member

latest docker image is still behind Core version.

Just tagged a new one now, should be about 10 mins. I didn't tag before because there wasn't anything massively relevant to docker in the core hotfix.

where does the image live?

If you just run docker images on your CLI - it should be listed there among any others you have downloaded

@GDT1978
Copy link

GDT1978 commented Feb 24, 2025 via email

@PromoFaux
Copy link
Member

Sorry, no it wont be, the fix is not in those tags, rather still in development.

Please try with the nightly tag instead of latest, or per @yubiuser's instructions above, clone the docker repo and build a local image with ./build.sh -f development

@vitaliy-sk
Copy link

@PromoFaux tried on nightly, function in error message is different, but still failing

pihole    | 2025-02-25 22:55:34.775 GMT [61M] ERROR: getrandom() failed in get_secure_randomness(): Function not implemented
pihole    | 2025-02-25 22:55:34.775 GMT [61M] ERROR: Failed to generate CLI password hash!
pihole    |   [i] Version info:
pihole    |       Core
pihole    |           Version is 9840b42 (Latest: null)
pihole    |           Branch is development
pihole    |           Hash is 9840b428 (Latest: 002536ae)
pihole    |       Web
pihole    |           Version is 3cdce06 (Latest: null)
pihole    |           Branch is development
pihole    |           Hash is 3cdce060 (Latest: 22bfbd9d)
pihole    |       FTL
pihole    |           Version is vDev-acec7a8 (Latest: null)
pihole    |           Branch is development
pihole    |           Hash is acec7a86 (Latest: a0e4cc91)
nightly: Pulling from pihole/pihole
Digest: sha256:0d19b5dd748ac36e2c332c2423b32006ddd09f3497a73f290035fe15176ea189

@PromoFaux
Copy link
Member

  • DSM 7.2.2-72806 Update 3
    Docker Compose via Synology Container Manager 20.10.23-1437

If everyone here is on Synology, what versions are you running? @uurf is the above.

I am running OK on a DS218+ with:

  • DSM 7.2.1-69057 Update 6
  • Synology Container Manager 24.0.2-1535 (container configured and managed via Portainer)

@vitaliy-sk
Copy link

DSM 7.1.1-42962 Update 8

Docker is from synology store:
Docker version 20.10.3, build 55f0773

@UnnaturalFlavors
Copy link

UnnaturalFlavors commented Feb 27, 2025

Same issue here with an older Synology:

DS1515+
Intel Atom C2538 ("bandaided", still limping along)
DSM 7.1.1-42962 Update 8

Docker version 20.10.3, build 55f0773

It appears to crash immediately when hitting FTL on port 443 visiting /admin, forced or otherwise. I'm planning on forcing FTL to port 80 to confirm but I did notice that http://pi.hole http(s)://<Server IP> (with no /admin) does indeed work and presents a 403 page linking to /admin.

My original thinking was an internal network loop (with FTL) but even after tweaking the compose file with multiple network flavors, the results were always the same.

One thing to note, and this could just be a bug with our version of Docker, the PORTS column under the docker ps command is empty. I can't recall if this was normally the case with other containers and this version of Docker/Synology:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bdc3f902c9a4 pihole/pihole:latest "start.sh" 13 hours ago Up 13 hours (healthy) pihole

Edit: Another thing to note- the container crashes as well, which could be triggered by FTL's crash.:

bdc3f902c9a4 pihole/pihole:latest "start.sh" 14 hours ago Up 7 seconds (health: starting) pihole

@PromoFaux
Copy link
Member

@DL6ER does this (changed) error message from the comment 4 above from here indicate a new direction to look in?

pihole    | 2025-02-25 22:55:34.775 GMT [61M] ERROR: getrandom() failed in get_secure_randomness(): Function not implemented

@UnnaturalFlavors - are you seeing the same issue with getRandom()?

@UnnaturalFlavors

This comment has been minimized.

@PromoFaux

This comment has been minimized.

@UnnaturalFlavors
Copy link

In testing I did discover that visiting http(s)://pi.hole had a forced redirect to /admin whereas http(s)://<Server IP> did not. The latter presented the 403 page and the prior seems to follow the same crash issue. Visiting http(s)://<Server IP>/admin crashes too.

I've flipped this thing upside down so many times that it's possible I was misremembering seeing the 403 page under the pi.hole URL.

@rdwebdesign
Copy link
Member

@UnnaturalFlavors

The latter presented the 403 page

Can you post a screenshot of the page? It looks like lighttpd is still running on port 80.

@UnnaturalFlavors
Copy link

UnnaturalFlavors commented Feb 27, 2025

Image

This is from a clean docker image pulled from pihole:development with no other services running on that docker host or IP. Screenshot taken visiting http(s)://<Server IP>

> docker exec pihole lsof
1       /bin/bash       0       /dev/null
1       /bin/bash       1       pipe:[22631705]
1       /bin/bash       2       pipe:[22631706]
1       /bin/bash       255     /usr/bin/start.sh
18      /bin/busybox    0       /dev/null
18      /bin/busybox    1       /dev/null
18      /bin/busybox    2       /dev/null
18      /bin/busybox    3       socket:[22623966]
167     /usr/bin/coreutils      0       /dev/null
167     /usr/bin/coreutils      1       pipe:[22631705]
167     /usr/bin/coreutils      2       pipe:[22631706]
167     /usr/bin/coreutils      3       /var/log/pihole/FTL.log
167     /usr/bin/coreutils      4       anon_inode:inotify
> docker exec pihole netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.11:53966        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:http            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:domain          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:https           0.0.0.0:*               LISTEN
tcp        0      0 :::http                 :::*                    LISTEN
tcp        0      0 :::domain               :::*                    LISTEN
tcp        0      0 :::https                :::*                    LISTEN
udp        0      0 127.0.0.11:54477        0.0.0.0:*
udp        0      0 pihole:34933            pihole:domain           ESTABLISHED
udp        0      0 0.0.0.0:domain          0.0.0.0:*
udp        0      0 0.0.0.0:ntp             0.0.0.0:*
udp        0      0 0.0.0.0:8472            0.0.0.0:*
udp        0      0 :::domain               :::*
udp        0      0 :::ntp                  :::*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  3      [ ]         STREAM     CONNECTED     22623998
unix  2      [ ]         DGRAM                    22623966
unix  3      [ ]         STREAM     CONNECTED     22623999

To add, this is a vanilla set up with no proxy or additional layers. Just a Docker container plopped onto an older Synology.

@rdwebdesign
Copy link
Member

This is expected.

You need to visit: http(s)://<Server IP>/admin

@UnnaturalFlavors
Copy link

Agreed, just mentioning here to showcase that the webserver is pushing content appropriately (at least on my instance) until visiting /admin 👍

@UnnaturalFlavors
Copy link

UnnaturalFlavors commented Feb 27, 2025

Apologies to keep spamming this thread but it does seem similar to #2288 and others listed in that thread regarding a segment fault (which is the same or similar to what I'm getting) and it got me thinking. After somewhat diving into the code and following the chain, during the initial processing there's generation of certificate(s) and x509 is listed as a key.

While slamming my face against the keyboard to test the host instance of openssl, I got the following error while running the following:

> openssl req -x509 -newkey rsa:2048 -keyout delete_me.pem -out delete_me_cert.pem -days 365
Can't load /root/.rnd into RNG
139764537488448:error:2406F079:random number generator:RAND_load_file:Cannot open file:crypto/rand/randfile.c:98:Filename=/root/.rnd

And to share:

OpenSSL> version
OpenSSL 1.1.1t  7 Feb 2023

I swear I remember reading or seeing something about older Synology and crypto issues but haven't taken any time to poke the search engines. Now it's certainly not definitive but could at least be part of the overall culprit to the issues we're experiencing. The type of processor could also play a part:

@uurf and I are both running Synology units with the Intel Atom C2538 Quad Core 2.4GHz processor on DS1817+ and DS1515+ respectively

@vitaliy-sk is running I believe the Intel Core i3-4130 Dual Core 3.4GHz on a DS3615xs (very nice)

@PromoFaux is running the Realtek RTD1296 quad-core 1.4GHz on a DS218+

There's a commonality between the processor that is working and the ones that aren't (Realtek vs Intel) with the Intel chips ~2013 and the Realtek ~2018 (best guess based on data found), there should be a fundamental difference in feature sets with supported cryptography being one but also backwards compatibility (not to mention architecture, x86 vs ARM).

Just a hunch so take the above with a grain of salt but considering we're all receiving similar errors regarding getrandom and hashes along with the hard crashes, there might be something to it.

Edit: To add, the openssl command does create the appropriate files but whether they are correct, I didn't go to that length yet.

A quick search shows that you can manually create the .rnd file (successfully) into /root using the following and it eliminates the "Can't load..." error above while using openssl:

sudo openssl rand -writerand /root/.rnd

Edit 2: Using certtool, I was able to verify the pem cert that was created using the above is valid (or at least properly structured):

> certtool -i < delete_me_cert.pem
X.509 Certificate Information:
        Version: 3
        ...
        Algorithm Security Level: Medium (2048 bits)
        ...

@UnnaturalFlavors
Copy link

For the missing getrandom function, I don't see libc inside the image's library (at least the nightly/development pulls). Unless it's under a different .so or mixed in with a compiled package?

> docker exec pihole find . -name "libc*"
./lib/libc.musl-x86_64.so.1
./lib/libcrypto.so.3
./lib/libcom_err.so.2
./lib/libcom_err.so.2.1
./sys/module/libcrc32c
./usr/lib/libcrypto.so.3
./usr/lib/libcap.so.2
./usr/lib/libcap.so.2.69
./usr/lib/libcares.so.2
./usr/lib/libcares.so.2.12.0
./usr/lib/libctf-nobfd.so.0
./usr/lib/libctf-nobfd.so.0.0.0
./usr/lib/libctf.so.0
./usr/lib/libctf.so.0.0.0
./usr/lib/libcurl.so.4
./usr/lib/libcurl.so.4.8.0

@DL6ER
Copy link
Member

DL6ER commented Feb 28, 2025

FTL is an all-static batteries-included binary that can run on any (even ancient) operating system regardless of the installed libc. To achieve this, we use a static variant of https://musl.libc.org/, there is no dynamic linkage at all to avoid dependency issues. This is relevant for Pi-hole as users install and use it on operating systems with older libc which in the past limited is in what we could use as to not break compatibility with them (or going the extra lengths of providing individual build per architecture and per glibc version)

@DL6ER
Copy link
Member

DL6ER commented Feb 28, 2025

Having said that, please also see my comment here: #2281 (comment)

It is worth trying this if you still see the issue. I started working on a software-entropy source - initialized at the very beginning of the pihole-FTL process to ensure it had time to collect some real entropy when it is needed - as fallback when the hardware-generator isn't available. But patching the crypto libraries (DNSSEC libnettle, TLS mbedtls) will be a lot less fun... I will continue working on this today after returning from the office (I am just on my way to there).

@PromoFaux

This comment has been minimized.

@PromoFaux
Copy link
Member

Lets try and put a bit of a spreadsheet together here to capture the range of this issue.

Link below is open to edit by anyone with link, please add to it if you are running on a Synology device (even if not facing this issue!)

https://docs.google.com/spreadsheets/d/13crEk428bO87JOkIhT6xbX7MU6QQCCJONzP9xU3yxo8/edit?usp=sharing

@PromoFaux

This comment has been minimized.

@PromoFaux
Copy link
Member

PromoFaux commented Feb 28, 2025

Image

Seems like it is an issue with Kernel versions < 3.16:

bugs.debian.org/cgi-bin/bugreport.cgi?bug=978045

(apparently Synology does not update the kernel even when new versions of DSM are released: https://www.reddit.com/r/synology/comments/y5nzm9/comment/it12gf8/

@DL6ER
Copy link
Member

DL6ER commented Feb 28, 2025

ERROR: getrandom() failed in get_secure_randomness(): Function not implemented

"Function not implemented" = ENOSYS means that there is no syscall to get randomness, because the kernel running on the machine is lacking it. The minimum required kernel for the getrandom syscall is 3.17 so what @PromoFaux already assumed seems to be true here. All the reported affected Synology machines running so have full hardware randomness support in the CPU itself, it is just that the ancient kernel is not offering it to the operating system.

It really seems to be a Synology issue, I have found other reports all over the web for various projects

Some users reported (for other projects) that upgrading their kernels helped but I guess that's no option for Synology?

I will come back to you later with a proposed fix how to source "sufficiently good" randomness even without hardware randomness support.

@Song-So-Mi
Copy link

Song-So-Mi commented Feb 28, 2025

I also have the same issue with a DS716+II on 7.2.2-72806 Update 3 with an INTEL Celeron N3160.

@PromoFaux

This comment has been minimized.

@DL6ER
Copy link
Member

DL6ER commented Feb 28, 2025

I pushed my proposed suggestion after doing careful randomness quality tests of all involved fallback layers. Please verify if FTL's branch fix/randomness_crash fixes this for you. I do wonder if building new images directly on the machine will work here @PromoFaux ? Or what would you suggest doing in this case?

@Song-So-Mi

This comment has been minimized.

@PromoFaux
Copy link
Member

I would have thought it would be fine to build locally per normal instructions - there is one way to find out!

But I will also build and push a special tag for x86_64

@PromoFaux
Copy link
Member

OK I have pushed a special tag for this: pihole/pihole:synology-crash

@uurf
Copy link
Author

uurf commented Feb 28, 2025

OK I have pushed a special tag for this: pihole/pihole:synology-crash

I will give this a try later this evening.

@UnnaturalFlavors
Copy link

@PromoFaux Good catch! I took specs from the wrong model (oops):
https://global.download.synology.com/download/Document/Hardware/DataSheet/DiskStation/18-year/DS218/enu/Synology_DS218_Data_Sheet_enu.pdf

The non-plus version apparently uses the Realtek CPU, interesting.

Pulling the special tag now 👍

@UnnaturalFlavors
Copy link

UnnaturalFlavors commented Feb 28, 2025

I rebuilt the image with the API password commented out just to make sure things were running but still have a password set via sudo pihole setpassword string

This is a good sign! We weren't even getting this page before:
Image

And utilizing https:
Image

Confirmed removing the password via CLI allowed immediate access into the site (as was our only means previously).

🥁

Removed the commented API password field in my compose, removed and rebuilt the container again and the first page to appear is correct!:
Image

And after using the set password (*not the CLI command, the static password in the compose), successfully logged in, well done!
Image

@UnnaturalFlavors
Copy link

UnnaturalFlavors commented Feb 28, 2025

Not specifically related to this issue (besides the consistent crashes 😂) but it looks like there's now db issues on my instance. The system isn't technically production so a wipe isn't out of the question but was working at least yesterday at some point per the logs.

Just a heads up if anyone else with a kernel-laden Synology had the crash issue and didn't back up the databases prior to the upgrade.

FYI, the issues below are not the fault of the special tag pihole/pihole:synology-crash image but most likely the many many crashes I put my system through.

Image

2025-02-28 12:18:44.636 ERROR SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-02-28 12:18:44.636 ERROR gravityDB_open("SELECT(... vw_gravity ...)") - SQL error prepare: SQL logic error

I'll make a copy of that db and dive into it later.

@UnnaturalFlavors
Copy link

Browsing through the FTL log and noticed this which I'm assuming was one of the workarounds for the getrandom/kernel issue:

2025-02-28 13:39:27.362 INFO B
2025-02-28 13:39:27.362 WARNING Getting secure randomness failed (Function not implemented), trying /dev/urandom
2025-02-28 13:39:27.363 INFO Fallback to /dev/urandom successful
2025-02-28 13:39:27.363 DEBUG_ANY Generated 24 bytes of secure randomness

Sounds like a terrible pickup line:

My heart generated 24 bytes of secure randomness and it's all yours

@DL6ER
Copy link
Member

DL6ER commented Feb 28, 2025

The last line will not be shown if you disable all debug options.

@vitaliy-sk
Copy link

vitaliy-sk commented Feb 28, 2025

@PromoFaux thanks, working as expected with pihole/pihole:synology-crash image

First start took 3 mins due to DB migration, next start is ok

(first start logs)

pihole    | 2025-02-28 20:22:47.954 GMT [51M] INFO: Database version is 21
pihole    | 2025-02-28 20:22:47.955 GMT [51M] INFO: Database successfully initialized
pihole    | 2025-02-28 20:26:37.648 GMT [51M] INFO: Imported 0 queries from the on-disk database (it has 5559994 rows)
pihole    | 2025-02-28 20:26:37.649 GMT [51M] INFO: Parsing queries in database
pihole    | 2025-02-28 20:26:37.649 GMT [51M] INFO: Imported 0 queries from the long-term database

Fallback to /dev/urandom is working

pihole    | 2025-02-28 20:29:22.841 GMT [50M] WARNING: Getting secure randomness failed (Function not implemented), trying /dev/urandom
pihole    | 2025-02-28 20:29:22.841 GMT [50M] INFO: Fallback to /dev/urandom successful

Tested Web UI with and without password, and HTTP and HTTPs are also working

@LukiLeu
Copy link

LukiLeu commented Mar 1, 2025

OK I have pushed a special tag for this: pihole/pihole:synology-crash

I tested this image also on my Synology and it fixes the crashes.

@uurf
Copy link
Author

uurf commented Mar 1, 2025

OK I have pushed a special tag for this: pihole/pihole:synology-crash

on my instance:

  1. if a password is not set with FTLCONF_webserver_api_password: '<password>' a random one is successfully generated.
  2. if a password is set with FTLCONF_webserver_api_password: '<password>', the password is used and the admin page is accessible without crashing FTL.
  3. in either case you are challenged with a password login when loading /admin

@joncamfield
Copy link

joncamfield commented Mar 2, 2025

This hasn't resolved what I had hoped was a related problem of the set webpassword not being recognized , but did get rid of the randomness errors in the log, but a sudo pihole setpassword is at least working

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests