Docker restart, wrong queries counts

[manilx]

Latest Docker piholev6 installed.

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - 53:53/tcp
      - 53:53/udp
      - 80:80/tcp
      - 443:443/tcp
    environment:
      TZ: Europe/Lisbon
      FTLCONF_webserver_api_password: password
      FTLCONF_dns_listeningMode: all
    volumes:
      - ./etc-pihole:/etc/pihole
      - ./etc-dnsmasq.d:/etc/dnsmasq.d
    cap_add:
      - NET_ADMIN
    restart: unless-stopped
networks: {}

When I stop and restart pihole, the Total_Queries and Queries_Blocked values are not the last ones before the stop. But from some time back.
They should be saved between restarts....?

[yubiuser]

Yes, on exit FTL should save the queries to the database.
Can you check your docker logs, if the shutdown was gracefully. Like

pihole  | 2025-02-25 19:51:24.510 UTC [54/T91] INFO: Compiled 0 allow and 0 deny regex for 0 client in 0.1 msec
pihole  | 
pihole  |   [i] Container stop requested...
pihole  |   [i] pihole-FTL is running - Attempting to shut it down cleanly
pihole  | 
pihole  | 2025-02-25 19:51:49.818 UTC [54M] INFO: Asked to terminate by "N/A" (PID 167, user root UID 0)
pihole  | 2025-02-25 19:51:49.852 UTC [54/T94] INFO: Terminating timer thread
pihole  | 2025-02-25 19:51:49.887 UTC [54/T91] INFO: Terminating database thread
pihole  | 2025-02-25 19:51:50.069 UTC [54M] INFO: Finished final database update
pihole  | 2025-02-25 19:51:50.069 UTC [54M] INFO: Waiting for threads to join
pihole  | 2025-02-25 19:51:50.069 UTC [54M] INFO: Thread housekeeper (1) is idle, terminating it.
pihole  | 2025-02-25 19:51:50.069 UTC [54M] INFO: Thread dns-client (2) is idle, terminating it.
pihole  | 2025-02-25 19:51:50.069 UTC [54M] INFO: All threads joined
pihole  | 2025-02-25 19:51:50.069 UTC [54M] INFO: PID file emptied
pihole  | 2025-02-25 19:51:50.070 UTC [54M] INFO: Stored 0 API sessions in the database
pihole  | 2025-02-25 19:51:50.457 UTC [54M] INFO: ########## FTL terminated after 26s  (code 0)! ##########
pihole  | 
pihole  |   [i] pihole-FTL exited with status 0
pihole  | 
pihole  |   [i] Container will now stop or restart depending on your restart policy
pihole  |       https://docs.docker.com/engine/containers/start-containers-automatically/#use-a-restart-policy

[manilx]

  [i] Container stop requested...

  [i] pihole-FTL is running - Attempting to shut it down cleanly

2025-02-25 19:57:10.449 WET [52M] INFO: Asked to terminate by "N/A" (PID 7682, user root UID 0)

2025-02-25 19:57:10.485 WET [52/T84] INFO: Terminating database thread

2025-02-25 19:57:10.518 WET [52/T87] INFO: Terminating timer thread

2025-02-25 19:57:10.702 WET [52M] INFO: Finished final database update

2025-02-25 19:57:10.702 WET [52M] INFO: Waiting for threads to join

2025-02-25 19:57:10.702 WET [52M] INFO: Thread housekeeper (1) is idle, terminating it.

2025-02-25 19:57:10.702 WET [52M] INFO: Thread dns-client (2) is idle, terminating it.

2025-02-25 19:57:10.702 WET [52M] INFO: All threads joined

2025-02-25 19:57:10.702 WET [52M] INFO: PID file emptied

2025-02-25 19:57:10.702 WET [52M] INFO: Stored 1 API session in the database

2025-02-25 19:57:11.446 WET [52M] INFO: ########## FTL terminated after 4h 38m 10s  (code 0)! ##########

  [i] pihole-FTL exited with status 0

  [i] Container will now stop or restart depending on your restart policy

      https://docs.docker.com/engine/containers/start-containers-automatically/#use-a-restart-policy

BUT upon restart query stats are not where they were but some time before!

[manilx]

Upon further testing they now are...... Weird

[yubiuser]

Please run a debug log with pihole -d -c and post the token

[manilx]

you mean in the docker container cli?

[yubiuser]

Yes.

[manilx]

https://tricorder.pi-hole.net/VO1Je8Lk/

[yubiuser]

Thanks. I just wanted to rule out that the databases are damaged

*** [ DIAGNOSING ]: Pi-hole FTL Query Database
-rw-r----- 1 pihole pihole 7.9M Feb 25 20:01 /etc/pihole/pihole-FTL.db
[i] Checking integrity of /etc/pihole/pihole-FTL.db ... (this can take several minutes)
[✓] Integrity of /etc/pihole/pihole-FTL.db intact
[i] Checking foreign key constraints of /etc/pihole/pihole-FTL.db ... (this can take several minutes)
[✓] No foreign key errors in /etc/pihole/pihole-FTL.db

*** [ DIAGNOSING ]: Gravity Database
-rw-rw---- 1 pihole pihole 311M Feb 24 22:38 /etc/pihole/gravity.db
[i] Checking integrity of /etc/pihole/gravity.db ... (this can take several minutes)
[✓] Integrity of /etc/pihole/gravity.db intact
[i] Checking foreign key constraints of /etc/pihole/gravity.db ... (this can take several minutes)
[✓] No foreign key errors in /etc/pihole/gravity.db

I have no good idea, I'll need to check myself

[manilx]

I have done numerous tests today and upon restart the counters were way back.... Now 3 tests and OK.
Something is weird ;)

[rdwebdesign]

Note:

If you are waiting for the container to stop, the data should be saved to disk (like your recent tests).

If you kill the container (or pull the plug, or something similar), Pi-hole will be instantly killed and some data will be lost.

Did you really wait for the container to stop every time?

[manilx]

Well you might have something there. Just tested shutting down the VM (running on xcp-ng) and data was lost but the shutdown command just iniciated a OS shutdown (debian) NO power plug kill... So this is weird.

[manilx]

Also when doing a "reboot" from the cli. Data lost

[manilx]

Now I just stopped the docker container and restarted and data was lost.
So it's inconsistent at least.

[manilx]

Looking at the logs: I clean shutdown the container with queries at 15107 and restarting:

2025-02-25 20:32:33.998 WET [53M] INFO: Imported 15089 queries from the on-disk database (it has 15089 rows)

2025-02-25 20:32:33.998 WET [53M] INFO: Parsing queries in database

2025-02-25 20:32:34.051 WET [53M] INFO:   10000 queries parsed...

2025-02-25 20:32:34.083 WET [53M] INFO: Imported 15089 queries from the long-term database

2025-02-25 20:32:34.083 WET [53M] INFO:  -> Total DNS queries: 15089

Queries totals are not all written to database upon shutdown. That's the problem

[yubiuser]

@DL6ER

Is their any circumstance where FTL would not write it's in-memory query log to the database on shutdown?

[manilx]

Stopped Docker with total queries in the 28.000...
Restarted and:

2025-02-25 23:16:03.918 WET [56M] INFO: Database successfully initialized

2025-02-25 23:16:04.185 WET [56M] INFO: Imported 26521 queries from the on-disk database (it has 26521 rows)

2025-02-25 23:16:04.185 WET [56M] INFO: Parsing queries in database

2025-02-25 23:16:04.239 WET [56M] INFO:   10000 queries parsed...

2025-02-25 23:16:04.299 WET [56M] INFO:   20000 queries parsed...

2025-02-25 23:16:04.329 WET [56M] INFO: Imported 26521 queries from the long-term database

The problem is quite obvious now.

I even changed the value of database.DBinterval to 5 in the settings (from 60), thinking that this would help but to no avail.....

[yubiuser]

When I stop and restart pihole, the Total_Queries and Queries_Blocked values are not the last ones before the stop. But from some time back.

Please note: the Total_Queries shown on the web interface are only 'A and AAAA' queries (see the mouse overlay). There might be other query types in your datatbase.

[yubiuser]

Wait.... the overlay might be wrong in v6

[yubiuser]

The overlay is wrong + I can confirm your bug.

I startet my bare-metal installations 2 times in a row and after each restart the number of queries reported from the API were lower

[manilx]

There you go. But in the meantime I switched again to Adguard Home.... Easier to deploy and maintain.
Will retest iPhole v6 after bug fixed.

[DL6ER]

Sorry, I missed this before. There is no real bug here, it is expected that the numbers are slightly different. The reason for this is efficiency. Pi-hole has been designed to use as few resources as possible. Part of this is that the total number of queries (as well as some other global sums) is not always 100% precise but only updated/recomputed every 10 minutes. This allows the API to deliver these numbers in sub-milliseconds and not needing to count on every API call. Hence, the numbers you are seeing shortly before shutdown of your container may have not been recomputed for ... say ... 5 minutes. Then, on the next start of the container, the sum obviously as to be computed afresh as we do not back up such meta-information in the database. This newly computed number is then expected to be slightly different. It is logical to be slightly lower (up about to 1% if you have a perfectly uniform querying behavior in your network) but it could even be higher if you are seeing a massive load of queries immediately after the restart.

TL;DR: There is nor a bug nor actually an inconsistency here. The global sums are recomputed not very often to be light on resources but a restart will force them to be recalculated. It is expected that they are slightly different by up to 1%.

[manilx]

Thanks for the explanation. But I don't think this should be as it is now.... I for one don't like it at all.

[DL6ER]

But why? As I said above, the total number is an estimate and, as such, it changes all the time. Recomputing it every time will just eat up resources without any real gain IMO. Why do you think it is worth the extra costs to always recompute it exactly each time?

If you need the exact numbers for something in your network, I'd propose you directly source this number from the database instead.

[manilx]

I heard you. I wouldn't even know to get it from the database....

Why don't you implement something like this: A Klick on the totaliers will recompute and show the current value.

[yubiuser]

Maybe we could compute the number on shutdown?

[DL6ER]

But what would we gain? When we compute it at the very end, it will immediately be lost. And recomputed anyway on the next start. We could theoretically add a signal and/or API action endpoint that triggers an immediate re-counting but it'd require careful testing what running this frequently causes to the overall performance of FTL.