Healthcheck not working properly on a prior docker stack.

[KalyaSc]

Hello,

First, pardon me for my lack of education if I'm doing something wrong here. I'm just a hobby tinkerer and self-host enthusiast. I'm writing this issue because I can't find any solution to the unhealthy mariaDB tag of a prior docker-compose stack. While reading through the different issues and proposed solution here and there nothing seems to work as expected.

2024-02-07 13:50:20 93 [Warning] Access denied for user 'healthcheck'@'localhost' (using password: YES)

I followed the official documentation new way for the healthcheck, but that only works on newly created database issuecomment-1610955161.

I also added the new env to recreate healthcheck user, but still the problem persists. I also tried the following proposed workaround but this also throws an error at the first command even If i'm connected as root:

docker exec -it vikunja-db-2-1 sh

# mariadb -uroot -psupersecret
# CREATE USER healthcheck@'127.0.0.1' IDENTIFIED BY '$healthCheckConnectPassEscaped';
> ERROR 1396 (HY000): Operation CREATE USER failed for 'healthcheck'@'127.0.0.1'

The .my-healthcheck.cnf file exists in my docker container:

cat /var/lib/mysql/.my-healthcheck.cnf

[mariadb-client]
port=3306
socket=/run/mysqld/mysqld.sock
user=healthcheck
password=B4}_|J7jd$?AY]xgiBz6}iPghfqxuQ:U
protocol=tcp

If I exec into the container and try to connect as the healthcheck user with the healthcheck.cnf or root password, it doesn't work either:

docker exec -it vikunja-db-2-1 sh

# mariadb -uhealthcheck -pB4}_|J7jd$?AY]xgiBz6}iPghfqxuQ:U
> ERROR 1045 (28000): Access denied for user 'healthcheck'@'localhost' (using password: YES)

# mariadb -uhealthcheck -psupersecret
> ERROR 1045 (28000): Access denied for user 'healthcheck'@'localhost' (using password: YES)

Just running the command into the container itself throws the same error:

# ./usr/local/bin/healthcheck.sh --innodb_initialized 

ERROR 1045 (28000): Access denied for user 'healthcheck'@'127.0.0.1' (using password: YES)
healthcheck innodb_initialized failed

Here is my truncated docker-compose.yaml file:

version: '3'

services:
  api:
    image: vikunja/api:latest
    ...

  db-2:
    image: mariadb:latest
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --max-connections=1000
    environment:
      - MARIADB_AUTO_UPGRADE=1
      - MARIADB_ROOT_PASSWORD=supersecret
      - MYSQL_USER=vikunja
      - MYSQL_PASSWORD=supersecret
      - MYSQL_DATABASE=vikunja

    healthcheck:
      test: [ "CMD", "healthcheck.sh", "--connect", "--innodb_initialized" ]
      start_period: 1m
      start_interval: 10s
      interval: 1m
      timeout: 5s
      retries: 3

I hope I'm not adding unnecessary noise in an already complicated environment I'm absolutely clueless about. But right now I'm out of ideas and explored all issues/workaround I could find. It's probably something on my side I'm doing wrong here and I could live without the healthcheck, but my perfectionist side likes to see the shiny healthy tag 😇.

Thank you !

---

Edit:

I recreated a completely new compose stack with a fresh database and the healthcheck works as expected. So there's probably an issue with the current database but I have no idea on how to fix it.

[grooverdan]

I haven't yet done the release with new #556 yet. Packages have just been released onto the mirrors so 1 or 2 more days and mariadb:latest will be updated.

[KalyaSc]

Ohh my bad :/ ! Thank you for your answer.

[grooverdan]

Up to a few hours after docker-library/official-images#16226 is merged it will be available.

[KalyaSc]

Hello :)

Waited a few days before checking in. At first it didn't worked as expected even with 11.2.3-MariaDB-1:11.2.3+maria~ubu2204 version.

To make it work I had to delete .my-healthcheck.cnf which kinda makes sense because it used the old .cnf file from an older version I guess :).

Now everything is fine, thank you !!

[grooverdan]

Yes. deleting the file is what triggered the resetting of healthcheck users.

Glad its working for you. Thanks for confirming.

[turchinc]

some help for others: I relocated my /var/lib/mysql folder in docker

cd my-mysql-folder
mkdir data
mv * data
# then adjust volume in compose.yml  

after that everything on the server was working but health was not achieved. I did not realize the .my-healthcheck.cnf was located in the data folder and not moved. after additionally mv .my-healthcheck.cnf data then health was again achieved.

[grooverdan]

If you loose the file for some reason MARIADB_AUTO_UPGRADE=1 will recreate it and the password of the healthcheck user.

[Edamijueda]

I resolve this issue by using a GA(General Availability) version of mariadb. In my case image: mariadb:11.6.2

healthcheck:
      test: [ "CMD", "healthcheck.sh", "--connect", "--innodb_initialized" ]
      start_period: 1m
      start_interval: 10s
      interval: 1m
      timeout: 5s
      retries: 3

HealthCheck is now healthy.

[neutronstriker]

If you loose the file for some reason MARIADB_AUTO_UPGRADE=1 will recreate it and the password of the healthcheck user.

Thanks, this helped resolve healthcheck failure issue after moving data from docker volume to a path on the disk.