"Unable to load dynamic library" warnings #200
Comments
|
Could it be that this is somehow related to startup speed? Today I was able to reproduce this locally for the very first time. A So is it possible that the PHP extension setup at runtime can fail or be slow? |
|
This is weird. The main command is not supposed to start before the php configuration files have been written. Is anyone else having the issue? Also, @mbrodala, could you share your docker-compose / Dockerfile? Are you doing anything special like redefining the Entrypoint or mounting PHP configuration directories? |
|
@mbrodala Can you reproduce this consistently on your local environment or did it happen only once? |
|
We do not override the entrypoint and also do not mount any PHP configuration. We do everything through the environment variables provided by the images here. I'll try to come up with a minimal example to show the issue. |
|
Thanks! I'm linking to #201 too which might be related (not sure, but I'll need to check this) |
|
I wasn't able to reliably reproduce this locally yet but can confirm that there's something fishy since I've prepended a plain Interestingly |
|
Interesting. I ran into this: https://www.linuxquestions.org/questions/slackware-14/php-7-1-in-testing-%BB-unable-to-load-dynamic-library-mysqli-so-4175618544/ It might be related to the order the Mysql related extensions are loaded. Note to self: I'll have to check this out. |
|
@moufmouf Where you already able to have a look at this? PHP extensions loaded in the wrong order would make more sense to me than the libraries missing in the filesystem ... |
|
Very weird. Here, I am making sure that the "mysqlnd" extensions is loaded if "mysqli" or "pdo_mysql' extensions are loaded. The "phpenmod" command should take care of the priority by prepending the "mysqlnd.ini" file with "10-" (while mysqli.ini is prepended with "20-") When I run this command locally: I am seeing this output: Since modules are loaded in alphabetical order, mysqlnd is loaded first. Could you try running |
|
Here's the output from various builds, unfortunately there is no pattern visible since technically all desired extensions seem to be loaded as requested: SuccessErrorAs you can see the warnings also show up on successful runs but don't cause it to fail. |
|
In both case, the The error is related, but it will be hard to solve if I can't reproduce it. Could you share more about your setup? Are you running "composer" inside a "Dockerfile" or inside "docker-compose"? Any way you can send me a trimmed down version of those files that reproduce the issue? |
|
Sorry for the delays, I am usually in a 2 week sprint with a 2 week break in between. Our setup is spun up with Docker Compose and Composer is then executed via With this there's at least some risk that not all services have fully started yet which we noticed in one project failing to connect to MySQL. I'm not sure if timing is also relevant for the PHP extensions/FPM startup. I'll see if I can manage to get a reproducible setup. Again, this only occurs when running CI builds, never locally. This would support the timing issue theory since the same command runs a lot faster locally. (Both using SSD however.) |
|
I think I am definitely onto something here: I'm currently trying to optimize our CI pipeline (GitLab) e.g. by splitting it into jobs. With this I now see the errors mentioned here in a project which was mostly unaffected as of now. This should mostly stem from the fact that most setup is performed only once and subsequent jobs are directly executed. So the time between environment startup for a job ( A job which runs functional tests against an actual MySQL database fails now:
And:
But a retry of this job interestingly leads to the last bunch of tests to succeed where I assume that the MySQL server finished its startup at this point. Before that it was very likely still starting and preparing the database:
Here's the current state with all logic in a single build job:
I know https://github.com/jwilder/dockerize which I could probably use to override/extend the entrypoint of the PHP image and properly wait for MySQL. But I'm not sure what exactly is going on within the PHP image after container start to have PHP extensions missing up until a certain point. Adding a plain |
|
FYI: I added waiting for MySQL with these two simple changes:
ARG DOCKERIZE_VERSION=v0.6.1
RUN curl -sfLO https://github.com/jwilder/dockerize/releases/download/${DOCKERIZE_VERSION}/dockerize-linux-amd64-${DOCKERIZE_VERSION}.tar.gz \
&& tar -C /usr/local/bin -xzvf dockerize-linux-amd64-${DOCKERIZE_VERSION}.tar.gz \
&& rm dockerize-linux-amd64-${DOCKERIZE_VERSION}.tar.gz
(I tried overriding the However, I still see failures due to |
|
@moufmouf Would you mind adding debugging output to all of your scripts so that a My current suspicion is that Docker (Compose) launches your entrypoint script and happily returns from that. At this points commands could be |
I suspect the error about the PHP Mysql extension is not related to the fact that MySQL listens to connections or not. However, the idea of a race condition between the docker container starting and a "exec" performed on the docker container makes plenty of sense! However, you mention you are using a Gitlab pipeline. This is weird to me because, in my understanding, Gitlab CI is not doing an "exec" but a "docker run". I'm using Gitlab myself and have never run into these problems.
To be honest, I feel a bit reluctant about adding debug info to a container used by so many people. However, I'd be happy to help you adding the logs, rebuilding the container and pushing it on a registry (you can push it on your Gitlab registry) to test any changes! |
Exactly, I just wanted to point out the coincidence of the timing issues here. :-)
We are currently migrating from Shippable to GitLab where we went straight for the shell executor. This allows us to use
And I understand that. But maybe some debug variants would be useful even for other people.
I'll see if I can figure something out by overriding the entrypoint and scripts related to PHP extension management. |
|
@moufmouf Could this be the issue?
Indeed all current errors are related to |
|
This is so, so weird! Yes, this could absolutely be the issue. But the fact is that I'm making sure that The only explanation I could find is that we have a race condition:
I could of course change the order of the array here to make sure mysqlnd is enabled before the other extensions, but that would only change the kind of issue you are seeing (mysqlnd would be enabled, but other extensions would not) The only solution I see around this would be to shadow the "php" command to run the extensions setup BEFORE running the real "php" command itself (basically, I could write a small bash script in place of "php" that read environment variables, configures PHP and then starts PHP). Can you confirm to me that if you put a "sleep" between the "docker-compose up" and the "exec", you are not seeing these errors anymore? |
|
Yes, as mentioned above (#200 (comment)) sleeping for a while already reduces the risk of this error showing up. It doesn't disappear completely since the issue also depends on I/O speed but sleeping for e.g. 30 seconds is not really an option since it would slow down our builds dramatically. One thing I'm not sure about ATM:
This is not the case AFAIS: Here all PHP extensions to enable are requested to be enabled at once. This would lead to the issue linked before that But as you said changing something here probably wouldn't help much: as soon as a PHP process is started while PHP extension setup is not yet finished, it will run with the state of its startup and won't change afterwards unless subprocesses are used. However, I think I was able to avoid the issue for now by doing the following right after So instead of waiting for the library files ( Now the job works just fine:
And one can see that I'm not sure how to fix this properly TBH ... |
|
I also tried to add the Then I did a build (based on Since the fat images are based on the slim images I assumed that one could also prebuild these images with PHP extensions. (If possible I would not use the slim images since then I'd need to install Node/NPM myself.) |
|
After giving this one more thought I found an (embarrassingly simple) solution: Do not use This way the With So no waiting for files via Of course, all other services relevant for execution (e.g. I'll watch this for a while but I'm actually convinced that the issue has been resolved now. In any case thank you @moufmouf for this amazing piece of development for developers. :-) |
|
Thanks for the feedback @mbrodala ! Actually, I'll keep your issue open. I've been beaten by a "somewhat" similar problem. The more I think about it, the more I'm convinced I need to "shadow" the "php" executable behind a simple script that will install the extensions before triggering php itself. |
|
That could probably work but then you'll also need to do the same for |
|
I'm having a similar problem just trying to load the redis extension, with the following error: php-apache-slim_1 | PHP Warning: PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/lib/php/20180731/redis.so (/usr/lib/php/20180731/redis.so: undefined symbol: igbinary_serialize), /usr/lib/php/20180731/redis.so.so (/usr/lib/php/20180731/redis.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
php-apache-slim_1 | PHP: syntax error, unexpected '(' in /etc/php/7.3/cli/conf.d/generated_conf.ini on line 2
php-apache-slim_1 | PHP Warning: PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/lib/php/20180731/redis.so (/usr/lib/php/20180731/redis.so: undefined symbol: igbinary_serialize), /usr/lib/php/20180731/redis.so.so (/usr/lib/php/20180731/redis.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
php-apache-slim_1 | bash: line 2: syntax error near unexpected token `('
php-apache-slim_1 | bash: line 2: `Warning: PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/lib/php/20180731/redis.so (/usr/lib/php/20180731/redis.so: undefined symbol: igbinary_serialize), /usr/lib/php/20180731/redis.so.so (/usr/lib/php/20180731/redis.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0'
ims__php-apache-slim_1 exited with code 2Using the following And the following Anything I'm doing wrong here? |
|
@thiagomeireless Which of these reflect your workflow:
The 1st suffers from the issue mentioned here, the 2nd one will work just fine. |
Both workflows throws the same errors for me: I found a similar issue in another repo and it might be related to |
|
In any case my issue is resolved. |
|
Reopening since this is still an issue when using |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please update it if any action still required. |
|
Yes, it is. |
Sorry, please can you provide a digest about the issue and related information to reproduce ? This issue mixed with v3 (no more supported) so maybe another topic may help to identify the actions that we have to do. |
|
To put it simply, and this applies to
At this point sometimes PHP or the init script is not yet done with setting up all PHP extensions which leads to errors like mentioned here. This also means that the error cannot occur with |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please update it if any action still required. |
|
Yup, still unresolved. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please update it if any action still required. |
|
No update, no fix yet. |
|
I assigned a bug tag to deny the autostale bot. A workaround is to add an healcheck (like it's described here https://medium.com/@saklani1408/configuring-healthcheck-in-docker-compose-3fa6439ee280 ). |
|
I also had the idea to extend the healthcheck accordingly but didn't succeed yet. Also my last attempt would be costly since it would mean invoking a command which e.g. tries to establish a DB connection only to ensure that the necessary DB extensions for PHP are loaded successfully. We'd need something else instead which really queries PHP which extensions are loaded ATM and compares this to the list of requested extensions ... |
mbrodala
changed the title
|
In the meantime we found another workaround which looks promising so far: services:
app:
environment:
STARTUP_COMMAND_DONE: touch ~/.startup_done
healthcheck:
test: ["CMD-SHELL", "test -f ~/.startup_done"]
timeout: 1s
start_interval: 1s
start_period: 10sIn words: create a Background: the current Docker entrypoint performs tasks in a particular order. Here PHP extension setup comes before execution of startup commands. Thus once startup commands run, we can be sure that the PHP extension setup has completed. This is a rather cheap check and heavily depends on the structure of the entrypoint, but I guess it'll do for now. Especially since it's extremely fast and cheap compared to e.g. an actual PHP script which performs tests. |
Expected Behavior
Container PHP usage should work without any warnings and all the time.
Current Behavior
We randomly/often get warnings like these with the PHP containers:
We currently use the
thecodingmachine/php:7.2-v3-fpm-node8image withPHP_EXTENSIONS: gdonly, so these extensions are not disabled intentionally.However, the error sounds more like the extension does not exist at all ...
Context
For some unknown reason this error currently only occurs in our CI environment causing various build failures. So far we haven't been able to trigger this locally.
Your Environment
7.2-v3-fpm-node8Ubuntu 16.04.6 LTSThe text was updated successfully, but these errors were encountered: