mysql 5.7 docker container times out

[stephenfin]

I'm trying to integrate tox-docker with Patchwork The main reason we want docker is to provide databases for us. I have a draft up, with the main change happening here, and I'm seeing some issues with the functions that test for connectivity. For the mysql:5.7 container, I'm seeing the following:

$ tox -e py27-django111-mysql                                                                                                                                                                                      
py27-django111-mysql installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,attrs==19.1.0,backports.functools-lru-cache==1.5,backports.functools-partialmethod==3.5.1.0,Django==1.11.21,django-filter==1.1.0,djangorestframework==3.9.4,enum34==1.1.6,functools32==3.2.3.post2,jsonschema==3.0.1,lazy-object-proxy==1.4.1,mysqlclient==1.3.14,openapi-core==0.8.0,openapi-spec-validator==0.2.7,pathlib2==2.3.3,pyrsistent==0.15.2,python-dateutil==2.8.0,pytz==2019.1,PyYAML==5.1,scandir==1.10.0,six==1.12.0,sqlparse==0.3.0,strict-rfc3339==0.7
py27-django111-mysql docker: run 'mysql:5.7'
Traceback (most recent call last):
  File "/usr/bin/tox", line 11, in <module>
    load_entry_point('tox==3.5.3', 'console_scripts', 'tox')()
  File "/usr/lib/python3.7/site-packages/tox/session.py", line 42, in cmdline
    main(args)
  File "/usr/lib/python3.7/site-packages/tox/session.py", line 49, in main
    retcode = build_session(config).runcommand()
  File "/usr/lib/python3.7/site-packages/tox/session.py", line 457, in runcommand
    return self.subcommand_test()
  File "/usr/lib/python3.7/site-packages/tox/session.py", line 595, in subcommand_test
    self.runtestenv(venv)
  File "/usr/lib/python3.7/site-packages/tox/session.py", line 616, in runtestenv
    self.hook.tox_runtest_pre(venv=venv)
  File "/usr/lib/python3.7/site-packages/pluggy/hooks.py", line 284, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/sfinucan/.local/lib/python3.7/site-packages/tox_docker.py", line 245, in tox_runtest_pre
    "Never got answer on port {} from {}".format(containerport, name)
Exception: Never got answer on port 33060/tcp from mysql

On the other hand, I've had to insert a manual sleep for the postgresql:9.6 variant of the same (e.g. py27-django111-postgre) because otherwise my tests start before the database is ready.

Am I doing something wrong here or is that port check broken somehow?

[stephenfin]

It's also worth noting that when this fails, the container is left running and is not cleaned up afterwards.

[stephenfin]

I added the following change:

diff --git a/tox_docker.py b/tox_docker.py
index 09e6c93..cbcb863 100644
--- a/tox_docker.py
+++ b/tox_docker.py
@@ -195,6 +195,8 @@ def tox_runtest_pre(venv):
         name, _, tag = image.partition(":")
         gateway_ip = _get_gateway_ip(container)
         for containerport, hostports in container.attrs["NetworkSettings"]["Ports"].items():
+            print('containerport: %r' % containerport)
+            print('hostports: %r' % hostports)
             for spec in hostports:
                 if spec["HostIp"] == "0.0.0.0":
                     hostport = spec["HostPort"]

and saw the following output:

$ tox -e py27-django111-mysql 
...
py27-django111-mysql docker: run 'mysql:5.7'
containerport: '3306/tcp'
hostports: [{'HostIp': '0.0.0.0', 'HostPort': '32775'}]
containerport: '33060/tcp'
hostports: [{'HostIp': '0.0.0.0', 'HostPort': '32774'}]
...

While that was running (i.e. before it timed out), I checked if the ports were open and indeed, they were:

$ sudo lsof -i -P -n | grep 32775
docker-pr 11686            root    4u  IPv6 336047      0t0  TCP *:32775 (LISTEN)
$ sudo lsof -i -P -n | grep 32774
docker-pr 11673            root    4u  IPv6 330638      0t0  TCP *:32774 (LISTEN)

Because it isn't cleaned up, these are still open after the tox run is done.

[stephenfin]

Looks like 33060 is the port for "the MySQL Database Extended Interface (the MySQL X Protocol)". It looks like this plugin is disabled by default before version 8.0 and I can't figure out a way to enable it without passing arbitrary commands to the container which isn't supported by tox-docker. I guess the solution here is to make port checking optional, unless we want to special-case the mysql container (I don't think we do).

[stephenfin]

Yup, with 8.0 things manage to get past that step (though they fail for other reasons). I'd like to see the ability to disable port checking, if necessary. If this makes sense to you, I'll submit a PR to do just that.

[dcrosta]

I've left a comment on #36 with an alternate idea for MySQL.

Re: postgres, I recently added health check support and have been using the following configuration with success:

[docker:postgres:10]
healthcheck_cmd = 'PGPASSWORD=$POSTGRES_PASSWORD psql --user=$POSTGRES_USER --dbname=$POSTGRES_DB --host=127.0.0.1 --quiet --no-align --tuples-only -1 --command="SELECT 1"'
healthcheck_timeout = 1
healthcheck_retries = 30
healthcheck_interval = 1
healthcheck_start_period = 1

Unfortunately the official postgres containers don't provide their own health check.

Regarding the running containers left over, I've opened #37 to track this work.

[dcrosta]

fixed by #49