Skip to content

Commit

Permalink
Ensure db is ready to accept connections prior to migration
Browse files Browse the repository at this point in the history
When we run `make docker-setup`, we occasionally find the following:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 75, in handle
    self.check(databases=[database])
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 419, in check
    all_issues = checks.run_checks(
  File "/usr/local/lib/python3.10/site-packages/django/core/checks/registry.py", line 76, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/usr/local/lib/python3.10/site-packages/django/core/checks/model_checks.py", line 34, in check_all_models
    errors.extend(model.check(**kwargs))
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 1303, in check
    *cls._check_indexes(databases),
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 1695, in _check_indexes
    connection.features.supports_covering_indexes or
  File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/features.py", line 92, in is_postgresql_11
    return self.connection.pg_version >= 110000
  File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 329, in pg_version
    with self.temporary_connection():
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 603, in temporary_connection
    with self.cursor() as cursor:
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 218, in ensure_connection
    with self.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?

make: *** [Makefile:52: docker-migrate] Error 1
```

Here we have started `docker compose up -d` then immediately tried to
run `docker compose exec -T app python manage.py migrate`, which
operates on the `db` container (PostgreSQL). This is rejected as
PostgreSQL is not ready to accept connections.

To ameliorate this, we:

 - introduce a healthcheck to the `db` container in `docker-compose.yml`
 - have `app` wait for `db`'s healthcheck to pass before continuing.

This allows `docker-setup` to work reliably.
  • Loading branch information
AlexandreCarlton committed Dec 12, 2024
1 parent ac05b6e commit 81b2917
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ services:
POSTGRES_DB: ${POSTGRES_DB:-pokeapi}
volumes:
- pg_data:/var/lib/postgresql/data
healthcheck:
test: pg_isready
interval: 1s
timeout: 5s
retries: 10
restart: always

app:
Expand All @@ -28,8 +33,10 @@ services:
- db
- cache
depends_on:
- db
- cache
db:
condition: service_healthy
cache:
condition: service_started
restart: always

web:
Expand Down

0 comments on commit 81b2917

Please sign in to comment.