SSL error? docker compose and init works flawlessly but cannot connect to RDM home page even with wget locally

[ivanbishop]

My docker ps -a looks like

(invenio) root@lxxxxxx:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2e156ece7e28 ghcr.io/front-matter/invenio-rdm-starter:latest "gunicorn invenio_ap…" 10 minutes ago Up 10 minutes 5000/tcp invenio-rdm-starter-web-1
063c1ad53dfa ghcr.io/front-matter/invenio-rdm-starter:latest "celery -A invenio_a…" 19 minutes ago Up 10 minutes 5000/tcp invenio-rdm-starter-worker-1
2c997284a211 postgres:16.4-bookworm "docker-entrypoint.s…" 19 minutes ago Up 10 minutes 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp invenio-rdm-starter-db-1
847dec28a7dc caddy:2.8.1 "caddy run --config …" 19 minutes ago Up 10 minutes 80/tcp, 443/tcp, 0.0.0.0:1443->1443/tcp, :::1443->1443/tcp, 2019/tcp, 443/udp, 0.0.0.0:7080->7080/tcp, :::7080->7080/tcp invenio-rdm-starter-proxy-1
4ccc03e56a7a opensearchproject/opensearch:2.9.0 "./opensearch-docker…" 19 minutes ago Up 10 minutes 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp, 0.0.0.0:9600->9600/tcp, :::9600->9600/tcp, 9650/tcp invenio-rdm-starter-search-1
572bcbd0154b valkey/valkey:7.2.5-bookworm "docker-entrypoint.s…" 19 minutes ago Up 10 minutes 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp invenio-rdm-starter-cache-1

LSOF of ports looks like (I have apache on 80 and 443 by default)

I've mapped Invenio to 7080 and 1443

(invenio) root@lxxxx:~# lsof -i:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
apache2 831 root 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1130640 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1186290 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1186294 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1193477 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1195474 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1204241 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1205876 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1216045 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1258163 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)
apache2 1258244 www-data 4u IPv6 7282 0t0 TCP *:http (LISTEN)

(invenio) root@lxxxxxx:~# lsof -i:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
apache2 831 root 8u IPv6 7290 0t0 TCP *:https (LISTEN)
ssm-agent 1329 root 14u IPv4 2290878 0t0 TCP ip-172-31-38-35.eu-central-1.compute.internal:37586->52.94.141.25:https (ESTABLISHED)
apache2 1130640 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1186290 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1186294 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1193477 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1195474 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1204241 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1205876 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1216045 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1258163 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)
apache2 1258244 www-data 8u IPv6 7290 0t0 TCP *:https (LISTEN)

(invenio) root@lxxxxx:~# lsof -i:7080
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 1273293 root 4u IPv4 2470714 0t0 TCP *:7080 (LISTEN)
docker-pr 1273316 root 4u IPv6 2470723 0t0 TCP *:7080 (LISTEN)
(invenio) root@lxxxxx:~# lsof -i:1443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 1273270 root 4u IPv4 2472966 0t0 TCP *:1443 (LISTEN)
docker-pr 1273281 root 4u IPv6 2472116 0t0 TCP *:1443 (LISTEN)

wget https://0.0.0.0:1443

--2024-11-08 23:07:14--  https://0.0.0.0:1443/
Connecting to 0.0.0.0:1443... connected.
OpenSSL: error:0A000126:SSL routines::unexpected eof while reading
Unable to establish SSL connection.

curl -v -k --proto '=https' --tlsv1.2 https://0.0.0.0:1443

*   Trying 0.0.0.0:1443...
* Connected to 0.0.0.0 (127.0.0.1) port 1443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

openssl version -a

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
built on: Tue Aug 20 17:27:32 2024 UTC
platform: debian-arm64
options:  bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-BW0rDL/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/aarch64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/aarch64-linux-gnu/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_armcap=0xbf

I have altered docker compose to maker caddy listen at 7080 and 1443

services:
  proxy:
    image: caddy:2.8.1
    restart: "unless-stopped"
    ports:
      - "7080:7080"
      - "1443:1443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config

If I browse from the internet I see BI-DIRECTIONAL traffic to /from port 7080(http) or 1443 (https) but just get NO connection

HOSTS ALLOW looks like
docker-compose.yml: - INVENIO_APP_ALLOWED_HOSTS=${INVENIO_APP_ALLOWED_HOSTS:-['172.31.38.35','AWS.EIP-address','0.0.0.0', 'localhost', '127.0.0.1','FQDN']}

Any ideas?
Anything to do with openssl/openssl#24810

[mfenner]

@ivanbishop caddy is picky about ports 80 and 443 as it uses them to automatically generate SSL certificates using letsencrypt, or his own thing if it is localhost (which letsencrypt doesn't support). The InvenioRDM project uses https even when running locally. So this Docker Compose conflicts with your Apache server running locally. For a setup running in AWS caddy might not be the best option as SSL termination is provided by the AWS Application Load Balancer.

[ivanbishop]

Hi Martin, I run a lot of services with commercial certs on AWS. Ports 443, 6443, 7443 etc (EC2) on the same big instance for testing.

They are opened via Security groups so I see these open ports from 0.0.0.0/0

By changing the caddy port to 1443 per
services: proxy: image: caddy:2.8.1 restart: "unless-stopped" ports: - "7080:7080" - "1443:1443"

1443 is just be another SSL/TLS service (caddy) running. I have opened it to 0.0.0.0/0 for tests

I think the fact running localhost as SSL enabled is the issue.

Is there a way to inject my commercial certs into the running Caddy environment as I cannot use letsencrypt?

I can get into Ash on the container and modify certs etc after they are launched and restart if need be.

I'm going to ping the caddy community https://caddyserver.com/docs/automatic-https#local-https
unless you have an idea :)

Commercial certs must be supported somehow!

Thanks
Ivan

[ivanbishop]

This may be the answer

https://caddy.community/t/ssl-configuration-in-caddy/15535

I feel caddy takes the load off SSL setup, but this is at odds with the "rest" of the SSL/TL config world , lol.

[mfenner]

You can also put this at the top (outside of localhost :443) of your Caddyfile:

{
	auto_https off
}

[mfenner]

Still, in an AWS setup you wouldn't really need Caddy. I also don't use it in production (where I use fly.io).

[mfenner]

Maybe I should add an AWS ECS example configuration for ìnvenio-rdm-starter.

[ivanbishop]

hmmm I used 11.0 of RDM using NGINX (and some work), is it possible to just NOT use caddy in 12.0 and point directly to the service via Nginx?

[ivanbishop]

caddy doesnt really buy me anything except convenience if I dont want to setup my own certs?

[ivanbishop]

Id assume for container service that'd be good.

[ivanbishop]

an nginx example would be great :)

[mfenner]

nginx is obviously fine and many people prefer it over caddy, but the configuration is more complicated for running Docker Compose. And invenio-rdm-starter has to be opinionated, I can't support the multiple options for everything out there. You can use the docker compose and nginx configuration provided by invenio-cli, the docker image works the same.

[ivanbishop]

Actually, the invenio one has an issue around PG, it appears to assume you have PG installed and configured and reachable from what I can tell and that's a weak assumption.

[ivanbishop]

Agreed too many irons in the fire otherwise Martin, I'll persevere with Caddy as your builds are excellent. Once I figure out how to install a commercial cert I'll document it here if that's OK. Others may find it useful (when letsencrypt isn't viable for reputational reasons).

[mfenner]

That sounds good. I like Caddy but have to better understand how to configure/reuse certs. Installing a commercial cert is certainly supported.

[ivanbishop]

Appreciate all your work here Martin, I'll figure out the commercial certs and document here (people are waiting on me and I explained this was my issue with caddy/certs :0

[ivanbishop]

hello martin, what admin username/pass do you use in your Postgres setup in your docker runs? thanks

[ivanbishop]

db:
image: postgres:16.4-bookworm
restart: "unless-stopped"
environment:
- POSTGRES_USER=${POSTGRES_USER:-inveniordm}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-inveniordm}
- POSTGRES_DB=${POSTGRES_DB:-inveniordm}

sorry... :0

[mfenner]

Yes, I use an env variable. No hard-coded passwords.

[ivanbishop]

hello Martin, I hope you have a little time to look over the attached log (log.txt) file from a compose up (with no -d) and let me know if there any blindingly terminal/obvious errors.

Attaching the Caddyfile , docker-compose and log out from from "compose up"

Trying to get some attention from the team at Caddy but no takers yet

Caddyfile.txt
docker-compose.yml.txt
log.txt

Many thanks

[ivanbishop]

Please close out this discussion Martin. I'll open a new one NOW I have commercial certs and NON 80/443 ports working with Caddy (after a lot of help from them)

I see some errors from the invenio side that I know you will be able to help with IF you have time