From 937bb0aef0552c7b1ce5048127d8faa980e12a82 Mon Sep 17 00:00:00 2001 From: "dea.duro.dd@gmail.com" <103927498+DeaDurro@users.noreply.github.com> Date: Tue, 23 Jul 2024 14:25:15 +0200 Subject: [PATCH 01/15] Update: Change to Support Subdomains and Contextpaths Fixes: #92 --- feasibility-portal/README.md | 33 ++++++ feasibility-portal/backend/.env.default | 14 +-- feasibility-portal/backend/docker-compose.yml | 10 +- .../gui/deploy-config.json.default | 4 +- feasibility-portal/keycloak/.env.default | 8 +- .../keycloak/docker-compose.yml | 15 +-- feasibility-portal/proxy/.env.default | 13 +- .../proxy/conf.d/backend.conf.template | 28 +++-- .../proxy/conf.d/gui.conf.template | 30 +++-- .../proxy/conf.d/keycloak.conf.template | 39 +++--- .../proxy/context-paths.nginx.conf | 112 ++++++++++++++++++ feasibility-portal/proxy/docker-compose.yml | 12 +- feasibility-portal/proxy/nginx.conf | 45 ------- .../proxy/subdomains.nginx.conf | 79 ++++++++++++ feasibility-triangle/.DS_Store | Bin 0 -> 6148 bytes .../rev-proxy/docker-compose.yml | 3 + 16 files changed, 322 insertions(+), 123 deletions(-) create mode 100644 feasibility-portal/proxy/context-paths.nginx.conf delete mode 100644 feasibility-portal/proxy/nginx.conf create mode 100644 feasibility-portal/proxy/subdomains.nginx.conf create mode 100644 feasibility-triangle/.DS_Store diff --git a/feasibility-portal/README.md b/feasibility-portal/README.md index dd9e0e15..53c200ba 100644 --- a/feasibility-portal/README.md +++ b/feasibility-portal/README.md @@ -66,6 +66,39 @@ The portal is configured by default to start the following services: - UI - Keycloak +For the reverse proxy you need to choose the configuration (variable `FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG` in +[proxy/.env](./proxy/.env)) which also decides what the changes to the `.env` files you have to make: + +- [./subdomains.nginx.conf](./proxy/subdomains.nginx.conf) with separate domains for the services (Backend, UI, Keycloak) + - All subdomains must point to the host machine the portal will run. + + - Set the service hostnames (`BACKEND_HOSTNAME`, `KEYCLOAK_HOSTNAME` and `GUI_HOSTNAME`, depending on which services you need) in [proxy/.env](./proxy/.env). +- Change the following variables in [keycloak/.env](./keycloak /.env): + - `FEASIBILITY_KC_HOSTNAME_URL`and `FEASIBILITY_KC_HOSTNAME_ADMIN_URL`: set the domain part to the value you set for `KEYCLOAK_HOSTNAME` before. + -` FEASIBILITY_KC_HTTP_RELATIVE_PATH`: set to `/auth`. +- Change the values for the variables `FEASIBILITY_BACKEND_API_BASE_URL` in [backend/.env](./backend/.env) and `FEASIBILITY_BACKEND_ALLOWED_ORIGINS` in [backend /.env](./backend/.env) + to the base url of your feasibility portal backend. In the [backend/.env](./backend/.env) change the values for the variable `FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_ISSUER` to the base url of your feasibility portal keycloak. +- Change the following variables in [gui/deploy-config.json](./gui/deploy-config.json): + - `uiBackendApi > baseUrl`: set the domain part of the local feasibility portal backend. + - `auth > baseUrl`: set the domain part of the local feasibility portal keycloak. +- On the [proxy/.env] use this variable `FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG=./subdomains.nginx.conf`. + +- [./context-paths.nginx.conf](./proxy/context-paths.nginx.conf) which requires only one domain and uses context paths (`/auth` for keycloak,`/api` for backend and `/`) for user interface. +- The domain must point to the host machine the portal will run. +- On the [proxy/.env] use this variable`FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG=./context-paths.nginx.conf` +- Change the following variable `FEASIBILITY_KC_HOSTNAME_URL` and `FEASIBILITY_KC_HOSTNAME_ADMIN_URL` in [keycloak/.env]: set the domain part of your domain. The path must be set to /auth at the end of the url. For example, https://example.org/auth. +- Add `/auth` in the following variable `FEASIBILITY_KC_HTTP_RELATIVE_PATH` in [keycloak/.env] +- Change the following variable `FEASIBILITY_BACKEND_API_BASE_URL` in [backend/.env]: set the domain part of your domain. The path must be set to /api at the end of the url. For example, https://example.org/api. +- Change the following variable `FEASIBILITY_BACKEND_ALLOWED_ORIGINS` in [backend/.env]: set the domain part of your domain. For example, https://example.org. +- Change the following variable`FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_ISSUER` in [backend/.env]: set the domain part of your domain. The path must be set to /api at the end of the url. For example, https://example.org/auth. +- Add `/auth` in the following variable `FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_JWK` in [backend/.env] +- Change the variable `FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL` when using the bundled keycloak in [backend/.env]replace the values with https://DOMAIN:REV_PROXY_PORT/auth/realms/blaze where DOMAIN is your domain and REV_PROXY_PORT is the port number set in rev-proxy/.env (default 444). For example, https://example.org:444/auth/realms/blaze. +- On the [gui/deploy-config.json] change the following variables: + - `uiBackendApi > baseUrl`: set the domain part of the local feasibility portal backend with the context path `/api`. For example https://example.org/api. + - `auth > baseUrl`: set the domain part of the local feasibility portal keycloak the context path `/auth`. For example https://example.org/auth. + +Please note that the keycloak provided here is an example setup, and we strongly recommend for each site to adjust the keycloak installation to their local security requirements or connect the local feasibility portal to a keycloak already provided at the site. + For more details on the environment variables see the paragraph **Configurable environment variables** of this README. ### Step 6 - Start the feasibility portal diff --git a/feasibility-portal/backend/.env.default b/feasibility-portal/backend/.env.default index 50d9d9ff..cc1299a0 100644 --- a/feasibility-portal/backend/.env.default +++ b/feasibility-portal/backend/.env.default @@ -1,8 +1,8 @@ # ----- app FEASIBILITY_BACKEND_CQL_TRANSLATE_ENABLED=true FEASIBILITY_BACKEND_FHIR_TRANSLATE_ENABLED=false -FEASIBILITY_BACKEND_API_BASE_URL=https://api.datenportal.localhost -FEASIBILITY_BACKEND_ALLOWED_ORIGINS=https://datenportal.localhost +FEASIBILITY_BACKEND_API_BASE_URL=https://example.org/api +FEASIBILITY_BACKEND_ALLOWED_ORIGINS=https://example.org FEASIBILITY_BACKEND_ONTOLOGY_ORDER="Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung" FEASIBILITY_BACKEND_MAX_SAVED_QUERIES_PER_USER=100 # ---- db config @@ -15,8 +15,8 @@ FEASIBILITY_BACKEND_KEYCLOAK_ENABLED=true FEASIBILITY_BACKEND_KEYCLOAK_ALLOWED_ROLE=FeasibilityUser FEASIBILITY_BACKEND_KEYCLOAK_POWER_ROLE=FeasibilityPowerUser FEASIBILITY_BACKEND_KEYCLOAK_ADMIN_ROLE=FeasibilityAdmin -FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_ISSUER=https://auth.datenportal.localhost -FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_JWK=http://auth:8080 +FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_ISSUER=https://example.org/auth +FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_JWK=http://auth:8080/auth FEASIBILITY_BACKEND_KEYCLOAK_REALM=feasibility #---- Direct broker FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_ENABLED=true @@ -24,9 +24,9 @@ FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_USE_CQL=false FEASIBILITY_BACKEND_BROKER_CLIENT_OBFUSCATE_RESULT_COUNT=false FEASIBILITY_BACKEND_FLARE_WEBSERVICE_BASE_URL=http://flare:8080 FEASIBILITY_BACKEND_CQL_SERVER_BASE_URL=http://fhir-server:8080/fhir -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME= -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD= -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL=https://keycloak.localhost:444/realms/blaze +FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME=admin +FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD=admin +FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL=https://example.org:444/auth/realms/blaze FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID=account FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET=insecure # ---- DSF broker diff --git a/feasibility-portal/backend/docker-compose.yml b/feasibility-portal/backend/docker-compose.yml index 33c474c2..038611e3 100644 --- a/feasibility-portal/backend/docker-compose.yml +++ b/feasibility-portal/backend/docker-compose.yml @@ -10,8 +10,8 @@ services: QUERY_VALIDATION_ENABLED: ${FEASIBILITY_BACKEND_QUERY_VALIDATION_ENABLED:-true} CQL_TRANSLATE_ENABLED: ${FEASIBILITY_BACKEND_CQL_TRANSLATE_ENABLED:-true} FHIR_TRANSLATE_ENABLED: ${FEASIBILITY_BACKEND_FHIR_TRANSLATE_ENABLED:-false} - API_BASE_URL: ${FEASIBILITY_BACKEND_API_BASE_URL:-https://localhost/api/} - ALLOWED_ORIGINS: ${FEASIBILITY_BACKEND_ALLOWED_ORIGINS:-https://localhost} + API_BASE_URL: ${FEASIBILITY_BACKEND_API_BASE_URL:-https://example.org/api/} + ALLOWED_ORIGINS: ${FEASIBILITY_BACKEND_ALLOWED_ORIGINS:-https://example.org} QUERYRESULT_EXPIRY_MINUTES: ${FEASIBILITY_BACKEND_QUERYRESULT_EXPIRY_MINUTES:-5} ONTOLOGY_ORDER: ${FEASIBILITY_BACKEND_ONTOLOGY_ORDER:-"Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung"} MAX_SAVED_QUERIES_PER_USER: ${FEASIBILITY_BACKEND_MAX_SAVED_QUERIES_PER_USER:-100} @@ -34,9 +34,9 @@ services: BROKER_CLIENT_OBFUSCATE_RESULT_COUNT: ${FEASIBILITY_BACKEND_BROKER_CLIENT_OBFUSCATE_RESULT_COUNT:-false} FLARE_WEBSERVICE_BASE_URL: ${FEASIBILITY_BACKEND_FLARE_WEBSERVICE_BASE_URL:-http://flare:8080} CQL_SERVER_BASE_URL: ${FEASIBILITY_BACKEND_CQL_SERVER_BASE_URL:-http://fhir-server:8080/fhir} - BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME} - BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD} - BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL:-https://keycloak.localhost:444/realms/blaze} + BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME:-admin} + BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD:-admin} + BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL:-https://example.org:444/auth/realms/blaze} BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID:-account} BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET:-insecure} # ---- Aktin broker diff --git a/feasibility-portal/gui/deploy-config.json.default b/feasibility-portal/gui/deploy-config.json.default index b5d0ff54..400a9a3f 100644 --- a/feasibility-portal/gui/deploy-config.json.default +++ b/feasibility-portal/gui/deploy-config.json.default @@ -6,10 +6,10 @@ "baseUrl": "/api" }, "uiBackendApi": { - "baseUrl": "https://api.datenportal.localhost/api/v3" + "baseUrl": "https://example.org/api/v3" }, "auth": { - "baseUrl": "https://auth.datenportal.localhost", + "baseUrl": "https://example.org/auth", "realm": "feasibility", "clientId": "feasibility-webapp", "roles": ["FeasibilityUser"] diff --git a/feasibility-portal/keycloak/.env.default b/feasibility-portal/keycloak/.env.default index 3b653f1e..bf2f01c3 100644 --- a/feasibility-portal/keycloak/.env.default +++ b/feasibility-portal/keycloak/.env.default @@ -2,10 +2,10 @@ FEASIBILITY_KC_DB=keycloakdb FEASIBILITY_KC_DB_USER=keycloakdbuser FEASIBILITY_KC_DB_PW=keycloakdbpw FEASIBILITY_KC_ADMIN_USER=admin -FEASIBILITY_KC_ADMIN_PW=adminpw -FEASIBILITY_KC_HTTP_RELATIVE_PATH=/ -FEASIBILITY_KC_HOSTNAME_URL=https://auth.datenportal.localhost -FEASIBILITY_KC_HOSTNAME_ADMIN_URL=https://auth.datenportal.localhost +FEASIBILITY_KC_ADMIN_PW=admin +FEASIBILITY_KC_HTTP_RELATIVE_PATH=/auth +FEASIBILITY_KC_HOSTNAME_URL=https://example.org/auth +FEASIBILITY_KC_HOSTNAME_ADMIN_URL=https://example.org/auth FEASIBILITY_KC_LOG_LEVEL=info FEASIBILITY_KC_PROXY=edge diff --git a/feasibility-portal/keycloak/docker-compose.yml b/feasibility-portal/keycloak/docker-compose.yml index d82b36a9..0aa53a8a 100644 --- a/feasibility-portal/keycloak/docker-compose.yml +++ b/feasibility-portal/keycloak/docker-compose.yml @@ -1,3 +1,4 @@ + services: auth-db: image: postgres:15-alpine @@ -16,13 +17,13 @@ services: environment: KC_DB: postgres KC_DB_URL: "jdbc:postgresql://auth-db:5432/${FEASIBILITY_KC_DB}" - KC_DB_USERNAME: ${FEASIBILITY_KC_DB_USER} - KC_DB_PASSWORD: ${FEASIBILITY_KC_DB_PW} - KEYCLOAK_ADMIN: ${FEASIBILITY_KC_ADMIN_USER} - KEYCLOAK_ADMIN_PASSWORD: ${FEASIBILITY_KC_ADMIN_PW} - KC_HTTP_RELATIVE_PATH: ${FEASIBILITY_KC_HTTP_RELATIVE_PATH} - KC_HOSTNAME: ${FEASIBILITY_KC_HOSTNAME_URL:-https://auth.datenportal.localhost} - KC_HOSTNAME_ADMIN: ${FEASIBILITY_KC_HOSTNAME_ADMIN_URL} + KC_DB_USERNAME: ${FEASIBILITY_KC_DB_USER:-keycloakdbuser} + KC_DB_PASSWORD: ${FEASIBILITY_KC_DB_PW:-keycloakdbpw} + KEYCLOAK_ADMIN: ${FEASIBILITY_KC_ADMIN_USER:-} + KEYCLOAK_ADMIN_PASSWORD: ${FEASIBILITY_KC_ADMIN_PW:-} + KC_HTTP_RELATIVE_PATH: ${FEASIBILITY_KC_HTTP_RELATIVE_PATH:-/auth} + KC_HOSTNAME: ${FEASIBILITY_KC_HOSTNAME_URL:-https://auth.localhost} + KC_HOSTNAME_ADMIN: ${FEASIBILITY_KC_HOSTNAME_ADMIN_URL:-https://auth.localhost} KC_LOG_LEVEL: ${FEASIBILITY_KC_LOG_LEVEL:-info} KC_PROXY: ${FEASIBILITY_KC_PROXY:-edge} volumes: diff --git a/feasibility-portal/proxy/.env.default b/feasibility-portal/proxy/.env.default index 42e49f69..ff8ebef3 100644 --- a/feasibility-portal/proxy/.env.default +++ b/feasibility-portal/proxy/.env.default @@ -1,6 +1,13 @@ +# Set separate hostnames if 'subdomains.nginx.conf' is used (see FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG below). +# For 'context-paths.nginx.conf' these values are ignored. BACKEND_HOSTNAME=api.datenportal.localhost KEYCLOAK_HOSTNAME=auth.datenportal.localhost GUI_HOSTNAME=datenportal.localhost -PROXY_CERTIFICATE_PATH=../auth/cert.pem -PROXY_CERTIFICATE_KEY_PATH=../auth/key.pem -PROXY_NGINX_CONFIG_PATH=./nginx.conf + +# Comment one of the FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG depending if it is used 'subdomains.nginx.conf' or 'context-paths.nginx.conf' +FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG=./subdomains.nginx.conf +#FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG=./context-paths.nginx.conf +FEASIBILITY_PORTAL_PROXY_CERTIFICATE_PATH=../auth/cert.pem +FEASIBILITY_PORTAL_PROXY_CERTIFICATE_KEY_PATH=../auth/key.pem + + diff --git a/feasibility-portal/proxy/conf.d/backend.conf.template b/feasibility-portal/proxy/conf.d/backend.conf.template index 9d0ca7a2..cd1413f1 100644 --- a/feasibility-portal/proxy/conf.d/backend.conf.template +++ b/feasibility-portal/proxy/conf.d/backend.conf.template @@ -1,12 +1,22 @@ - server { +server { + listen 8443 ssl; + http2 on; + server_name ${BACKEND_HOSTNAME}; - listen 8443 ssl; - server_name ${BACKEND_HOSTNAME}; - - location / { - proxy_pass http://feasibility-gui-backend:8090/; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Host $host; - } + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + location / { + #Used with 'subdomains.nginx.conf' + proxy_pass http://feasibility-gui-backend:8090/; + + #Used with 'context-paths.nginx.conf' + #set $backend_upstream http://feasibility-gui-backend:8090; + #proxy_pass http://$backend_upstream$request_uri; } + +} \ No newline at end of file diff --git a/feasibility-portal/proxy/conf.d/gui.conf.template b/feasibility-portal/proxy/conf.d/gui.conf.template index 874b4757..eed3716e 100644 --- a/feasibility-portal/proxy/conf.d/gui.conf.template +++ b/feasibility-portal/proxy/conf.d/gui.conf.template @@ -1,17 +1,23 @@ server { + listen 8443 ssl; + http2 on; + server_name ${GUI_HOSTNAME}; - listen 8443 ssl; - server_name ${GUI_HOSTNAME}; + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } - location / { - proxy_pass http://dataportal-ui:8080/; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header Host $host; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - proxy_read_timeout 86400; - } + location / { + #Used with 'subdomains.nginx.conf' + proxy_pass http://dataportal-ui:8080/; + + #Used with 'context-paths.nginx.conf' + #set $gui_upstream http://dataportal-ui:8080; + #proxy_pass http://$gui_upstream$request_uri; } + +} diff --git a/feasibility-portal/proxy/conf.d/keycloak.conf.template b/feasibility-portal/proxy/conf.d/keycloak.conf.template index fb27b44b..87f3f559 100644 --- a/feasibility-portal/proxy/conf.d/keycloak.conf.template +++ b/feasibility-portal/proxy/conf.d/keycloak.conf.template @@ -1,30 +1,21 @@ server { + listen 8443 ssl; + http2 on; + server_name ${KEYCLOAK_HOSTNAME}; - listen 8443 ssl; - server_name ${KEYCLOAK_HOSTNAME}; + # redirect server error pages to the static page /50x.html + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } - location / { - proxy_buffers 4 128k; - proxy_busy_buffers_size 128k; - proxy_buffer_size 64k; - proxy_pass http://auth:8080; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-Port $server_port; - proxy_set_header X-Real-IP $remote_addr; - } + location / { + #Used with 'subdomains.nginx.conf' + proxy_pass http://auth:8080; - #location /keycloakadmin { - # proxy_buffers 4 128k; - # proxy_busy_buffers_size 128k; - # proxy_buffer_size 64k; - # proxy_pass http://auth:8080; - # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - # proxy_set_header X-Forwarded-Proto $scheme; - # proxy_set_header X-Forwarded-Host $host; - # proxy_set_header X-Forwarded-Port $server_port; - # proxy_set_header X-Real-IP $remote_addr; - # } + #Used with 'context-paths.nginx.conf' + #set $auth_upstream http://auth:8080; + #proxy_pass http://$auth_upstream$request_uri; } +} \ No newline at end of file diff --git a/feasibility-portal/proxy/context-paths.nginx.conf b/feasibility-portal/proxy/context-paths.nginx.conf new file mode 100644 index 00000000..1646a2f9 --- /dev/null +++ b/feasibility-portal/proxy/context-paths.nginx.conf @@ -0,0 +1,112 @@ +pid /tmp/nginx.pid; + +events { + worker_connections 1024; +} + +http { + proxy_read_timeout 7d; + proxy_set_header X-Real-IP $remote_addr; + + include /etc/nginx/mime.types; + gzip on; + gzip_vary on; + gzip_min_length 10240; + gzip_proxied expired no-cache no-store private auth; + gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml; + gzip_disable "MSIE [1-6]\."; + + # Check if a X-Forwarded-Proto header (set by reverse-proxy) is already present. If not take the scheme used to call our nginx server. + map $http_x_forwarded_proto $x_forwarded_proto { + default $http_x_forwarded_proto; + "" $scheme; # Note that if the reverse-proxy does not add a X-Forwarded-Proto header, it may be incorrect if the protocol used by the reverse proxy is not the same as the one on which your nginx server is listening. In this case you have no solution than harcode the correct value. + } + + # Check if a X-Forwarded-Host header (set by reverse-proxy) is already present. If not take the value of the 'Host' header. + map $http_x_forwarded_host $x_forwarded_host { + default $http_x_forwarded_host; + "" $http_host; + } + + # Set the default port of each scheme/protocol (80 for http, 443 for https) + map $x_forwarded_proto $default_http_port { + default 80; + "https" 443; + } + + # Extract the real port of the client request url (unfortunatly nginx has no variable to get this info) + map $http_host $request_port { + default $default_http_port; # If port not explicitely defined in url take the default one associated to the calling scheme/protocol (80 for http, 443 for https) + "~^[^\:]+:(?

\d+)$" $p; + } + + # Check if a X-Forwarded-Port header (set by reverse-proxy) is already present. If not take the port from the client request url + map $http_x_forwarded_port $x_forwarded_port { + default $http_x_forwarded_port; + "" $request_port; + } + + server { + listen 8443 ssl; + http2 on; + + # DNS resolver needed for Docker + resolver 127.0.0.11 valid=10s; + + # SSL-Certificate and private key + ssl_certificate /etc/nginx/certs/cert.pem; + ssl_certificate_key /etc/nginx/certs/key.pem; + + # The supported SSL Protocols + ssl_protocols TLSv1.2 TLSv1.3; + + # NGINX can impose its TLS cipher suite choices over those of a connecting browser, provided the browser supports them. + ssl_prefer_server_ciphers on; + + # The supported SSL Ciphers + ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:RC4-SHA'; + + ssl_session_cache builtin:1000 shared:SSL:10m; + + # OCSP Stapling + # When enabled, NGINX will make OCSP requests on behalf of connecting browsers. The response received from the OCSP server is added to NGINX’s browser response, which eliminates the need for browsers to verify a certificate’s revocation status by connecting directly to an OCSP server. + ssl_stapling on; + ssl_stapling_verify on; + + # Header Options + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Content-Type-Options nosniff; + add_header X-Frame-Options SAMEORIGIN; + + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $x_forwarded_proto; + proxy_set_header X-Forwarded-Host $x_forwarded_host; + proxy_set_header X-Forwarded-Port $x_forwarded_port; + + location /api { + set $backend_upstream http://feasibility-gui-backend:8090; + proxy_pass $backend_upstream; + + proxy_buffer_size 8k; + proxy_request_buffering off; + client_max_body_size 100M; + } + + location /auth { + set $auth_upstream http://auth:8080; + proxy_pass $auth_upstream; + } + + location / { + set $gui_upstream http://dataportal-ui:8080; + proxy_pass $gui_upstream; + } + } +} diff --git a/feasibility-portal/proxy/docker-compose.yml b/feasibility-portal/proxy/docker-compose.yml index 30101839..14de4be8 100644 --- a/feasibility-portal/proxy/docker-compose.yml +++ b/feasibility-portal/proxy/docker-compose.yml @@ -1,7 +1,8 @@ +# Comment one of the FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG depending if it is used 'subdomains.nginx.conf' or 'context-paths.nginx.conf' services: dataportal-nginx: restart: unless-stopped - image: nginxinc/nginx-unprivileged:1.23-alpine + image: nginxinc/nginx-unprivileged:1.27-alpine environment: BACKEND_HOSTNAME: ${BACKEND_HOSTNAME:-api.datenportal.localhost} GUI_HOSTNAME: ${GUI_HOSTNAME:-datenportal.localhost} @@ -9,9 +10,10 @@ services: ports: - 443:8443 volumes: - - ${PROXY_CERTIFICATE_PATH:-../auth/cert.pem}:/etc/nginx/conf.d/cert.pem - - ${PROXY_CERTIFICATE_KEY_PATH:-../auth/key.pem}:/etc/nginx/conf.d/key.pem - - ${PROXY_NGINX_CONFIG_PATH:-./nginx.conf}:/etc/nginx/nginx.conf:ro - - ./conf.d:/etc/nginx/templates + - ${FEASIBILITY_PORTAL_PROXY_CERTIFICATE_PATH:-../auth/cert.pem}:/etc/nginx/certs/cert.pem + - ${FEASIBILITY_PORTAL_PROXY_CERTIFICATE_KEY_PATH:-../auth/key.pem}:/etc/nginx/certs/key.pem + #- ${FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG:-./context-paths.nginx.conf}:/etc/nginx/nginx.conf:ro + - ${FEASIBILITY_PORTAL_PROXY_NGINX_CONFIG:-./subdomains.nginx.conf}:/etc/nginx/nginx.conf:ro + - ./conf.d:/etc/nginx/templates:ro diff --git a/feasibility-portal/proxy/nginx.conf b/feasibility-portal/proxy/nginx.conf deleted file mode 100644 index 0ffdd73b..00000000 --- a/feasibility-portal/proxy/nginx.conf +++ /dev/null @@ -1,45 +0,0 @@ -pid /tmp/nginx.pid; - -events { - worker_connections 1024; - } - -http { - - gzip on; - gzip_vary on; - gzip_min_length 10240; - gzip_proxied expired no-cache no-store private auth; - gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml; - gzip_disable "MSIE [1-6]\."; - - # SSL-Certificate and private key - ssl_certificate /etc/nginx/conf.d/cert.pem; - ssl_certificate_key /etc/nginx/conf.d/key.pem; - - # The supported SSL Protocols - ssl_protocols TLSv1.2 TLSv1.3; - - # NGINX can impose its TLS cipher suite choices over those of a connecting browser, provided the browser supports them. - ssl_prefer_server_ciphers on; - - # The supported SSL Ciphers - ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:RC4-SHA'; - - # Path to the Diffie-Hellman Prime - # to create: openssl dhparam -out /etc/nginx/dhparam.pem 4096 - # ssl_dhparam /etc/nginx/dhparam.pem; - - - ssl_session_cache builtin:1000 shared:SSL:10m; - - # OCSP Stapling - # When enabled, NGINX will make OCSP requests on behalf of connecting browsers. The response received from the OCSP server is added to NGINX’s browser response, which eliminates the need for browsers to verify a certificate’s revocation status by connecting directly to an OCSP server. - ssl_stapling on; - ssl_stapling_verify on; - - # Header Options - add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; - add_header X-Content-Type-Options nosniff; - - include /etc/nginx/conf.d/*.conf; -} diff --git a/feasibility-portal/proxy/subdomains.nginx.conf b/feasibility-portal/proxy/subdomains.nginx.conf new file mode 100644 index 00000000..6dc10c88 --- /dev/null +++ b/feasibility-portal/proxy/subdomains.nginx.conf @@ -0,0 +1,79 @@ +pid /tmp/nginx.pid; + +events { + worker_connections 1024; + } + +http { + + + # Check if a X-Forwarded-Proto header (set by reverse-proxy) is already present. If not take the scheme used to call our nginx server. + map $http_x_forwarded_proto $x_forwarded_proto { + default $http_x_forwarded_proto; + "" $scheme; # Note that if the reverse-proxy does not add a X-Forwarded-Proto header, it may be incorrect if the protocol used by the reverse proxy is not the same as the one on which your nginx server is listening. In this case you have no solution than harcode the correct value. + } + + # Check if a X-Forwarded-Host header (set by reverse-proxy) is already present. If not take the value of the 'Host' header. + map $http_x_forwarded_host $x_forwarded_host { + default $http_x_forwarded_host; + "" $http_host; + } + + # Set the default port of each scheme/protocol (80 for http, 443 for https) + map $x_forwarded_proto $default_http_port { + default 80; + "https" 443; + } + + # Extract the real port of the client request url (unfortunatly nginx has no variable to get this info) + map $http_host $request_port { + default $default_http_port; # If port not explicitely defined in url take the default one associated to the calling scheme/protocol (80 for http, 443 for https) + "~^[^\:]+:(?

\d+)$" $p; + } + + # Check if a X-Forwarded-Port header (set by reverse-proxy) is already present. If not take the port from the client request url + map $http_x_forwarded_port $x_forwarded_port { + default $http_x_forwarded_port; + "" $request_port; + } + + proxy_read_timeout 7d; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $x_forwarded_proto; + proxy_set_header X-Forwarded-Host $x_forwarded_host; + proxy_set_header X-Forwarded-Port $x_forwarded_port; + + gzip on; + gzip_vary on; + gzip_min_length 10240; + gzip_proxied expired no-cache no-store private auth; + gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml; + gzip_disable "MSIE [1-6]\."; + + # SSL-Certificate and private key + ssl_certificate /etc/nginx/certs/cert.pem; + ssl_certificate_key /etc/nginx/certs/key.pem; + + # The supported SSL Protocols + ssl_protocols TLSv1.2 TLSv1.3; + + # NGINX can impose its TLS cipher suite choices over those of a connecting browser, provided the browser supports them. + ssl_prefer_server_ciphers on; + + # The supported SSL Ciphers + ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:RC4-SHA'; + ssl_session_cache builtin:1000 shared:SSL:10m; + + # OCSP Stapling + # When enabled, NGINX will make OCSP requests on behalf of connecting browsers. The response received from the OCSP server is added to NGINX’s browser response, which eliminates the need for browsers to verify a certificate’s revocation status by connecting directly to an OCSP server. + ssl_stapling on; + ssl_stapling_verify on; + + # Header Options + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Content-Type-Options nosniff; + add_header X-Frame-Options SAMEORIGIN; + + include /etc/nginx/conf.d/*.conf; +} diff --git a/feasibility-triangle/.DS_Store b/feasibility-triangle/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..fc219757637bed579085dde8d6aeffe5cb88aac4 GIT binary patch literal 6148 zcmeHK%}T>S5T0$TO({YS3OxqAR&1>b;w8lT0!H+pQWH{bFlHs`&!H4@))(?gd>&_Z zw_>S1co8f!F!RmM&Scqd!~Ot(@J3M^zy<&dm9SJp^Mg=7>5`O8gixq&>_HBF$Y6|? z@baFV}-i3~C0J$y;xeAsGzh*G(-v01gMR^7Vu2Xf*MgW)Lm zg6xJ`mr^F-+z-O5xIgLH+h;Nz265WYR6!i|G3E9;P9r(-poGQA2YxV{F(vU9~@Ld*I=PhZ5?Rn`bhm1LK5`pEkWoQ zbPX07u?I!yQbb)U%o9WCa*R7B&ox+R)a4-5$oL&IvM?_cp+?8JqtZdR8rd=f%)lZ8 zW!<#t{6Am){$DKO7Bj#MtQ7;I()GI@PRX3DOOvCs)L|usI*J>p aN-*w_gXkJ8G-3pWF9M1NHq5}EGVlpip-v$H literal 0 HcmV?d00001 diff --git a/feasibility-triangle/rev-proxy/docker-compose.yml b/feasibility-triangle/rev-proxy/docker-compose.yml index b9f94b3b..ab8abd7e 100644 --- a/feasibility-triangle/rev-proxy/docker-compose.yml +++ b/feasibility-triangle/rev-proxy/docker-compose.yml @@ -4,7 +4,10 @@ services: image: nginxinc/nginx-unprivileged:1.25.5-alpine environment: FHIR_SERVER_HOSTNAME: ${FHIR_SERVER_HOSTNAME:-fhir.localhost} + KEYCLOAK_HOSTNAME: ${KEYCLOAK_HOSTNAME:-auth.localhost} + + FLARE_HOSTNAME: ${FLARE_HOSTNAME:-flare.localhost} ports: - ${FEASIBILITY_TRIANGLE_REV_PROXY_PORT:-444}:8443 From 43cd762b1273d6ad286eda5f72d4752d762dacda Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Wed, 4 Sep 2024 15:01:32 +0200 Subject: [PATCH 02/15] Update ui and backend, add elastic search, change config nginx backend template --- feasibility-portal/backend/.env.default | 95 ++++---- feasibility-portal/backend/docker-compose.yml | 203 +++++++++++------- feasibility-portal/backend/elastic-init.sh | 54 +++++ .../gui/deploy-config.json.default | 2 +- feasibility-portal/gui/docker-compose.yml | 2 +- .../proxy/conf.d/backend.conf.template | 2 +- 6 files changed, 235 insertions(+), 123 deletions(-) create mode 100755 feasibility-portal/backend/elastic-init.sh diff --git a/feasibility-portal/backend/.env.default b/feasibility-portal/backend/.env.default index 50d9d9ff..b098fe72 100644 --- a/feasibility-portal/backend/.env.default +++ b/feasibility-portal/backend/.env.default @@ -1,56 +1,57 @@ # ----- app -FEASIBILITY_BACKEND_CQL_TRANSLATE_ENABLED=true -FEASIBILITY_BACKEND_FHIR_TRANSLATE_ENABLED=false -FEASIBILITY_BACKEND_API_BASE_URL=https://api.datenportal.localhost -FEASIBILITY_BACKEND_ALLOWED_ORIGINS=https://datenportal.localhost -FEASIBILITY_BACKEND_ONTOLOGY_ORDER="Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung" -FEASIBILITY_BACKEND_MAX_SAVED_QUERIES_PER_USER=100 +DATAPORTAL_BACKEND_CQL_TRANSLATE_ENABLED=true +DATAPORTAL_BACKEND_FHIR_TRANSLATE_ENABLED=false +DATAPORTAL_BACKEND_API_BASE_URL=https://api.datenportal.localhost +DATAPORTAL_BACKEND_ALLOWED_ORIGINS=https://datenportal.localhost +DATAPORTAL_BACKEND_ONTOLOGY_ORDER="Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung" +DATAPORTAL_BACKEND_MAX_SAVED_QUERIES_PER_USER=100 # ---- db config -FEASIBILITY_BACKEND_DATASOURCE_HOST=feasibility-gui-backend-db -FEASIBILITY_BACKEND_DATASOURCE_PORT=5432 -FEASIBILITY_BACKEND_DATASOURCE_USERNAME=guidbuser -FEASIBILITY_BACKEND_DATASOURCE_PASSWORD=guidbpw +DATAPORTAL_BACKEND_DATASOURCE_HOST=feasibility-gui-backend-db +DATAPORTAL_BACKEND_DATASOURCE_PORT=5432 +DATAPORTAL_BACKEND_DATASOURCE_USERNAME=guidbuser +DATAPORTAL_BACKEND_DATASOURCE_PASSWORD=guidbpw # ---- auth -FEASIBILITY_BACKEND_KEYCLOAK_ENABLED=true -FEASIBILITY_BACKEND_KEYCLOAK_ALLOWED_ROLE=FeasibilityUser -FEASIBILITY_BACKEND_KEYCLOAK_POWER_ROLE=FeasibilityPowerUser -FEASIBILITY_BACKEND_KEYCLOAK_ADMIN_ROLE=FeasibilityAdmin -FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_ISSUER=https://auth.datenportal.localhost -FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_JWK=http://auth:8080 -FEASIBILITY_BACKEND_KEYCLOAK_REALM=feasibility +DATAPORTAL_BACKEND_KEYCLOAK_ENABLED=true +DATAPORTAL_BACKEND_KEYCLOAK_ALLOWED_ROLE=FeasibilityUser +DATAPORTAL_BACKEND_KEYCLOAK_POWER_ROLE=FeasibilityPowerUser +DATAPORTAL_BACKEND_KEYCLOAK_ADMIN_ROLE=FeasibilityAdmin +DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL_ISSUER=https://auth.datenportal.localhost +DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL_JWK=http://auth:8080 +DATAPORTAL_BACKEND_KEYCLOAK_REALM=feasibility #---- Direct broker -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_ENABLED=true -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_USE_CQL=false -FEASIBILITY_BACKEND_BROKER_CLIENT_OBFUSCATE_RESULT_COUNT=false -FEASIBILITY_BACKEND_FLARE_WEBSERVICE_BASE_URL=http://flare:8080 -FEASIBILITY_BACKEND_CQL_SERVER_BASE_URL=http://fhir-server:8080/fhir -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME= -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD= -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL=https://keycloak.localhost:444/realms/blaze -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID=account -FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET=insecure +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_ENABLED=true +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_USE_CQL=false +DATAPORTAL_BACKEND_BROKER_CLIENT_OBFUSCATE_RESULT_COUNT=false +DATAPORTAL_BACKEND_FLARE_WEBSERVICE_BASE_URL=http://flare:8080 +DATAPORTAL_BACKEND_CQL_SERVER_BASE_URL=http://fhir-server:8080/fhir +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME= +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD= +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL=https://keycloak.localhost:444/realms/blaze +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID=account +DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET=insecure # ---- DSF broker -FEASIBILITY_BACKEND_DSF_ENABLED=false -FEASIBILITY_BACKEND_DSF_CACERT=/opt/codex-feasibility-security/ca.pem -FEASIBILITY_BACKEND_DSF_DSF_SECURITY_KEYSTORE_P12FILE=/opt/codex-feasibility-security/test-user.p12 -FEASIBILITY_BACKEND_DSF_SECURITY_KEYSTORE_PASSWORD=password -FEASIBILITY_BACKEND_DSF_WEBSERVICE_BASE_URL=https://dsf-zars-fhir-proxy/fhir -FEASIBILITY_BACKEND_DSF_WEBSOCKET_URL=wss://dsf-zars-fhir-proxy:443/fhir/ws -FEASIBILITY_BACKEND_DSF_ORGANIZATION_ID=Test_ZARS +DATAPORTAL_BACKEND_DSF_ENABLED=false +DATAPORTAL_BACKEND_DSF_CACERT=/opt/codex-feasibility-security/ca.pem +DATAPORTAL_BACKEND_DSF_DSF_SECURITY_KEYSTORE_P12FILE=/opt/codex-feasibility-security/test-user.p12 +DATAPORTAL_BACKEND_DSF_SECURITY_KEYSTORE_PASSWORD=password +DATAPORTAL_BACKEND_DSF_WEBSERVICE_BASE_URL=https://dsf-zars-fhir-proxy/fhir +DATAPORTAL_BACKEND_DSF_WEBSOCKET_URL=wss://dsf-zars-fhir-proxy:443/fhir/ws +DATAPORTAL_BACKEND_DSF_ORGANIZATION_ID=Test_ZARS # ---- privacy -FEASIBILITY_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_AMOUNT=3 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_INTERVALMINUTES=1 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_HARD_CREATE_AMOUNT=50 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_HARD_CREATE_INTERVALMINUTES=10080 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_SUMMARY_POLLINGINTERVALSECONDS=5 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_POLLINGINTERVALSECONDS=10 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_AMOUNT=10 -FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_INTERVALSECONDS=7200 -FEASIBILITY_BACKEND_PRIVACY_THRESHOLD_RESULTS=0 -FEASIBILITY_BACKEND_PRIVACY_THRESHOLD_SITES=1 -FEASIBILITY_BACKEND_QUERYRESULT_EXPIRY_MINUTES=5 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_AMOUNT=3 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_INTERVALMINUTES=1 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_HARD_CREATE_AMOUNT=50 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_HARD_CREATE_INTERVALMINUTES=10080 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_SUMMARY_POLLINGINTERVALSECONDS=5 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_POLLINGINTERVALSECONDS=10 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_AMOUNT=10 +DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_INTERVALSECONDS=7200 +DATAPORTAL_BACKEND_PRIVACY_THRESHOLD_RESULTS=0 +DATAPORTAL_BACKEND_PRIVACY_THRESHOLD_SITES=1 +DATAPORTAL_BACKEND_PRIVACY_THRESHOLD_SITES_RESULT=0 +DATAPORTAL_BACKEND_QUERYRESULT_EXPIRY_MINUTES=5 # ---- logging -FEASIBILITY_BACKEND_LOG_LEVEL_SQL=info -FEASIBILITY_BACKEND_LOG_LEVEL=info +DATAPORTAL_BACKEND_LOG_LEVEL_SQL=info +DATAPORTAL_BACKEND_LOG_LEVEL=info diff --git a/feasibility-portal/backend/docker-compose.yml b/feasibility-portal/backend/docker-compose.yml index 33c474c2..3a799850 100644 --- a/feasibility-portal/backend/docker-compose.yml +++ b/feasibility-portal/backend/docker-compose.yml @@ -1,95 +1,152 @@ services: - feasibility-gui-backend: - image: ghcr.io/medizininformatik-initiative/feasibility-backend:5.0.1 + dataportal-backend: + restart: unless-stopped + image: ghcr.io/medizininformatik-initiative/feasibility-backend:6.0.0-alpha.2 ports: - - ${FEASIBILITY_BACKEND_PORT:-127.0.0.1:8091}:8090 + - ${DATAPORTAL_BACKEND_PORT:-127.0.0.1:8091}:8090 depends_on: - - feasibility-gui-backend-db + dataportal-postgres: + condition: service_started + dataportal-elastic: + condition: service_healthy + init-elasticsearch: + condition: service_completed_successfully environment: # ----- app - QUERY_VALIDATION_ENABLED: ${FEASIBILITY_BACKEND_QUERY_VALIDATION_ENABLED:-true} - CQL_TRANSLATE_ENABLED: ${FEASIBILITY_BACKEND_CQL_TRANSLATE_ENABLED:-true} - FHIR_TRANSLATE_ENABLED: ${FEASIBILITY_BACKEND_FHIR_TRANSLATE_ENABLED:-false} - API_BASE_URL: ${FEASIBILITY_BACKEND_API_BASE_URL:-https://localhost/api/} - ALLOWED_ORIGINS: ${FEASIBILITY_BACKEND_ALLOWED_ORIGINS:-https://localhost} - QUERYRESULT_EXPIRY_MINUTES: ${FEASIBILITY_BACKEND_QUERYRESULT_EXPIRY_MINUTES:-5} - ONTOLOGY_ORDER: ${FEASIBILITY_BACKEND_ONTOLOGY_ORDER:-"Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung"} - MAX_SAVED_QUERIES_PER_USER: ${FEASIBILITY_BACKEND_MAX_SAVED_QUERIES_PER_USER:-100} + QUERY_VALIDATION_ENABLED: ${DATAPORTAL_BACKEND_QUERY_VALIDATION_ENABLED:-true} + CQL_TRANSLATE_ENABLED: ${DATAPORTAL_BACKEND_CQL_TRANSLATE_ENABLED:-true} + FHIR_TRANSLATE_ENABLED: ${DATAPORTAL_BACKEND_FHIR_TRANSLATE_ENABLED:-false} + API_BASE_URL: ${DATAPORTAL_BACKEND_API_BASE_URL:-https://localhost/api/} + ALLOWED_ORIGINS: ${DATAPORTAL_BACKEND_ALLOWED_ORIGINS:-https://localhost} + QUERYRESULT_EXPIRY_MINUTES: ${DATAPORTAL_BACKEND_QUERYRESULT_EXPIRY_MINUTES:-5} + ONTOLOGY_ORDER: ${DATAPORTAL_BACKEND_ONTOLOGY_ORDER:-"Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung"} + MAX_SAVED_QUERIES_PER_USER: ${DATAPORTAL_BACKEND_MAX_SAVED_QUERIES_PER_USER:-100} # ---- db config - FEASIBILITY_DATABASE_HOST: ${FEASIBILITY_BACKEND_DATASOURCE_HOST:-feasibility-gui-backend-db} - FEASIBILITY_DATABASE_PORT: ${FEASIBILITY_BACKEND_DATASOURCE_PORT:-5432} - FEASIBILITY_DATABASE_USER: ${FEASIBILITY_BACKEND_DATASOURCE_USERNAME:-guidbuser} - FEASIBILITY_DATABASE_PASSWORD: ${FEASIBILITY_BACKEND_DATASOURCE_PASSWORD:-guidbpw} + DATABASE_HOST: ${DATAPORTAL_BACKEND_DATASOURCE_HOST:-dataportal-backend-db} + DATABASE_PORT: ${DATAPORTAL_BACKEND_DATASOURCE_PORT:-5432} + DATABASE_USER: ${DATAPORTAL_BACKEND_DATASOURCE_USERNAME:-dataportaluser} + DATABASE_PASSWORD: ${DATAPORTAL_BACKEND_DATASOURCE_PASSWORD:-dataportalpw} + DATABASE_DBNAME: ${DATAPORTAL_BACKEND_DATASOURCE_DBNAME:-dataportal} + # ---- ontology + ONTOLOGY_FILES_FOLDER_UI: ${DATAPORTAL_BACKEND_ONTOLOGY_FILES_FOLDER:-/opt/dataportal-backend/ontology} + ONTOLOGY_DB_MIGRATION_FOLDER: ${DATAPORTAL_BACKEND_ONTOLOGY_DB_MIGRATION_FOLDER:-/opt/dataportal-backend/ontology/migration} + MAPPINGS_FILE: ${DATAPORTAL_BACKEND_ONTOLOGY_FILES_FOLDER:-/opt/dataportal-backend/ontology}/codex-term-code-mapping.json + CONCEPT_TREE_FILE: ${DATAPORTAL_BACKEND_ONTOLOGY_FILES_FOLDER:-/opt/dataportal-backend/ontology}/codex-code-tree.json # ---- auth - KEYCLOAK_ENABLED: ${FEASIBILITY_BACKEND_KEYCLOAK_ENABLED:-true} - KEYCLOAK_ALLOWED_ROLE: ${FEASIBILITY_BACKEND_KEYCLOAK_ALLOWED_ROLE:-FeasibilityUser} - KEYCLOAK_POWER_ROLE: ${FEASIBILITY_BACKEND_KEYCLOAK_POWER_ROLE:-FeasibilityPowerUser} - KEYCLOAK_ADMIN_ROLE: ${FEASIBILITY_BACKEND_KEYCLOAK_ADMIN_ROLE:-FeasibilityAdmin} - KEYCLOAK_BASE_URL_ISSUER: ${FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_ISSUER:-http://auth:8080} - KEYCLOAK_BASE_URL_JWK: ${FEASIBILITY_BACKEND_KEYCLOAK_BASE_URL_JWK:-http://auth:8080} - KEYCLOAK_REALM: ${FEASIBILITY_BACKEND_KEYCLOAK_REALM:-feasibility} + KEYCLOAK_ENABLED: ${DATAPORTAL_BACKEND_KEYCLOAK_ENABLED:-true} + KEYCLOAK_BASE_URL: ${DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL:-http://keycloak:8080} + KEYCLOAK_CLIENT_ID: ${DATAPORTAL_BACKEND_KEYCLOAK_CLIENT_ID:-dataportal-gui} + KEYCLOAK_ALLOWED_ROLE: ${DATAPORTAL_BACKEND_KEYCLOAK_ALLOWED_ROLE:-DataportalUser} + KEYCLOAK_POWER_ROLE: ${DATAPORTAL_BACKEND_KEYCLOAK_POWER_ROLE:-DataportalPowerUser} + KEYCLOAK_ADMIN_ROLE: ${DATAPORTAL_BACKEND_KEYCLOAK_ADMIN_ROLE:-DataportalAdmin} + KEYCLOAK_BASE_URL_ISSUER: ${DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL_ISSUER:-http://auth:8080} + KEYCLOAK_BASE_URL_JWK: ${DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL_JWK:-http://auth:8080} + KEYCLOAK_REALM: ${DATAPORTAL_BACKEND_KEYCLOAK_REALM:-dataportal} + #---- Mock broker + BROKER_CLIENT_MOCK_ENABLED: ${DATAPORTAL_BACKEND_BROKER_CLIENT_MOCK_ENABLED:-true} #---- Direct broker - BROKER_CLIENT_DIRECT_ENABLED: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_ENABLED:-false} - BROKER_CLIENT_DIRECT_USE_CQL: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_USE_CQL:-false} - BROKER_CLIENT_OBFUSCATE_RESULT_COUNT: ${FEASIBILITY_BACKEND_BROKER_CLIENT_OBFUSCATE_RESULT_COUNT:-false} - FLARE_WEBSERVICE_BASE_URL: ${FEASIBILITY_BACKEND_FLARE_WEBSERVICE_BASE_URL:-http://flare:8080} - CQL_SERVER_BASE_URL: ${FEASIBILITY_BACKEND_CQL_SERVER_BASE_URL:-http://fhir-server:8080/fhir} - BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME} - BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD} - BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL:-https://keycloak.localhost:444/realms/blaze} - BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID:-account} - BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET: ${FEASIBILITY_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET:-insecure} + BROKER_CLIENT_DIRECT_ENABLED: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_ENABLED:-false} + BROKER_CLIENT_DIRECT_USE_CQL: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_USE_CQL:-false} + BROKER_CLIENT_OBFUSCATE_RESULT_COUNT: ${DATAPORTAL_BACKEND_BROKER_CLIENT_OBFUSCATE_RESULT_COUNT:-false} + BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_USERNAME} + BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_BASIC_PASSWORD} + BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_ISSUER_URL:-https://keycloak.localhost:444/realms/blaze} + BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID:-account} + BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET:-insecure} + FLARE_WEBSERVICE_BASE_URL: ${DATAPORTAL_BACKEND_FLARE_WEBSERVICE_BASE_URL:-http://flare:8080} + CQL_SERVER_BASE_URL: ${DATAPORTAL_BACKEND_CQL_SERVER_BASE_URL:-http://fhir-server:8080/fhir} # ---- Aktin broker - BROKER_CLIENT_AKTIN_ENABLED: ${FEASIBILITY_BACKEND_AKTIN_ENABLED:-false} - AKTIN_BROKER_BASE_URL: ${FEASIBILITY_BACKEND_AKTIN_BROKER_BASE_URL:-http://aktin-broker:8080/broker/} - AKTIN_BROKER_API_KEY: ${FEASIBILITY_BACKEND_AKTIN_BROKER_API_KEY:-xxxApiKeyAdmin123} + BROKER_CLIENT_AKTIN_ENABLED: ${DATAPORTAL_BACKEND_AKTIN_ENABLED:-false} + AKTIN_BROKER_BASE_URL: ${DATAPORTAL_BACKEND_AKTIN_BROKER_BASE_URL:-http://aktin-broker:8080/broker/} + AKTIN_BROKER_API_KEY: ${DATAPORTAL_BACKEND_AKTIN_BROKER_API_KEY:-xxxApiKeyAdmin123} # ---- DSF broker - BROKER_CLIENT_DSF_ENABLED: ${FEASIBILITY_BACKEND_DSF_ENABLED:-false} - DSF_SECURITY_CACERT: ${FEASIBILITY_BACKEND_DSF_CACERT:-/opt/codex-feasibility-security/ca.pem} - DSF_SECURITY_KEYSTORE_P12FILE: ${FEASIBILITY_BACKEND_DSF_DSF_SECURITY_KEYSTORE_P12FILE:-/opt/codex-feasibility-security/test-user.p12} - DSF_SECURITY_KEYSTORE_PASSWORD: ${FEASIBILITY_BACKEND_DSF_SECURITY_KEYSTORE_PASSWORD:-password} - DSF_PROXY_HOST: ${FEASIBILITY_BACKEND_DSF_PROXY_HOST} - DSF_PROXY_USERNAME: ${FEASIBILITY_BACKEND_DSF_PROXY_USERNAME} - DSF_PROXY_PASSWORD: ${FEASIBILITY_BACKEND_DSF_PROXY_PASSWORD} - DSF_WEBSERVICE_BASE_URL: ${FEASIBILITY_BACKEND_DSF_WEBSERVICE_BASE_URL:-https://dsf-zars-fhir-proxy/fhir} - DSF_WEBSOCKET_URL: ${FEASIBILITY_BACKEND_DSF_WEBSOCKET_URL:-wss://dsf-zars-fhir-proxy:443/fhir/ws} - DSF_ORGANIZATION_ID: ${FEASIBILITY_BACKEND_DSF_ORGANIZATION_ID:-Test_ZARS} + BROKER_CLIENT_DSF_ENABLED: ${DATAPORTAL_BACKEND_DSF_ENABLED:-false} + DSF_SECURITY_CACERT: ${DATAPORTAL_BACKEND_DSF_CACERT:-/opt/dataportal-security/ca.pem} + DSF_SECURITY_KEYSTORE_P12FILE: ${DATAPORTAL_BACKEND_DSF_DSF_SECURITY_KEYSTORE_P12FILE:-/opt/dataportal-security/test-user.p12} + DSF_SECURITY_KEYSTORE_PASSWORD: ${DATAPORTAL_BACKEND_DSF_SECURITY_KEYSTORE_PASSWORD:-password} + DSF_PROXY_HOST: ${DATAPORTAL_BACKEND_DSF_PROXY_HOST} + DSF_PROXY_USERNAME: ${DATAPORTAL_BACKEND_DSF_PROXY_USERNAME} + DSF_PROXY_PASSWORD: ${DATAPORTAL_BACKEND_DSF_PROXY_PASSWORD} + DSF_WEBSERVICE_BASE_URL: ${DATAPORTAL_BACKEND_DSF_WEBSERVICE_BASE_URL:-https://dsf-zars-fhir-proxy/fhir} + DSF_WEBSOCKET_URL: ${DATAPORTAL_BACKEND_DSF_WEBSOCKET_URL:-wss://dsf-zars-fhir-proxy:443/fhir/ws} + DSF_ORGANIZATION_ID: ${DATAPORTAL_BACKEND_DSF_ORGANIZATION_ID:-Test_ZARS} # ---- privacy - PRIVACY_QUOTA_SOFT_CREATE_AMOUNT: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_AMOUNT:-3} - PRIVACY_QUOTA_SOFT_CREATE_INTERVALMINUTES: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_INTERVALMINUTES:-1} - PRIVACY_QUOTA_HARD_CREATE_AMOUNT: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_HARD_CREATE_AMOUNT:-50} - PRIVACY_QUOTA_HARD_CREATE_INTERVALMINUTES: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_HARD_CREATE_INTERVALMINUTES:-10080} - PRIVACY_QUOTA_READ_SUMMARY_POLLINGINTERVALSECONDS: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_SUMMARY_POLLINGINTERVALSECONDS:-10} - PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_POLLINGINTERVALSECONDS: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_POLLINGINTERVALSECONDS:-10} - PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_AMOUNT: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_AMOUNT:-3} - PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_INTERVALSECONDS: ${FEASIBILITY_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_INTERVALSECONDS:-7200} - PRIVACY_THRESHOLD_RESULTS: ${FEASIBILITY_BACKEND_PRIVACY_THRESHOLD_RESULTS:-20} - PRIVACY_THRESHOLD_SITES: ${FEASIBILITY_BACKEND_PRIVACY_THRESHOLD_SITES:-3} + PRIVACY_QUOTA_SOFT_CREATE_AMOUNT: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_AMOUNT:-3} + PRIVACY_QUOTA_SOFT_CREATE_INTERVALMINUTES: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_SOFT_CREATE_INTERVALMINUTES:-1} + PRIVACY_QUOTA_HARD_CREATE_AMOUNT: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_HARD_CREATE_AMOUNT:-50} + PRIVACY_QUOTA_HARD_CREATE_INTERVALMINUTES: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_HARD_CREATE_INTERVALMINUTES:-10080} + PRIVACY_QUOTA_READ_SUMMARY_POLLINGINTERVALSECONDS: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_SUMMARY_POLLINGINTERVALSECONDS:-10} + PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_POLLINGINTERVALSECONDS: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_POLLINGINTERVALSECONDS:-10} + PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_AMOUNT: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_AMOUNT:-3} + PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_INTERVALSECONDS: ${DATAPORTAL_BACKEND_PRIVACY_QUOTA_READ_DETAILED_OBFUSCATED_INTERVALSECONDS:-7200} + PRIVACY_THRESHOLD_RESULTS: ${DATAPORTAL_BACKEND_PRIVACY_THRESHOLD_RESULTS:-20} + PRIVACY_THRESHOLD_SITES: ${DATAPORTAL_BACKEND_PRIVACY_THRESHOLD_SITES:-3} + PRIVACY_THRESHOLD_SITES_RESULT: ${DATAPORTAL_BACKEND_PRIVACY_THRESHOLD_SITES_RESULT:-0} QUERYRESULT_DISABLE_LOG_FILE_ENCRYPTION: "true" + # ---- Elastic Search + ELASTIC_SEARCH_ENABLED: ${DATAPORTAL_BACKEND_ELASTIC_SEARCH_ENABLED:-true} + ELASTIC_SEARCH_HOST: ${DATAPORTAL_BACKEND_ELASTIC_SEARCH_HOST:-dataportal-elastic} + ELASTIC_SEARCH_FILTER: ${DATAPORTAL_BACKEND_ELASTIC_SEARCH_FILTER:-context,terminology,kds_module} # ---- logging - LOG_LEVEL_SQL: ${FEASIBILITY_BACKEND_LOG_LEVEL_SQL:-warn} - LOG_LEVEL: ${FEASIBILITY_BACKEND_LOG_LEVEL:-warn} - extra_hosts: - - "keycloak.localhost:host-gateway" - restart: unless-stopped + LOG_LEVEL_SQL: ${DATAPORTAL_BACKEND_LOG_LEVEL_SQL:-warn} + LOG_LEVEL: ${DATAPORTAL_BACKEND_LOG_LEVEL:-warn} volumes: - - ${FEASIBILITY_BACKEND_CERTS_PATH:-../secrets}:/opt/codex-feasibility-security + - ${DATAPORTAL_BACKEND_CERTS_PATH:-../secrets}:/opt/dataportal-security - ./certs:/opt/codex-feasibility-backend/certs - feasibility-gui-backend-db: - image: 'postgres:15-alpine' + dataportal-postgres: + image: 'postgres:16-alpine' ports: - - ${FEASIBILITY_BACKEND_DB_PORT:-127.0.0.1:5432}:5432 + - ${DATAPORTAL_BACKEND_DB_PORT:-127.0.0.1:5432}:5432 environment: - POSTGRES_USER: ${FEASIBILITY_BACKEND_DATASOURCE_USERNAME:-guidbuser} - POSTGRES_PASSWORD: ${FEASIBILITY_BACKEND_DATASOURCE_PASSWORD:-guidbpw} - POSTGRES_DB: codex_ui + POSTGRES_USER: ${DATAPORTAL_BACKEND_DATASOURCE_USERNAME:-dataportaluser} + POSTGRES_PASSWORD: ${DATAPORTAL_BACKEND_DATASOURCE_PASSWORD:-dataportalpw} + POSTGRES_DB: dataportal restart: unless-stopped volumes: - - type: volume - source: feas-backend-db-data - target: /var/lib/postgresql/data + - type: volume + source: dataportal-postgres-data + target: /var/lib/postgresql/data + dataportal-elastic: + image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0 + ports: + - "127.0.0.1:9200:9200" + - "127.0.0.1:9300:9300" + healthcheck: + test: [ "CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1" ] + interval: 30s + timeout: 30s + retries: 3 + environment: + discovery.type: single-node + ES_JAVA_OPTS: -Xmx512m -Xms512m + node.name: es01 + cluster.name: elasticsearch + xpack.security.enabled: false + volumes: + - type: volume + source: dataportal-elastic-data + target: /usr/share/elasticsearch/data + init-elasticsearch: + image: curlimages/curl:8.8.0 + depends_on: + dataportal-elastic: + condition: service_healthy + environment: + ELASTIC_HOST: http://dataportal-elastic:9200 + ELASTIC_GIT_TAG: v3.0.0-test.2 + ELASTIC_FILEPATH: https://github.com/medizininformatik-initiative/fhir-ontology-generator/raw/TAGPLACEHOLDER/example/fdpg-ontology/ + ELASTIC_FILENAME: elastic.zip + # if set to false, existing indices are not overridden. + OVERRIDE_EXISTING: true + entrypoint: [ "sh", "-c", "/tmp/init.sh" ] + volumes: + - type: bind + source: ./elastic-init.sh + target: /tmp/init.sh volumes: - feas-backend-db-data: - name: "feas-backend-db-data" + dataportal-postgres-data: + name: "dataportal-postgres-data" + dataportal-elastic-data: + name: "dataportal-elastic-data" \ No newline at end of file diff --git a/feasibility-portal/backend/elastic-init.sh b/feasibility-portal/backend/elastic-init.sh new file mode 100755 index 00000000..491dc65f --- /dev/null +++ b/feasibility-portal/backend/elastic-init.sh @@ -0,0 +1,54 @@ +#!/bin/sh + +# Wait for Elasticsearch to start up before doing anything +until curl -X GET "$ELASTIC_HOST/_cluster/health" | grep -q '"status":"green"\|"status":"yellow"'; do + echo "Waiting for Elasticsearch..." + sleep 5 +done + +ABSOLUTE_FILEPATH="${ELASTIC_FILEPATH//TAGPLACEHOLDER/$ELASTIC_GIT_TAG}$ELASTIC_FILENAME" +echo "Downloading $ABSOLUTE_FILEPATH" +response_onto_dl=$(curl --write-out "%{http_code}" -sLO "$ABSOLUTE_FILEPATH") + +if [ "$response_onto_dl" -ne 200 ]; then + echo "Could not download ontology file. Maybe the tag $ELASTIC_GIT_TAG does not exist? Error code was $response_onto_dl" + exit 1 +fi + +unzip -o "$ELASTIC_FILENAME" + +if [ "$OVERRIDE_EXISTING" = "true" ]; then + echo "(Trying to) delete existing indices" + curl -s -DELETE "$ELASTIC_HOST/ontology" + curl -s -DELETE "$ELASTIC_HOST/codeable_concept" +fi + +echo "Creating ontology index..." +response_onto=$(curl --write-out "%{http_code}" -s --output /dev/null -XPUT -H 'Content-Type: application/json' "$ELASTIC_HOST/ontology" -d @elastic/ontology_index.json) +echo "Creating codeable concept index..." +response_cc=$(curl --write-out "%{http_code}" -s --output /dev/null -XPUT -H 'Content-Type: application/json' "$ELASTIC_HOST/codeable_concept" -d @elastic/codeable_concept_index.json) +echo "Done" + +for FILE in elastic/*; do + if [ -f "$FILE" ]; then + BASENAME=$(basename "$FILE") + if [[ $BASENAME == onto_es__ontology* && $BASENAME == *.json ]]; then + if [[ "$response_onto" -eq 200 || "$OVERRIDE_EXISTING" = "true" ]]; then + echo "Uploading $BASENAME" + curl -s --output /dev/null -XPOST -H 'Content-Type: application/json' --data-binary @"$FILE" "$ELASTIC_HOST/ontology/_bulk?pretty" + else + echo "Skipping $BASENAME because index was already existing. Set OVERRIDE_EXISTING to true to force creating a new index" + fi + fi + if [[ $BASENAME == onto_es__codeable_concept* && $BASENAME == *.json ]]; then + if [[ "$response_cc" -eq 200 || "$OVERRIDE_EXISTING" = "true" ]]; then + echo "Uploading $BASENAME" + curl -s --output /dev/null -XPOST -H 'Content-Type: application/json' --data-binary @"$FILE" "$ELASTIC_HOST/codeable_concept/_bulk?pretty" + else + echo "Skipping $BASENAME because index was already existing. Set OVERRIDE_EXISTING to true to force creating a new index" + fi + fi + fi +done + +echo "All done" diff --git a/feasibility-portal/gui/deploy-config.json.default b/feasibility-portal/gui/deploy-config.json.default index b5d0ff54..beaef208 100644 --- a/feasibility-portal/gui/deploy-config.json.default +++ b/feasibility-portal/gui/deploy-config.json.default @@ -6,7 +6,7 @@ "baseUrl": "/api" }, "uiBackendApi": { - "baseUrl": "https://api.datenportal.localhost/api/v3" + "baseUrl": "https://api.datenportal.localhost/api/v4" }, "auth": { "baseUrl": "https://auth.datenportal.localhost", diff --git a/feasibility-portal/gui/docker-compose.yml b/feasibility-portal/gui/docker-compose.yml index 3f833f4f..9391085f 100644 --- a/feasibility-portal/gui/docker-compose.yml +++ b/feasibility-portal/gui/docker-compose.yml @@ -1,6 +1,6 @@ services: dataportal-ui: - image: ghcr.io/medizininformatik-initiative/feasibility-gui:5.0.0 + image: ghcr.io/medizininformatik-initiative/feasibility-gui:6.0.0-test.1 restart: unless-stopped ports: - "127.0.0.1:8080:8080" diff --git a/feasibility-portal/proxy/conf.d/backend.conf.template b/feasibility-portal/proxy/conf.d/backend.conf.template index 9d0ca7a2..a7410dcc 100644 --- a/feasibility-portal/proxy/conf.d/backend.conf.template +++ b/feasibility-portal/proxy/conf.d/backend.conf.template @@ -4,7 +4,7 @@ server_name ${BACKEND_HOSTNAME}; location / { - proxy_pass http://feasibility-gui-backend:8090/; + proxy_pass http://dataportal-backend:8090/; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; } From 71713bd9445f83754487fb0be6f0dcb3730bc383 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Wed, 4 Sep 2024 16:49:06 +0200 Subject: [PATCH 03/15] fix: update backend db url --- feasibility-portal/backend/.env.default | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/feasibility-portal/backend/.env.default b/feasibility-portal/backend/.env.default index b098fe72..5daf7c88 100644 --- a/feasibility-portal/backend/.env.default +++ b/feasibility-portal/backend/.env.default @@ -6,7 +6,7 @@ DATAPORTAL_BACKEND_ALLOWED_ORIGINS=https://datenportal.localhost DATAPORTAL_BACKEND_ONTOLOGY_ORDER="Diagnose, Prozedur, Person, Laboruntersuchung, Medikamentenverabreichung, Bioprobe, Einwilligung" DATAPORTAL_BACKEND_MAX_SAVED_QUERIES_PER_USER=100 # ---- db config -DATAPORTAL_BACKEND_DATASOURCE_HOST=feasibility-gui-backend-db +DATAPORTAL_BACKEND_DATASOURCE_HOST=dataportal-postgres DATAPORTAL_BACKEND_DATASOURCE_PORT=5432 DATAPORTAL_BACKEND_DATASOURCE_USERNAME=guidbuser DATAPORTAL_BACKEND_DATASOURCE_PASSWORD=guidbpw From b4a6f50ce9da4b20faa94c9f4a943aaa282eac19 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Wed, 4 Sep 2024 16:57:09 +0200 Subject: [PATCH 04/15] fix secrets and certs paths --- feasibility-portal/backend/.env.default | 4 ++-- feasibility-portal/backend/docker-compose.yml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/feasibility-portal/backend/.env.default b/feasibility-portal/backend/.env.default index 5daf7c88..6766d42c 100644 --- a/feasibility-portal/backend/.env.default +++ b/feasibility-portal/backend/.env.default @@ -31,8 +31,8 @@ DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_ID=account DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_AUTH_OAUTH_CLIENT_SECRET=insecure # ---- DSF broker DATAPORTAL_BACKEND_DSF_ENABLED=false -DATAPORTAL_BACKEND_DSF_CACERT=/opt/codex-feasibility-security/ca.pem -DATAPORTAL_BACKEND_DSF_DSF_SECURITY_KEYSTORE_P12FILE=/opt/codex-feasibility-security/test-user.p12 +DATAPORTAL_BACKEND_DSF_CACERT=/opt/dataportal-security/ca.pem +DATAPORTAL_BACKEND_DSF_DSF_SECURITY_KEYSTORE_P12FILE=/opt/dataportal-security/test-user.p12 DATAPORTAL_BACKEND_DSF_SECURITY_KEYSTORE_PASSWORD=password DATAPORTAL_BACKEND_DSF_WEBSERVICE_BASE_URL=https://dsf-zars-fhir-proxy/fhir DATAPORTAL_BACKEND_DSF_WEBSOCKET_URL=wss://dsf-zars-fhir-proxy:443/fhir/ws diff --git a/feasibility-portal/backend/docker-compose.yml b/feasibility-portal/backend/docker-compose.yml index 3a799850..e9393d5f 100644 --- a/feasibility-portal/backend/docker-compose.yml +++ b/feasibility-portal/backend/docker-compose.yml @@ -92,7 +92,7 @@ services: LOG_LEVEL: ${DATAPORTAL_BACKEND_LOG_LEVEL:-warn} volumes: - ${DATAPORTAL_BACKEND_CERTS_PATH:-../secrets}:/opt/dataportal-security - - ./certs:/opt/codex-feasibility-backend/certs + - ./certs:/opt/dataportal-backend/certs dataportal-postgres: image: 'postgres:16-alpine' From 2131381cae3cde60238805a3a91c34a25c2ce549 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Sun, 8 Sep 2024 19:38:46 +0200 Subject: [PATCH 05/15] Update ui and backend --- feasibility-portal/backend/docker-compose.yml | 2 +- feasibility-portal/gui/docker-compose.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/feasibility-portal/backend/docker-compose.yml b/feasibility-portal/backend/docker-compose.yml index e9393d5f..f5c34cad 100644 --- a/feasibility-portal/backend/docker-compose.yml +++ b/feasibility-portal/backend/docker-compose.yml @@ -1,7 +1,7 @@ services: dataportal-backend: restart: unless-stopped - image: ghcr.io/medizininformatik-initiative/feasibility-backend:6.0.0-alpha.2 + image: ghcr.io/medizininformatik-initiative/feasibility-backend:6.0.0-test.1 ports: - ${DATAPORTAL_BACKEND_PORT:-127.0.0.1:8091}:8090 depends_on: diff --git a/feasibility-portal/gui/docker-compose.yml b/feasibility-portal/gui/docker-compose.yml index 9391085f..3d534d34 100644 --- a/feasibility-portal/gui/docker-compose.yml +++ b/feasibility-portal/gui/docker-compose.yml @@ -1,6 +1,6 @@ services: dataportal-ui: - image: ghcr.io/medizininformatik-initiative/feasibility-gui:6.0.0-test.1 + image: ghcr.io/medizininformatik-initiative/feasibility-gui:6.0.0-test.4 restart: unless-stopped ports: - "127.0.0.1:8080:8080" From d67d9b8f05bff2478161e3af586d28f6456449e2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 14 Oct 2024 19:35:09 +0200 Subject: [PATCH 06/15] add torch to deploy --- feasibility-triangle/down-triangle.sh | 1 + .../fhir-server/docker-compose.yml | 4 -- .../initialise-triangle-env-files.sh | 2 +- .../rev-proxy/conf.d/torch.conf.template | 29 ++++++++++ .../rev-proxy/context-paths.nginx.conf | 18 ++++++ .../rev-proxy/docker-compose.yml | 4 +- feasibility-triangle/start-triangle.sh | 11 ++++ feasibility-triangle/stop-triangle.sh | 1 + feasibility-triangle/torch/.env.default | 15 +++++ feasibility-triangle/torch/docker-compose.yml | 37 ++++++++++++ feasibility-triangle/torch/example-crtdl.json | 56 +++++++++++++++++++ feasibility-triangle/torch/execute-crtdl.sh | 45 +++++++++++++++ feasibility-triangle/torch/torch.nginx.conf | 27 +++++++++ 13 files changed, 242 insertions(+), 8 deletions(-) create mode 100644 feasibility-triangle/rev-proxy/conf.d/torch.conf.template create mode 100644 feasibility-triangle/torch/.env.default create mode 100644 feasibility-triangle/torch/docker-compose.yml create mode 100644 feasibility-triangle/torch/example-crtdl.json create mode 100755 feasibility-triangle/torch/execute-crtdl.sh create mode 100644 feasibility-triangle/torch/torch.nginx.conf diff --git a/feasibility-triangle/down-triangle.sh b/feasibility-triangle/down-triangle.sh index 27de6e72..b55baf90 100755 --- a/feasibility-triangle/down-triangle.sh +++ b/feasibility-triangle/down-triangle.sh @@ -4,6 +4,7 @@ COMPOSE_PROJECT=${FEASIBILITY_COMPOSE_PROJECT:-feasibility-deploy} BASE_DIR="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 || exit 1 ; pwd -P )" docker compose -p "$COMPOSE_PROJECT" -f "$BASE_DIR"/rev-proxy/docker-compose.yml \ + -f "$BASE_DIR"/torch/docker-compose.yml \ -f "$BASE_DIR"/fhir-server/docker-compose.yml \ -f "$BASE_DIR"/fhir-server/keycloak.docker-compose.yml \ -f "$BASE_DIR"/flare/docker-compose.yml down -v diff --git a/feasibility-triangle/fhir-server/docker-compose.yml b/feasibility-triangle/fhir-server/docker-compose.yml index 511045f5..c08a88ee 100644 --- a/feasibility-triangle/fhir-server/docker-compose.yml +++ b/feasibility-triangle/fhir-server/docker-compose.yml @@ -15,10 +15,6 @@ services: DB_BLOCK_CACHE_SIZE: ${BLAZE_BLOCK_CACHE_SIZE:-256} DB_RESOURCE_CACHE_SIZE: ${BLAZE_DB_RESOURCE_CACHE_SIZE:-2000000} DB_SEARCH_PARAM_BUNDLE: "/app/custom-search-parameters.json" - ENABLE_ADMIN_API: "true" - OPENID_PROVIDER_URL: ${FHIR_SERVER_OPENID_PROVIDER_URL} - OPENID_CLIENT_TRUST_STORE: "/app/trust-store.p12" - OPENID_CLIENT_TRUST_STORE_PASS: ${FHIR_SERVER_OPENID_CLIENT_TRUST_STORE_PASS:-insecure} ports: - ${PORT_FHIR_SERVER_LOCALHOST:-127.0.0.1:8081}:8080 extra_hosts: diff --git a/feasibility-triangle/initialise-triangle-env-files.sh b/feasibility-triangle/initialise-triangle-env-files.sh index 39fdc8ac..b32415ac 100755 --- a/feasibility-triangle/initialise-triangle-env-files.sh +++ b/feasibility-triangle/initialise-triangle-env-files.sh @@ -1,7 +1,7 @@ #!/bin/bash BASE_DIR="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 || exit 1 ; pwd -P )" -envfiles=( "$BASE_DIR/fhir-server/.env" "$BASE_DIR/flare/.env" "$BASE_DIR/rev-proxy/.env") +envfiles=( "$BASE_DIR/fhir-server/.env" "$BASE_DIR/flare/.env" "$BASE_DIR/torch/.env" "$BASE_DIR/rev-proxy/.env") for file in "${envfiles[@]}" do diff --git a/feasibility-triangle/rev-proxy/conf.d/torch.conf.template b/feasibility-triangle/rev-proxy/conf.d/torch.conf.template new file mode 100644 index 00000000..516e5fe5 --- /dev/null +++ b/feasibility-triangle/rev-proxy/conf.d/torch.conf.template @@ -0,0 +1,29 @@ +server { + listen 8443 ssl; + http2 on; + server_name ${TORCH_HOSTNAME}; + + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + + location / { + auth_basic "Restricted"; + auth_basic_user_file /etc/nginx/.htpasswd; + set $upstream torch:8080; + proxy_pass http://$upstream$request_uri; + proxy_set_header Authorization ""; + } + + location /fileserver/ { + auth_basic "Restricted"; + auth_basic_user_file /etc/nginx/.htpasswd; + set $upstream torch-nginx:8080; + rewrite ^/fileserver/(.*)$ /$1 break; + proxy_pass http://torch-nginx:8080; + proxy_set_header Authorization ""; + } +} diff --git a/feasibility-triangle/rev-proxy/context-paths.nginx.conf b/feasibility-triangle/rev-proxy/context-paths.nginx.conf index c2c2732f..eac543b9 100644 --- a/feasibility-triangle/rev-proxy/context-paths.nginx.conf +++ b/feasibility-triangle/rev-proxy/context-paths.nginx.conf @@ -109,6 +109,22 @@ http { proxy_pass $auth_upstream; } + location /torch { + auth_basic "Restricted"; + auth_basic_user_file /etc/nginx/.htpasswd; + set $torch_upstream http://torch:8080; + proxy_pass $torch_upstream; + proxy_set_header Authorization ""; + } + + location /torch/fileserver { + auth_basic "Restricted"; + auth_basic_user_file /etc/nginx/.htpasswd; + set $torch_nginx_upstream http://torch-nginx:8080; + proxy_pass $torch_nginx_upstream; + proxy_set_header Authorization ""; + } + location / { auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; @@ -116,5 +132,7 @@ http { proxy_pass $flare_upstream; proxy_set_header Authorization ""; } + + } } diff --git a/feasibility-triangle/rev-proxy/docker-compose.yml b/feasibility-triangle/rev-proxy/docker-compose.yml index ab8abd7e..8e1bfa70 100644 --- a/feasibility-triangle/rev-proxy/docker-compose.yml +++ b/feasibility-triangle/rev-proxy/docker-compose.yml @@ -4,11 +4,9 @@ services: image: nginxinc/nginx-unprivileged:1.25.5-alpine environment: FHIR_SERVER_HOSTNAME: ${FHIR_SERVER_HOSTNAME:-fhir.localhost} - KEYCLOAK_HOSTNAME: ${KEYCLOAK_HOSTNAME:-auth.localhost} - - FLARE_HOSTNAME: ${FLARE_HOSTNAME:-flare.localhost} + TORCH_HOSTNAME: ${TORCH_HOSTNAME:-torch.localhost} ports: - ${FEASIBILITY_TRIANGLE_REV_PROXY_PORT:-444}:8443 volumes: diff --git a/feasibility-triangle/start-triangle.sh b/feasibility-triangle/start-triangle.sh index ecd21f57..d06df279 100755 --- a/feasibility-triangle/start-triangle.sh +++ b/feasibility-triangle/start-triangle.sh @@ -40,3 +40,14 @@ else rm "$BASE_DIR/rev-proxy/conf.d/flare.conf" fi fi + +if [ -f "$BASE_DIR/torch/.env" ] && grep -qE '^TORCH_ENABLED=true\s*$' "$BASE_DIR/torch/.env"; then + if [ ! -f "$BASE_DIR/rev-proxy/conf.d/torch.conf" ]; then + cp "$BASE_DIR/rev-proxy/conf.d/torch.conf.template" "$BASE_DIR/rev-proxy/conf.d/torch.conf" + fi + COMPOSE_IGNORE_ORPHANS=True docker compose -p "$COMPOSE_PROJECT" -f "$BASE_DIR"/torch/docker-compose.yml up -d +else + if [ -f "$BASE_DIR/rev-proxy/conf.d/torch.conf" ]; then + rm "$BASE_DIR/rev-proxy/conf.d/torch.conf" + fi +fi \ No newline at end of file diff --git a/feasibility-triangle/stop-triangle.sh b/feasibility-triangle/stop-triangle.sh index b32e0e60..6a57880f 100755 --- a/feasibility-triangle/stop-triangle.sh +++ b/feasibility-triangle/stop-triangle.sh @@ -5,6 +5,7 @@ COMPOSE_PROJECT=${FEASIBILITY_COMPOSE_PROJECT:-feasibility-deploy} BASE_DIR="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 || exit 1 ; pwd -P )" docker compose -p "$COMPOSE_PROJECT" -f "$BASE_DIR"/flare/docker-compose.yml \ + -f "$BASE_DIR"/torch/docker-compose.yml \ -f "$BASE_DIR"/fhir-server/docker-compose.yml \ -f "$BASE_DIR"/fhir-server/keycloak.docker-compose.yml \ -f "$BASE_DIR"/rev-proxy/docker-compose.yml stop diff --git a/feasibility-triangle/torch/.env.default b/feasibility-triangle/torch/.env.default new file mode 100644 index 00000000..c3ec73f8 --- /dev/null +++ b/feasibility-triangle/torch/.env.default @@ -0,0 +1,15 @@ +TORCH_SERVER_PORT=8080 +TORCH_PROFILE_DIR=/app/structureDefinitions +TORCH_MAPPING_CONSENT=/app/Mappings/consent-mappings_fhir.json +TORCH_MAPPING_CONSENT_TO_PROFILE=/app/Mappings/profile_to_consent.json +TORCH_FHIR_URL=http://torch-data-store:8080/fhir +TORCH_FLARE_URL=http://torch-flare:8080 +TORCH_RESULTS_DIR=/app/output +TORCH_RESULTS_PERSISTENCE=PT12H30M5S +TORCH_LOG_LEVEL=debug +TORCH_NGINX_SERVERNAME=localhost +TORCH_NGINX_FILELOCATION=http://localhost:80 +TORCH_BATCHSIZE=100 +TORCH_MAXCONCURRENCY=4 +TORCH_USE_CQL=true +TORCH_ENABLED=true \ No newline at end of file diff --git a/feasibility-triangle/torch/docker-compose.yml b/feasibility-triangle/torch/docker-compose.yml new file mode 100644 index 00000000..521103a0 --- /dev/null +++ b/feasibility-triangle/torch/docker-compose.yml @@ -0,0 +1,37 @@ +services: + torch-nginx: + restart: unless-stopped + image: nginxinc/nginx-unprivileged:1.25.5-alpine + ports: + - ${TORCH_NGINX_PORT:-127.0.0.1:80}:8080 + volumes: + - ./torch.nginx.conf:/etc/nginx/nginx.conf + - triangle-torch-data-store:/app/output + torch: + restart: unless-stopped + image: ghcr.io/medizininformatik-initiative/torch:develop + ports: + - ${TORCH_PORT:-127.0.0.1:8086}:8080 + environment: + SERVER_PORT: ${TORCH_SERVER_PORT:-8080} + TORCH_PROFILE_DIR: ${TORCH_PROFILE_DIR:-/app/structureDefinitions} + TORCH_MAPPING_CONSENT: ${TORCH_MAPPING_CONSENT:-/app/Mappings/consent-mappings_fhir.json} + TORCH_MAPPING_CONSENT_TO_PROFILE: ${TORCH_MAPPING_CONSENT_TO_PROFILE:-/app/Mappings/profile_to_consent.json} + TORCH_FHIR_URL: ${TORCH_FHIR_URL:-http://torch-data-store:8080/fhir} + TORCH_FLARE_URL: ${TORCH_FLARE_URL:-http://torch-flare:8080} + TORCH_RESULTS_DIR: ${TORCH_RESULTS_DIR:-/app/output} + TORCH_RESULTS_PERSISTENCE: ${TORCH_RESULTS_PERSISTENCE:-PT12H30M5S} + LOG_LEVEL: ${TORCH_LOG_LEVEL:-debug} + NGINX_SERVERNAME: ${TORCH_NGINX_SERVERNAME:-localhost} + NGINX_FILELOCATION: ${TORCH_NGINX_FILELOCATION:-http://localhost:80} + TORCH_BATCHSIZE: ${TORCH_BATCHSIZE:-100} + TORCH_MAXCONCURRENCY: ${TORCH_MAXCONCURRENCY:-4} + TORCH_USE_CQL: ${TORCH_USE_CQL:-true} + volumes: + - "triangle-torch-data-store:/app/output" + - ../auth:/app/certs + extra_hosts: + - "auth.localhost:host-gateway" + +volumes: + triangle-torch-data-store: \ No newline at end of file diff --git a/feasibility-triangle/torch/example-crtdl.json b/feasibility-triangle/torch/example-crtdl.json new file mode 100644 index 00000000..009f2ee2 --- /dev/null +++ b/feasibility-triangle/torch/example-crtdl.json @@ -0,0 +1,56 @@ +{ + "display": "", + "version": "http://json-schema.org/to-be-done/schema#", + "cohortDefinition": { + "version": "http://to_be_decided.com/draft-1/schema#", + "display": "Ausgewählte Merkmale", + "inclusionCriteria": [ + [ + { + "termCodes": [ + { + "code": "263495000", + "display": "Geschlecht", + "system": "http://snomed.info/sct", + "version": "" + } + ], + "context": { + "code": "Patient", + "display": "Patient", + "system": "fdpg.mii.cds", + "version": "1.0.0" + }, + "valueFilter": { + "selectedConcepts": [ + { + "code": "male", + "display": "Male", + "system": "http://hl7.org/fhir/administrative-gender", + "version": "2099" + } + ], + "type": "concept" + } + } + ] + ] + }, + "dataExtraction": { + "attributeGroups": [ + { + "groupReference": "https://www.medizininformatik-initiative.de/fhir/core/modul-diagnose/StructureDefinition/Diagnose", + "attributes": [ + { + "attributeRef": "Condition.code", + "mustHave": false + }, + { + "attributeRef": "Condition.recordedDate", + "mustHave": false + } + ] + } + ] + } +} \ No newline at end of file diff --git a/feasibility-triangle/torch/execute-crtdl.sh b/feasibility-triangle/torch/execute-crtdl.sh new file mode 100755 index 00000000..b131b2eb --- /dev/null +++ b/feasibility-triangle/torch/execute-crtdl.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env sh + +TORCH_BASE_URL=${TORCH_BASE_URL:-https://torch.localhost:444} +TORCH_USERNAME=${TORCH_USERNAME:-"test"} +TORCH_PASSWORD=${TORCH_PASSWORD:-"tast"} +CURL_INSECURE=${CURL_INSECURE:-false} + +CURL_OPTIONS="" +if [ "$CURL_INSECURE" == "true" ]; then + CURL_OPTIONS="-k" +fi + +TORCH_AUTHORIZATION=$(echo -n "${TORCH_USERNAME}:${TORCH_PASSWORD}" | base64) + +if [ $# -ne 1 ]; then + echo "Usage: $0 " + exit 1 +fi + +json_file="$1" +json_string=$(<"$json_file") + +base64_encoded=$(echo -n "$json_string" | base64) + +response=$(curl --location -i $CURL_OPTIONS "${TORCH_BASE_URL}/fhir/\$extract-data" \ +--header 'Content-Type: application/fhir+json' \ +--header "Authorization: Basic ${TORCH_AUTHORIZATION}" \ +--data '{ + "resourceType": "Parameters", + "id": "example6", + "parameter": [ + { + "name": "crtdl", + "valueBase64Binary": "'"$base64_encoded"'" + } + ] +}') + +content_location=$(echo "$response" | grep -i 'Content-Location:' | awk '{print $2}' | tr -d '\r') + +if [ -n "$content_location" ]; then + echo "Extraction submitted, find your extraction under: $TORCH_BASE_URL$content_location" +else + echo "Content-Location header not found in the response." +fi \ No newline at end of file diff --git a/feasibility-triangle/torch/torch.nginx.conf b/feasibility-triangle/torch/torch.nginx.conf new file mode 100644 index 00000000..33a10d62 --- /dev/null +++ b/feasibility-triangle/torch/torch.nginx.conf @@ -0,0 +1,27 @@ +events{} +pid /tmp/nginx.pid; + +http { + server { + listen 8080; + server_name localhost; + + root /app/output; + + index index.html; + + location / { + try_files $uri $uri/ =404; + autoindex off; + } + + access_log /var/log/nginx/access.log; + error_log /var/log/nginx/error.log; + + error_page 404 /404.html; + location = /404.html { + internal; + } + } +} + From 9a2155db97dc051f570c945de0e2e8c7efcf2be4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 14 Oct 2024 19:36:20 +0200 Subject: [PATCH 07/15] undo remove of fhir oauth --- feasibility-triangle/fhir-server/docker-compose.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/feasibility-triangle/fhir-server/docker-compose.yml b/feasibility-triangle/fhir-server/docker-compose.yml index c08a88ee..511045f5 100644 --- a/feasibility-triangle/fhir-server/docker-compose.yml +++ b/feasibility-triangle/fhir-server/docker-compose.yml @@ -15,6 +15,10 @@ services: DB_BLOCK_CACHE_SIZE: ${BLAZE_BLOCK_CACHE_SIZE:-256} DB_RESOURCE_CACHE_SIZE: ${BLAZE_DB_RESOURCE_CACHE_SIZE:-2000000} DB_SEARCH_PARAM_BUNDLE: "/app/custom-search-parameters.json" + ENABLE_ADMIN_API: "true" + OPENID_PROVIDER_URL: ${FHIR_SERVER_OPENID_PROVIDER_URL} + OPENID_CLIENT_TRUST_STORE: "/app/trust-store.p12" + OPENID_CLIENT_TRUST_STORE_PASS: ${FHIR_SERVER_OPENID_CLIENT_TRUST_STORE_PASS:-insecure} ports: - ${PORT_FHIR_SERVER_LOCALHOST:-127.0.0.1:8081}:8080 extra_hosts: From 36fd2ed17f4d1d097db4e8d8dc82d2cb5b3ac22e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 14 Oct 2024 19:44:26 +0200 Subject: [PATCH 08/15] fix lint issues execute crtdl --- feasibility-triangle/torch/execute-crtdl.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/feasibility-triangle/torch/execute-crtdl.sh b/feasibility-triangle/torch/execute-crtdl.sh index b131b2eb..550c3873 100755 --- a/feasibility-triangle/torch/execute-crtdl.sh +++ b/feasibility-triangle/torch/execute-crtdl.sh @@ -6,21 +6,21 @@ TORCH_PASSWORD=${TORCH_PASSWORD:-"tast"} CURL_INSECURE=${CURL_INSECURE:-false} CURL_OPTIONS="" -if [ "$CURL_INSECURE" == "true" ]; then +if [ "$CURL_INSECURE" = "true" ]; then CURL_OPTIONS="-k" fi -TORCH_AUTHORIZATION=$(echo -n "${TORCH_USERNAME}:${TORCH_PASSWORD}" | base64) +TORCH_AUTHORIZATION=$(printf "%s:%s" "$TORCH_USERNAME" "$TORCH_PASSWORD" | base64) -if [ $# -ne 1 ]; then - echo "Usage: $0 " +if [ "$#" -ne 1 ]; then + printf "Usage: %s \n" "$0" exit 1 fi json_file="$1" -json_string=$(<"$json_file") +json_string=$(cat "$json_file") -base64_encoded=$(echo -n "$json_string" | base64) +base64_encoded=$(printf "%s" "$json_string" | base64) response=$(curl --location -i $CURL_OPTIONS "${TORCH_BASE_URL}/fhir/\$extract-data" \ --header 'Content-Type: application/fhir+json' \ @@ -36,10 +36,10 @@ response=$(curl --location -i $CURL_OPTIONS "${TORCH_BASE_URL}/fhir/\$extract-da ] }') -content_location=$(echo "$response" | grep -i 'Content-Location:' | awk '{print $2}' | tr -d '\r') +content_location=$(printf "%s" "$response" | grep -i 'Content-Location:' | awk '{print $2}' | tr -d '\r') if [ -n "$content_location" ]; then - echo "Extraction submitted, find your extraction under: $TORCH_BASE_URL$content_location" + printf "Extraction submitted, find your extraction under: %s$content_location\n" "$TORCH_BASE_URL" else - echo "Content-Location header not found in the response." -fi \ No newline at end of file + printf "Content-Location header not found in the response.\n" +fi From f1b5d5aacef211b18163b6458f91c600192a0bcd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 14 Oct 2024 21:26:16 +0200 Subject: [PATCH 09/15] fix context path torch setup --- feasibility-triangle/rev-proxy/context-paths.nginx.conf | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/feasibility-triangle/rev-proxy/context-paths.nginx.conf b/feasibility-triangle/rev-proxy/context-paths.nginx.conf index eac543b9..0ffe0719 100644 --- a/feasibility-triangle/rev-proxy/context-paths.nginx.conf +++ b/feasibility-triangle/rev-proxy/context-paths.nginx.conf @@ -109,18 +109,20 @@ http { proxy_pass $auth_upstream; } - location /torch { + location /torch/ { auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; set $torch_upstream http://torch:8080; + rewrite ^/torch/(.*)$ /$1 break; proxy_pass $torch_upstream; proxy_set_header Authorization ""; } - location /torch/fileserver { + location /torch/fileserver/ { auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; set $torch_nginx_upstream http://torch-nginx:8080; + rewrite ^/torch/fileserver/(.*)$ /$1 break; proxy_pass $torch_nginx_upstream; proxy_set_header Authorization ""; } From c1ceabd5c0defd0cd85387f8508fb352d68ee073 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 14 Oct 2024 22:04:58 +0200 Subject: [PATCH 10/15] add torch to readme, remove unnecessary torch env vars --- feasibility-triangle/README.md | 17 ++++++++++++++--- feasibility-triangle/rev-proxy/.env.default | 1 + feasibility-triangle/torch/.env.default | 6 ------ feasibility-triangle/torch/docker-compose.yml | 6 ------ 4 files changed, 15 insertions(+), 15 deletions(-) diff --git a/feasibility-triangle/README.md b/feasibility-triangle/README.md index e7e9bba0..0b86ccca 100644 --- a/feasibility-triangle/README.md +++ b/feasibility-triangle/README.md @@ -94,7 +94,7 @@ For the reverse proxy you need to choose the configuration (variable `FEASIBILIT - [./subdomains.nginx.conf](./rev-proxy/subdomains.nginx.conf) with separate domains for the services (fhir-server (incl) and optionally flare and keycloak) - All subdomains must point to the host machine the triangle will run on. - - Set the service hostnames (`FLARE_HOSTNAME`, `FHIR_HOSTNAME` and `KEYCLOAK_HOSTNAME`, depending on which services you need) in [rev-proxy/.env](./rev-proxy/.env). + - Set the service hostnames (`FLARE_HOSTNAME`, `FHIR_HOSTNAME`, `TORCH_HOSTNAME` and `KEYCLOAK_HOSTNAME`, depending on which services you need) in [rev-proxy/.env](./rev-proxy/.env). - You can change the default external port the reverse proxy listens on in [rev-proxy/.env](./rev-proxy/.env) (variable `FEASIBILITY_TRIANGLE_REV_PROXY_PORT`). Any value other than `443` needs to be added to all external url's in the `.env` files and to the url's used for accessing the triangle from outside. The default value is `444` to avoid a conflict with the proxy of the local feasibility portal when it is deployed on the same host. @@ -156,7 +156,7 @@ to `false` in `/opt/feasibility-deploy/feasibility-triangle/fhir-server/.env`. To start the triangle execute `/opt/feasibility-deploy/feasibility-triangle/start-triangle.sh`. This starts the following default triangle: -FLARE (FHIR Search executor) - BLAZE (FHIR Server) - Keycloak (optional) +FLARE (FHIR Search executor) - TORCH (FHIR Data Extractor) - BLAZE (FHIR Server) - Keycloak (optional) If you would like to pick other component combinations you can start each component individually by setting your compose project (`export FEASIBILITY_COMPOSE_PROJECT=feasibility-deploy`) navigating to the respective components folder and executing: @@ -186,6 +186,7 @@ These are the URLs for access to the webclients via nginx: | Component | URL | User | Password | |-------------|------------------------------------------------------------------|------------------|------------------| | Flare | `https://your-flare-subdomain.your-domain:configured-port/` | chosen in step 3 | chosen in step 3 | +| TORCH | `https://your-torch-subdomain.your-domain:configured-port/` | chosen in step 3 | chosen in step 3 | | FHIR Server | `https://your-fhir-subdomain.your-domain:configured-port/fhir` | chosen in step 3 | chosen in step 3 | > [!NOTE] @@ -201,7 +202,8 @@ Accessible service via localhost: | Component | URL | Authentication Type | Notes | |-------------|----------------------------------|---------------------|----------------------| | Flare | | None required | | -| FHIR Server | | Bearer Token | Configured in step 8 | +| TORCH | | None required | | +| FHIR Server | | Bearer Token | Configured in step 8 | Please be aware that you will need to set up an ssh tunnel to your server and forward the respective ports if you would like to access the services on localhost without a password. @@ -322,6 +324,15 @@ If new search parameters have been added follow the "fhir-server/README.md -> Re | FHIR_SERVER_HOSTNAME | change the default value of the domain names where the services are reachable | http://fhir-server:8080 | | REV-PROXY| | KEYCLOAK_HOSTNAME | change the default value of the domain names where the services are reachable | https://keycloak.localhost:444/realms/blaze | | REV-PROXY | | FLARE_HOSTNAME |change the default value of the domain names where the services are reachable | http://fhir-server:8080/fhir | | REV-PROXY | +| TORCH_FHIR_URL | The base URL of the FHIR server which contains the patient data Torch is used to extract. | http://fhir-server:8080/fhir | | TORCH | +| TORCH_FLARE_URL | The base URL of the FLARE component if used. | http://flare:8080 | | TORCH | +| TORCH_RESULTS_PERSISTENCE | | PT12H30M5S | | TORCH | +| TORCH_LOG_LEVEL | Log level | debug | | TORCH | +| TORCH_NGINX_FILELOCATION | The base URL of the NGINX, which is used to serve the extracted patient data. | https://torch.localhost:444/fileserver | | TORCH | +| TORCH_BATCHSIZE | The number of patients Torch processes in one batch. | 100 | | TORCH | +| TORCH_MAXCONCURRENCY | Maximum number of parallel threads Torch uses to process the data extractions. | 4 | | TORCH | +| TORCH_USE_CQL | Whether or not to use CQL - if false FLARE is used. | true | | TORCH | +| TORCH_ENABLED | Whether or not Torch should be started. | true | | TORCH | ### Support for self-singed certificates diff --git a/feasibility-triangle/rev-proxy/.env.default b/feasibility-triangle/rev-proxy/.env.default index 32de70ff..8f20a522 100644 --- a/feasibility-triangle/rev-proxy/.env.default +++ b/feasibility-triangle/rev-proxy/.env.default @@ -3,6 +3,7 @@ FHIR_SERVER_HOSTNAME=fhir.example.org KEYCLOAK_HOSTNAME=auth.example.org FLARE_HOSTNAME=flare.example.org +TORCH_HOSTNAME=torch.localhost FEASIBILITY_TRIANGLE_REV_PROXY_NGINX_CONFIG=./context-paths.nginx.conf FEASIBILITY_TRIANGLE_REV_PROXY_PORT=444 diff --git a/feasibility-triangle/torch/.env.default b/feasibility-triangle/torch/.env.default index c3ec73f8..71525b2a 100644 --- a/feasibility-triangle/torch/.env.default +++ b/feasibility-triangle/torch/.env.default @@ -1,13 +1,7 @@ -TORCH_SERVER_PORT=8080 -TORCH_PROFILE_DIR=/app/structureDefinitions -TORCH_MAPPING_CONSENT=/app/Mappings/consent-mappings_fhir.json -TORCH_MAPPING_CONSENT_TO_PROFILE=/app/Mappings/profile_to_consent.json TORCH_FHIR_URL=http://torch-data-store:8080/fhir TORCH_FLARE_URL=http://torch-flare:8080 -TORCH_RESULTS_DIR=/app/output TORCH_RESULTS_PERSISTENCE=PT12H30M5S TORCH_LOG_LEVEL=debug -TORCH_NGINX_SERVERNAME=localhost TORCH_NGINX_FILELOCATION=http://localhost:80 TORCH_BATCHSIZE=100 TORCH_MAXCONCURRENCY=4 diff --git a/feasibility-triangle/torch/docker-compose.yml b/feasibility-triangle/torch/docker-compose.yml index 521103a0..915b9a53 100644 --- a/feasibility-triangle/torch/docker-compose.yml +++ b/feasibility-triangle/torch/docker-compose.yml @@ -13,16 +13,10 @@ services: ports: - ${TORCH_PORT:-127.0.0.1:8086}:8080 environment: - SERVER_PORT: ${TORCH_SERVER_PORT:-8080} - TORCH_PROFILE_DIR: ${TORCH_PROFILE_DIR:-/app/structureDefinitions} - TORCH_MAPPING_CONSENT: ${TORCH_MAPPING_CONSENT:-/app/Mappings/consent-mappings_fhir.json} - TORCH_MAPPING_CONSENT_TO_PROFILE: ${TORCH_MAPPING_CONSENT_TO_PROFILE:-/app/Mappings/profile_to_consent.json} TORCH_FHIR_URL: ${TORCH_FHIR_URL:-http://torch-data-store:8080/fhir} TORCH_FLARE_URL: ${TORCH_FLARE_URL:-http://torch-flare:8080} - TORCH_RESULTS_DIR: ${TORCH_RESULTS_DIR:-/app/output} TORCH_RESULTS_PERSISTENCE: ${TORCH_RESULTS_PERSISTENCE:-PT12H30M5S} LOG_LEVEL: ${TORCH_LOG_LEVEL:-debug} - NGINX_SERVERNAME: ${TORCH_NGINX_SERVERNAME:-localhost} NGINX_FILELOCATION: ${TORCH_NGINX_FILELOCATION:-http://localhost:80} TORCH_BATCHSIZE: ${TORCH_BATCHSIZE:-100} TORCH_MAXCONCURRENCY: ${TORCH_MAXCONCURRENCY:-4} From 4ef4e5590c709c6e87a7af51edc41883c8c864fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 14 Oct 2024 22:31:46 +0200 Subject: [PATCH 11/15] fix elastic-init lint --- feasibility-portal/backend/elastic-init.sh | 49 +++++++++---------- feasibility-portal/gui/docker-compose.yml | 2 - feasibility-triangle/torch/docker-compose.yml | 3 ++ 3 files changed, 27 insertions(+), 27 deletions(-) diff --git a/feasibility-portal/backend/elastic-init.sh b/feasibility-portal/backend/elastic-init.sh index 491dc65f..8bdff646 100755 --- a/feasibility-portal/backend/elastic-init.sh +++ b/feasibility-portal/backend/elastic-init.sh @@ -1,26 +1,25 @@ #!/bin/sh -# Wait for Elasticsearch to start up before doing anything until curl -X GET "$ELASTIC_HOST/_cluster/health" | grep -q '"status":"green"\|"status":"yellow"'; do echo "Waiting for Elasticsearch..." sleep 5 done -ABSOLUTE_FILEPATH="${ELASTIC_FILEPATH//TAGPLACEHOLDER/$ELASTIC_GIT_TAG}$ELASTIC_FILENAME" +ABSOLUTE_FILEPATH=$(echo "$ELASTIC_FILEPATH" | sed "s/TAGPLACEHOLDER/$ELASTIC_GIT_TAG/")"$ELASTIC_FILENAME" echo "Downloading $ABSOLUTE_FILEPATH" response_onto_dl=$(curl --write-out "%{http_code}" -sLO "$ABSOLUTE_FILEPATH") if [ "$response_onto_dl" -ne 200 ]; then - echo "Could not download ontology file. Maybe the tag $ELASTIC_GIT_TAG does not exist? Error code was $response_onto_dl" - exit 1 + echo "Could not download ontology file. Maybe the tag $ELASTIC_GIT_TAG does not exist? Error code was $response_onto_dl" + exit 1 fi unzip -o "$ELASTIC_FILENAME" if [ "$OVERRIDE_EXISTING" = "true" ]; then - echo "(Trying to) delete existing indices" - curl -s -DELETE "$ELASTIC_HOST/ontology" - curl -s -DELETE "$ELASTIC_HOST/codeable_concept" + echo "(Trying to) delete existing indices" + curl -s -DELETE "$ELASTIC_HOST/ontology" + curl -s -DELETE "$ELASTIC_HOST/codeable_concept" fi echo "Creating ontology index..." @@ -30,25 +29,25 @@ response_cc=$(curl --write-out "%{http_code}" -s --output /dev/null -XPUT -H 'Co echo "Done" for FILE in elastic/*; do - if [ -f "$FILE" ]; then - BASENAME=$(basename "$FILE") - if [[ $BASENAME == onto_es__ontology* && $BASENAME == *.json ]]; then - if [[ "$response_onto" -eq 200 || "$OVERRIDE_EXISTING" = "true" ]]; then - echo "Uploading $BASENAME" - curl -s --output /dev/null -XPOST -H 'Content-Type: application/json' --data-binary @"$FILE" "$ELASTIC_HOST/ontology/_bulk?pretty" - else - echo "Skipping $BASENAME because index was already existing. Set OVERRIDE_EXISTING to true to force creating a new index" - fi + if [ -f "$FILE" ]; then + BASENAME=$(basename "$FILE") + if echo "$BASENAME" | grep -q "^onto_es__ontology.*\.json$"; then + if [ "$response_onto" -eq 200 ] || [ "$OVERRIDE_EXISTING" = "true" ]; then + echo "Uploading $BASENAME" + curl -s --output /dev/null -XPOST -H 'Content-Type: application/json' --data-binary @"$FILE" "$ELASTIC_HOST/ontology/_bulk?pretty" + else + echo "Skipping $BASENAME because index was already existing. Set OVERRIDE_EXISTING to true to force creating a new index" + fi + fi + if echo "$BASENAME" | grep -q "^onto_es__codeable_concept.*\.json$"; then + if [ "$response_cc" -eq 200 ] || [ "$OVERRIDE_EXISTING" = "true" ]; then + echo "Uploading $BASENAME" + curl -s --output /dev/null -XPOST -H 'Content-Type: application/json' --data-binary @"$FILE" "$ELASTIC_HOST/codeable_concept/_bulk?pretty" + else + echo "Skipping $BASENAME because index was already existing. Set OVERRIDE_EXISTING to true to force creating a new index" + fi + fi fi - if [[ $BASENAME == onto_es__codeable_concept* && $BASENAME == *.json ]]; then - if [[ "$response_cc" -eq 200 || "$OVERRIDE_EXISTING" = "true" ]]; then - echo "Uploading $BASENAME" - curl -s --output /dev/null -XPOST -H 'Content-Type: application/json' --data-binary @"$FILE" "$ELASTIC_HOST/codeable_concept/_bulk?pretty" - else - echo "Skipping $BASENAME because index was already existing. Set OVERRIDE_EXISTING to true to force creating a new index" - fi - fi - fi done echo "All done" diff --git a/feasibility-portal/gui/docker-compose.yml b/feasibility-portal/gui/docker-compose.yml index 3d534d34..6d630e6f 100644 --- a/feasibility-portal/gui/docker-compose.yml +++ b/feasibility-portal/gui/docker-compose.yml @@ -2,7 +2,5 @@ services: dataportal-ui: image: ghcr.io/medizininformatik-initiative/feasibility-gui:6.0.0-test.4 restart: unless-stopped - ports: - - "127.0.0.1:8080:8080" volumes: - ./deploy-config.json:/usr/share/nginx/html/assets/config/config.deploy.json diff --git a/feasibility-triangle/torch/docker-compose.yml b/feasibility-triangle/torch/docker-compose.yml index 915b9a53..9530db6d 100644 --- a/feasibility-triangle/torch/docker-compose.yml +++ b/feasibility-triangle/torch/docker-compose.yml @@ -15,6 +15,9 @@ services: environment: TORCH_FHIR_URL: ${TORCH_FHIR_URL:-http://torch-data-store:8080/fhir} TORCH_FLARE_URL: ${TORCH_FLARE_URL:-http://torch-flare:8080} + TORCH_MAPPING_CONSENT: /app/Mappings/consent-mappings_fhir.json + TORCH_MAPPING_CONSENT_TO_PROFILE: /app/Mappings/profile_to_consent.json + TORCH_PROFILE_DIR: /app/structureDefinitions TORCH_RESULTS_PERSISTENCE: ${TORCH_RESULTS_PERSISTENCE:-PT12H30M5S} LOG_LEVEL: ${TORCH_LOG_LEVEL:-debug} NGINX_FILELOCATION: ${TORCH_NGINX_FILELOCATION:-http://localhost:80} From 792d17da9a77ec5d1a9f9ed3177b367ee1401dc3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Sat, 19 Oct 2024 15:54:15 +0200 Subject: [PATCH 12/15] update torch to new version, add oauth env vars --- feasibility-triangle/.DS_Store | Bin 6148 -> 10244 bytes feasibility-triangle/README.md | 7 +++++-- feasibility-triangle/torch/.env.default | 7 ++++++- feasibility-triangle/torch/docker-compose.yml | 13 +++++++++---- 4 files changed, 20 insertions(+), 7 deletions(-) diff --git a/feasibility-triangle/.DS_Store b/feasibility-triangle/.DS_Store index fc219757637bed579085dde8d6aeffe5cb88aac4..ffe9c66c59bfa680879c139db1419bef69aa2525 100644 GIT binary patch literal 10244 zcmeHMYitx%6h3F#!Ym_n3KSeHuu|0WD71x!m*Dml9tDbh(DG{8-5KbF=}g_3-SQBv z{_uyOKYd0)e}GZLA3;TM!q3F41mqa6-} z2!sfP2!sfP2!sgS2?)?Pn>ASxH0(nJLIgqt1`uHHhbUc)#(df@D1UWO$F~4P^J$nj zx>wl^(Wp;jKJ6EjB-But5*4QCZ!u7q(|FX^i}|!)P+<<}Z$8jp8T}m!^3`enC|3uJ z2^#hx0wDs|BEa_U`4ERRc+ef(zbASv)AcGU&Y+Z+Uoc{%JW^KVEiK(nUyB#_(r#@$ zwTW@9j$U#q4NhRb|!Qy%D*%bncSot{qm7?d%gVp#L6#(L-X|9}Gx$ zlU9NpWI6*;TH7mIl1@*rFr2q0N9Jbg#0f^;qAUXEvIs;*DavTAZGC&khK*gSns$w5 z!-{)m#;A{bhTEkUx&+9|q+_p385shiJ?U8K9-AaZ$+(qFld@4`SuN&np`A=QmbK1t zP0w`f4IQ4b(_=C=QI4o%qwA!+IzCy4N%IS>v22Z|US{mBH&QOqNaV0s*GyPOs~zt#B1Jem z!maMsCTSh%g!xF?^mfT=smwQ`cX1&#B^MG|J#pqrkP z@*Q4Hacp1{*62EqVys$JsA+oT0!B*UY2~47a+t9%@W^gC&!YA=*+=tKLv-mt7#Mkh3 zd<#Fo5Ah@X7{A1mcnZJ6Gx!r;!prz8UX?~jBPCfXlEzBoq++Q|njuw5OQd>frPM6^ zm$DVOX&@ecBPEFp7Mq*#j|8P7!xvP^@)avrHLkv63+n6`UP9xGCrpg8x>~rjW*`Mc z&SujznEX8BbLxa=JkN(A*?O7^?Vb{4%G7Kf4z15NDivd=%qmkI>1&rxm#I|L0-r6L zL2RQsLa-&}b0}AjsS;JQ$%eRP%jyN{m?GOi4+715I;6 zm2Xvse+?($2RIMEP=^1Fc_`yVj8axt;X+)74^dv%VFRwkCd%w)p4Xk&g>lMl3wzMU zUQFR0d=&TM;bD9O-^6$DC?3Q2@Hl1sXZQ_e`Z>HXB#YT+ zAkgqPVg|Cf$EmOP54Pp@ant5p?)su8SFYbs-rF7g!Gsi)g9`@H4C{km ztm~-ce7`XH`}HRay8Zo2uU_2U#@)Woygklf5I5J(2?-CfkP0Xyi+)R_z{v3tMzMbu z0@9#g>yIC>D@(awIU0k$46l*1KD|S!McB#&V>!J_Q{cc(*cF6x7;&zfOAie{NzmPM zS1&F;+{xV?X1*YVH`dH`|2%VFrk`L2<$?z~f#^Xz=$-uj6?RN>jZVH4Q1{66N7^$W_)Nr#&g6{q>P{Lg^@a-bIO P|HJ)%aBJ{C+5i6oREwIX delta 213 zcmZn(XfcprU|?W$DortDU=RQ@Ie-{MGjUEV6q~50D9Qwq2aD-46f>kU6fu-Bq;4!+ z&N$gXg=KP!xHwNzd2vBfPJR+l;f~2F0)^@l)zyZ^raB5H#)h>z3e}cI1|YUsZ7nB< zsItCwP<(byZeD)ZBgn}hH?e?dp!+~>28(Tu=b6I{096kvk^lez diff --git a/feasibility-triangle/README.md b/feasibility-triangle/README.md index 0b86ccca..de1cc2cc 100644 --- a/feasibility-triangle/README.md +++ b/feasibility-triangle/README.md @@ -10,8 +10,9 @@ The Feasibility Triangle is composed of four components: 1. A Middleware Client (DSF) 2. A Feasibility Analysis Request Executor (FLARE) -3. A FHIR Server (Blaze) -4. Reverse Proxy (NGINX) +3. A Dataselection and Extraction Executor (TORCH) +4. A FHIR Server (Blaze) +5. Reverse Proxy (NGINX) The reverse proxy allows for integration into a site's multi-server infrastructure. It also provides basic auth capability for FHIR server and FLARE components. @@ -61,6 +62,7 @@ Running this setup safely at your site requires a valid certificate and domains. - FHIR server - FLARE +- TORCH - Keycloak (optional, see step 8) You will require two .pem files: a `cert.pem` (certificate) and `cert.key` (private key). @@ -140,6 +142,7 @@ For the reverse proxy you need to choose the configuration (variable `FEASIBILIT The triangle is configured by default to start the following services: - FLARE: A Rest Service, which is needed to translate, execute and evaluate a feasibility query on a FHIR Server using FHIR Search +- TORCH: A Rest Service, which is needed to execute Dataselection and Extraction queries (CRTDLs - clinical-resource-transfer-definition-languge) - BLAZE: The FHIR Server which holds the patient data for feasibility queries - Keycloak (optional): OpenID Connect provider for authorization used by BLAZE component - We recommend using your own keyloak and configuring a blaze realm there diff --git a/feasibility-triangle/torch/.env.default b/feasibility-triangle/torch/.env.default index 71525b2a..2fb1f8fa 100644 --- a/feasibility-triangle/torch/.env.default +++ b/feasibility-triangle/torch/.env.default @@ -6,4 +6,9 @@ TORCH_NGINX_FILELOCATION=http://localhost:80 TORCH_BATCHSIZE=100 TORCH_MAXCONCURRENCY=4 TORCH_USE_CQL=true -TORCH_ENABLED=true \ No newline at end of file +TORCH_ENABLED=true +TORCH_FHIR_USER= +TORCH_FHIR_PASSWORD= +TORCH_FHIR_OAUTH_ISSUER_URI=https://auth.localhost:444/realms/blaze +TORCH_FHIR_OAUTH_CLIENT_ID=account +TORCH_FHIR_OAUTH_CLIENT_SECRET=insecure \ No newline at end of file diff --git a/feasibility-triangle/torch/docker-compose.yml b/feasibility-triangle/torch/docker-compose.yml index 9530db6d..f6dda5a8 100644 --- a/feasibility-triangle/torch/docker-compose.yml +++ b/feasibility-triangle/torch/docker-compose.yml @@ -15,15 +15,20 @@ services: environment: TORCH_FHIR_URL: ${TORCH_FHIR_URL:-http://torch-data-store:8080/fhir} TORCH_FLARE_URL: ${TORCH_FLARE_URL:-http://torch-flare:8080} - TORCH_MAPPING_CONSENT: /app/Mappings/consent-mappings_fhir.json - TORCH_MAPPING_CONSENT_TO_PROFILE: /app/Mappings/profile_to_consent.json + TORCH_MAPPING_CONSENT: /app/mappings/consent-mappings_fhir.json + TORCH_MAPPING_CONSENT_TO_PROFILE: /app/mappings/profile_to_consent.json TORCH_PROFILE_DIR: /app/structureDefinitions TORCH_RESULTS_PERSISTENCE: ${TORCH_RESULTS_PERSISTENCE:-PT12H30M5S} - LOG_LEVEL: ${TORCH_LOG_LEVEL:-debug} + LOG_LEVEL: ${TORCH_LOG_LEVEL:-INFO} NGINX_FILELOCATION: ${TORCH_NGINX_FILELOCATION:-http://localhost:80} TORCH_BATCHSIZE: ${TORCH_BATCHSIZE:-100} TORCH_MAXCONCURRENCY: ${TORCH_MAXCONCURRENCY:-4} TORCH_USE_CQL: ${TORCH_USE_CQL:-true} + TORCH_FHIR_USER: ${TORCH_FHIR_USER:-""} + TORCH_FHIR_PASSWORD: ${TORCH_FHIR_PASSWORD:-""} + TORCH_FHIR_OAUTH_ISSUER_URI: ${TORCH_FHIR_OAUTH_ISSUER_URI:-https://auth.localhost:444/realms/blaze} + TORCH_FHIR_OAUTH_CLIENT_ID: ${TORCH_FHIR_OAUTH_CLIENT_ID:-account} + TORCH_FHIR_OAUTH_CLIENT_SECRET: ${TORCH_FHIR_OAUTH_CLIENT_SECRET:-insecure} volumes: - "triangle-torch-data-store:/app/output" - ../auth:/app/certs @@ -31,4 +36,4 @@ services: - "auth.localhost:host-gateway" volumes: - triangle-torch-data-store: \ No newline at end of file + triangle-torch-data-store: From 06f38143b29c508282e87fbf0f61173ee5caf99f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 21 Oct 2024 01:42:33 +0200 Subject: [PATCH 13/15] bump versions of components, add triangle auth as extra host to portal, remove old mapping paths --- feasibility-portal/backend/docker-compose.yml | 8 ++++---- feasibility-portal/gui/docker-compose.yml | 2 +- feasibility-triangle/flare/docker-compose.yml | 2 +- feasibility-triangle/torch/docker-compose.yml | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/feasibility-portal/backend/docker-compose.yml b/feasibility-portal/backend/docker-compose.yml index f5c34cad..631cdcdc 100644 --- a/feasibility-portal/backend/docker-compose.yml +++ b/feasibility-portal/backend/docker-compose.yml @@ -1,7 +1,7 @@ services: dataportal-backend: restart: unless-stopped - image: ghcr.io/medizininformatik-initiative/feasibility-backend:6.0.0-test.1 + image: ghcr.io/medizininformatik-initiative/feasibility-backend:6.0.0-alpha.3 ports: - ${DATAPORTAL_BACKEND_PORT:-127.0.0.1:8091}:8090 depends_on: @@ -30,8 +30,6 @@ services: # ---- ontology ONTOLOGY_FILES_FOLDER_UI: ${DATAPORTAL_BACKEND_ONTOLOGY_FILES_FOLDER:-/opt/dataportal-backend/ontology} ONTOLOGY_DB_MIGRATION_FOLDER: ${DATAPORTAL_BACKEND_ONTOLOGY_DB_MIGRATION_FOLDER:-/opt/dataportal-backend/ontology/migration} - MAPPINGS_FILE: ${DATAPORTAL_BACKEND_ONTOLOGY_FILES_FOLDER:-/opt/dataportal-backend/ontology}/codex-term-code-mapping.json - CONCEPT_TREE_FILE: ${DATAPORTAL_BACKEND_ONTOLOGY_FILES_FOLDER:-/opt/dataportal-backend/ontology}/codex-code-tree.json # ---- auth KEYCLOAK_ENABLED: ${DATAPORTAL_BACKEND_KEYCLOAK_ENABLED:-true} KEYCLOAK_BASE_URL: ${DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL:-http://keycloak:8080} @@ -43,7 +41,7 @@ services: KEYCLOAK_BASE_URL_JWK: ${DATAPORTAL_BACKEND_KEYCLOAK_BASE_URL_JWK:-http://auth:8080} KEYCLOAK_REALM: ${DATAPORTAL_BACKEND_KEYCLOAK_REALM:-dataportal} #---- Mock broker - BROKER_CLIENT_MOCK_ENABLED: ${DATAPORTAL_BACKEND_BROKER_CLIENT_MOCK_ENABLED:-true} + BROKER_CLIENT_MOCK_ENABLED: ${DATAPORTAL_BACKEND_BROKER_CLIENT_MOCK_ENABLED:-false} #---- Direct broker BROKER_CLIENT_DIRECT_ENABLED: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_ENABLED:-false} BROKER_CLIENT_DIRECT_USE_CQL: ${DATAPORTAL_BACKEND_BROKER_CLIENT_DIRECT_USE_CQL:-false} @@ -90,6 +88,8 @@ services: # ---- logging LOG_LEVEL_SQL: ${DATAPORTAL_BACKEND_LOG_LEVEL_SQL:-warn} LOG_LEVEL: ${DATAPORTAL_BACKEND_LOG_LEVEL:-warn} + extra_hosts: + - "auth.localhost:host-gateway" volumes: - ${DATAPORTAL_BACKEND_CERTS_PATH:-../secrets}:/opt/dataportal-security - ./certs:/opt/dataportal-backend/certs diff --git a/feasibility-portal/gui/docker-compose.yml b/feasibility-portal/gui/docker-compose.yml index 6d630e6f..73ad4fd0 100644 --- a/feasibility-portal/gui/docker-compose.yml +++ b/feasibility-portal/gui/docker-compose.yml @@ -1,6 +1,6 @@ services: dataportal-ui: - image: ghcr.io/medizininformatik-initiative/feasibility-gui:6.0.0-test.4 + image: ghcr.io/medizininformatik-initiative/feasibility-gui:6.0.0-alpha.4 restart: unless-stopped volumes: - ./deploy-config.json:/usr/share/nginx/html/assets/config/config.deploy.json diff --git a/feasibility-triangle/flare/docker-compose.yml b/feasibility-triangle/flare/docker-compose.yml index f874107b..cdfc31a5 100644 --- a/feasibility-triangle/flare/docker-compose.yml +++ b/feasibility-triangle/flare/docker-compose.yml @@ -1,6 +1,6 @@ services: flare: - image: ghcr.io/medizininformatik-initiative/flare:2.3.0 + image: ghcr.io/medizininformatik-initiative/flare:2.4.0-alpha.2 ports: - ${FEASIBILITY_FLARE_PORT:-127.0.0.1:8084}:8080 environment: diff --git a/feasibility-triangle/torch/docker-compose.yml b/feasibility-triangle/torch/docker-compose.yml index f6dda5a8..61eee20a 100644 --- a/feasibility-triangle/torch/docker-compose.yml +++ b/feasibility-triangle/torch/docker-compose.yml @@ -9,7 +9,7 @@ services: - triangle-torch-data-store:/app/output torch: restart: unless-stopped - image: ghcr.io/medizininformatik-initiative/torch:develop + image: ghcr.io/medizininformatik-initiative/torch:1.0.0-alpha.1 ports: - ${TORCH_PORT:-127.0.0.1:8086}:8080 environment: From 3db4f71474194fc7a74294480d8666126bb31a5c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 21 Oct 2024 09:42:39 +0200 Subject: [PATCH 14/15] Release v5.0.0-alpha --- CHANGELOG.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb3620dc..a9e87a92 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,27 @@ Please refer to the respective repositories for a more in depth changelog of sin |Backend|| |DSF Feasibility Plugin|| |FLARE|| -|Blaze FHIR server|| +|TORCH|| + + +## [5.0.0-alpha] - 2024-10-21 + +### Features + +| Feature | Affected Components | +| -- | -- | +|UI Re-Desig, Restructuring of Code|UI, Backend| +|Extended Criteria Search (Elastic Search)|UI, Backend, Ontology Generation| +|Add OAuth2 to triangle components|TORCH, FLARE| +|Added Dataselection and Extraction |UI, Backend, Ontology Generation, TORCH| +|Migrated from Mapping code system tree strcture to poly tree structure to support non strict hierarchical code systems like sct |UI, Backend, Ontology Generation, TORCH, FLARE| +|Loading and displaying of criteria availability |UI, Backend, Ontology Generation| + +### Overall + +- Updated all components to new versions +- Added TORCH component for data selection and extraction in the triangle ## [4.1.0] - 2024-07-16 From 3036a6e95f636329a6014ecbf95bca777f4ac727 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20Gr=C3=BCndner?= Date: Mon, 21 Oct 2024 09:53:10 +0200 Subject: [PATCH 15/15] increment version in readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 637626cd..20efe81d 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Feasibility Deploy -[![version](https://img.shields.io/badge/version-4.0.0-green.svg)](https://github.com/medizininformatik-initiative/feasibility-deploy/releases) +[![version](https://img.shields.io/badge/version-5.0.0-green.svg)](https://github.com/medizininformatik-initiative/feasibility-deploy/releases) This feasibility deployment repository offers an example deployment repository using docker-compose and official images to set up a feasibility portal (central) as well as feasibility triangle (decentral - at site)