Dynamically create the service_conf.yaml file by replacing environmen…

…t variables from .env (infiniflow#3341) ### What problem does this PR solve? This pull request implements the feature mentioned in infiniflow#3322. Instead of manually having to edit the `service_conf.yaml` file when changes have been made to `.env` and mapping it into the docker container at runtime, a template file is used and the values replaced by the environment variables from the `.env` file when the container is started. ### Type of change - [X] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Kevin Hu <[email protected]> Co-authored-by: Zhichang Yu <[email protected]>
stackhpc · Nov 12, 2024 · 0c95a33 · 0c95a33
1 parent 7274420
commit 0c95a33
Show file tree

Hide file tree

Showing 12 changed files with 122 additions and 35 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -129,6 +129,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"
 
 ENV PYTHONPATH=/ragflow/
 
+COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
 COPY docker/entrypoint.sh ./entrypoint.sh
 RUN chmod +x ./entrypoint.sh
 

diff --git a/Dockerfile.scratch.oc9 b/Dockerfile.scratch.oc9
@@ -53,6 +53,7 @@ RUN conda run -n py11 python -m nltk.downloader wordnet
 ENV PYTHONPATH=/ragflow/
 ENV HF_ENDPOINT=https://hf-mirror.com
 
+COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
 ADD docker/entrypoint.sh ./entrypoint.sh
 RUN chmod +x ./entrypoint.sh
 

diff --git a/Dockerfile.slim b/Dockerfile.slim
@@ -122,6 +122,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"
 
 ENV PYTHONPATH=/ragflow/
 
+COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
 COPY docker/entrypoint.sh ./entrypoint.sh
 RUN chmod +x ./entrypoint.sh
 

diff --git a/README.md b/README.md
@@ -216,7 +216,7 @@ releases! 🌟
 5. In your web browser, enter the IP address of your server and log in to RAGFlow.
    > With the default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default
    HTTP serving port `80` can be omitted when using the default configurations.
-6. In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM factory in `user_default_llm` and update
+6. In [service_conf.yaml.template](./docker/service_conf.yaml.template), select the desired LLM factory in `user_default_llm` and update
    the `API_KEY` field with the corresponding API key.
 
    > See [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) for more information.
@@ -229,16 +229,11 @@ When it comes to system configurations, you will need to manage the following fi
 
 - [.env](./docker/.env): Keeps the fundamental setups for the system, such as `SVR_HTTP_PORT`, `MYSQL_PASSWORD`, and
   `MINIO_PASSWORD`.
-- [service_conf.yaml](./docker/service_conf.yaml): Configures the back-end services.
-- [docker-compose.yml](./docker/docker-compose.yml): The system relies
-  on [docker-compose.yml](./docker/docker-compose.yml) to start up.
-
-You must ensure that changes to the [.env](./docker/.env) file are in line with what are in the [service_conf.yaml](./docker/service_conf.yaml) file.
+- [service_conf.yaml.template](./docker/service_conf.yaml.template): Configures the back-end services. The environment variables in this file will be automatically populated when the Docker container starts. Any environment variables set within the Docker container will be available for use, allowing you to customize service behavior based on the deployment environment.
+- [docker-compose.yml](./docker/docker-compose.yml): The system relies on [docker-compose.yml](./docker/docker-compose.yml) to start up.
 
 > The [./docker/README](./docker/README.md) file provides a detailed description of the environment settings and service
-> configurations, and you are REQUIRED to ensure that all environment settings listed in
-> the [./docker/README](./docker/README.md) file are aligned with the corresponding configurations in
-> the [service_conf.yaml](./docker/service_conf.yaml) file.
+> configurations which can be used as `${ENV_VARS}` in the [service_conf.yaml.template](./docker/service_conf.yaml.template) file.
 
 To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80`
 to `<YOUR_SERVING_PORT>:80`.
@@ -293,11 +288,11 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
    docker compose -f docker/docker-compose-base.yml up -d
    ```
 
-   Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`:
+   Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/.env** to `127.0.0.1`:
    ```
    127.0.0.1       es01 infinity mysql minio redis
    ```  
-   In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
+   In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
 
 4. If you cannot access HuggingFace, set the `HF_ENDPOINT` environment variable to use a mirror site:
 

diff --git a/docker/.env b/docker/.env
@@ -1,6 +1,9 @@
 # The version of Elasticsearch.
 STACK_VERSION=8.11.3
 
+# The hostname where the Elasticsearch service is exposed
+ES_HOST=es01
+
 # The port used to expose the Elasticsearch service to the host machine, 
 # allowing EXTERNAL access to the service running inside the Docker container.
 ES_PORT=1200
@@ -27,10 +30,16 @@ INFINITY_PSQL_PORT=5432
 # The password for MySQL. 
 # When updated, you must revise the `mysql.password` entry in service_conf.yaml.
 MYSQL_PASSWORD=infini_rag_flow
+# The hostname where the MySQL service is exposed
+MYSQL_HOST=mysql
+# The database of the MySQL service to use
+MYSQL_DBNAME=rag_flow
 # The port used to expose the MySQL service to the host machine, 
 # allowing EXTERNAL access to the MySQL database running inside the Docker container. 
 MYSQL_PORT=5455
 
+# The hostname where the MySQL service is exposed
+MINIO_HOST=minio
 # The port used to expose the MinIO console interface to the host machine, 
 # allowing EXTERNAL access to the web-based console running inside the Docker container. 
 MINIO_CONSOLE_PORT=9001
@@ -44,6 +53,8 @@ MINIO_USER=rag_flow
 # When updated, you must revise the `minio.password` entry in service_conf.yaml accordingly.
 MINIO_PASSWORD=infini_rag_flow
 
+# The hostname where the Redis service is exposed
+REDIS_HOST=redis
 # The port used to expose the Redis service to the host machine, 
 # allowing EXTERNAL access to the Redis service running inside the Docker container.
 REDIS_PORT=6379

diff --git a/docker/README.md b/docker/README.md
@@ -27,7 +27,7 @@ The [.env](./.env) file contains important environment variables for Docker.
 - `ES_PORT`  
   The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container.  Defaults to `1200`.
 - `ELASTIC_PASSWORD`  
-  The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for Elasticsearch. 
 
 ### Kibana
 
@@ -46,7 +46,7 @@ The [.env](./.env) file contains important environment variables for Docker.
 ### MySQL
 
 - `MYSQL_PASSWORD`  
-  The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for MySQL. 
 - `MYSQL_PORT`  
   The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
 
@@ -57,16 +57,16 @@ The [.env](./.env) file contains important environment variables for Docker.
 - `MINIO_PORT`  
   The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
 - `MINIO_USER`  
-  The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The username for MinIO.
 - `MINIO_PASSWORD`  
-  The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for MinIO. 
 
 ### Redis
 
 - `REDIS_PORT`  
   The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
 - `REDIS_PASSWORD`  
-  The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for Redis.
 
 ### RAGFlow
 
@@ -119,7 +119,7 @@ The [.env](./.env) file contains important environment variables for Docker.
 
 ## 🐋 Service configuration
 
-[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
+[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. In a dockerized setup, this file is automatically created based on the [service_conf.yaml.template](./service_conf.yaml.template) file (replacing all environment variables by their values).
 
 - `ragflow`
   - `host`: The API server's IP address inside the Docker container. Defaults to `0.0.0.0`.
@@ -139,11 +139,11 @@ The [.env](./.env) file contains important environment variables for Docker.
   - `host`: The MinIO serving IP *and* port inside the Docker container. Defaults to `minio:9000`.
 
 - `oauth`  
-  The OAuth configuration for signing up or signing in to RAGFlow using a third-party account.  It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
+  The OAuth configuration for signing up or signing in to RAGFlow using a third-party account.  It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
   - `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings page](https://github.com/settings/developers) to obtain your client_id and secret_key.
 
 - `user_default_llm`  
-  The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.  
+  The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.  
   - `factory`: The LLM supplier. Available options:
     - `"OpenAI"`
     - `"DeepSeek"`

diff --git a/docker/docker-compose-gpu.yml b/docker/docker-compose-gpu.yml
@@ -16,7 +16,6 @@ services:
       - 80:80
       - 443:443
     volumes:
-      - ./service_conf.yaml:/ragflow/conf/service_conf.yaml
       - ./ragflow-logs:/ragflow/logs
       - ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
       - ./nginx/proxy.conf:/etc/nginx/proxy.conf

diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
@@ -1,5 +1,12 @@
 #!/bin/bash
 
+# replace env variables in the service_conf.yaml file
+rm -rf /ragflow/conf/service_conf.yaml
+while IFS= read -r line || [[ -n "$line" ]]; do
+    # Use eval to interpret the variable with default values
+    eval "echo \"$line\"" >> /ragflow/conf/service_conf.yaml
+done < /ragflow/conf/service_conf.yaml.template
+
 # unset http proxy which maybe set by docker daemon
 export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
 

diff --git a/docker/service_conf.yaml.template b/docker/service_conf.yaml.template
@@ -0,0 +1,74 @@
+ragflow:
+  host: ${RAGFLOW_HOST:-0.0.0.0}
+  http_port: 9380
+mysql:
+  name: '${MYSQL_DBNAME:-rag_flow}'
+  user: '${MYSQL_USER:-root}'
+  password: '${MYSQL_PASSWORD:-infini_rag_flow}'
+  host: '${MYSQL_HOST:-mysql}'
+  port: 3306
+  max_connections: 100
+  stale_timeout: 30
+minio:
+  user: '${MINIO_USER:-rag_flow}'
+  password: '${MINIO_PASSWORD:-infini_rag_flow}'
+  host: '${MINIO_HOST:-minio}:9000'
+es:
+  hosts: 'http://${ES_HOST:-es01}:9200'
+  username: '${ES_USER:-elastic}'
+  password: '${ES_PASSWORD:-infini_rag_flow}'
+redis:
+  db: 1
+  password: '${REDIS_PASSWORD:-infini_rag_flow}' 
+  host: '${REDIS_HOST:-redis}:6379'
+
+# postgres:
+#   name: '${POSTGRES_DBNAME:-rag_flow}'
+#   user: '${POSTGRES_USER:-rag_flow}'
+#   password: '${POSTGRES_PASSWORD:-infini_rag_flow}'
+#   host: '${POSTGRES_HOST:-postgres}'
+#   port: 5432
+#   max_connections: 100
+#   stale_timeout: 30
+# s3:
+#   endpoint: 'endpoint'
+#   access_key: 'access_key'
+#   secret_key: 'secret_key'
+#   region: 'region'
+# azure:
+#   auth_type: 'sas'
+#   container_url: 'container_url'
+#   sas_token: 'sas_token'
+# azure:
+#   auth_type: 'spn'
+#   account_url: 'account_url'
+#   client_id: 'client_id'
+#   secret: 'secret'
+#   tenant_id: 'tenant_id'
+#   container_name: 'container_name'
+# user_default_llm:
+#   factory: 'Tongyi-Qianwen'
+#   api_key: 'sk-xxxxxxxxxxxxx'
+#   base_url: ''
+# oauth:
+#   github:
+#     client_id: xxxxxxxxxxxxxxxxxxxxxxxxx
+#     secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
+#     url: https://github.com/login/oauth/access_token
+#   feishu:
+#     app_id: cli_xxxxxxxxxxxxxxxxxxx
+#     app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
+#     app_access_token_url: https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal
+#     user_access_token_url: https://open.feishu.cn/open-apis/authen/v1/oidc/access_token
+#     grant_type: 'authorization_code'
+# authentication:
+#   client:
+#     switch: false
+#     http_app_key:
+#     http_secret_key:
+#   site:
+#     switch: false
+# permission:
+#   switch: false
+#   component: false
+#   dataset: false
diff --git a/docs/configurations.md b/docs/configurations.md
@@ -12,11 +12,9 @@ Configurations for installing RAGFlow via Docker.
 When it comes to system configurations, you will need to manage the following files:
 
 - [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env): Contains important environment variables for Docker.
-- [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
+- [service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor. Upon container startup, the `service_conf.yaml` file will be generated based on this template file. This process replaces any environment variables within the template, allowing for dynamic configuration tailored to the container's environment.
 - [docker-compose.yml](https://github.com/infiniflow/ragflow/blob/main/docker/docker-compose.yml): The Docker Compose file for starting up the RAGFlow service.
 
-You must ensure that changes to the [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file are in line with what are in the [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) file.
-
 To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80`
 to `<YOUR_SERVING_PORT>:80`.
 
@@ -47,7 +45,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
 - `ES_PORT`  
   The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container.  Defaults to `1200`.
 - `ELASTIC_PASSWORD`  
-  The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
+  The password for Elasticsearch. 
 
 ### Kibana
 
@@ -66,7 +64,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
 ### MySQL
 
 - `MYSQL_PASSWORD`  
-  The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
+  The password for MySQL. 
 - `MYSQL_PORT`  
   The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
 
@@ -77,16 +75,16 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
 - `MINIO_PORT`  
   The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
 - `MINIO_USER`  
-  The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
+  The username for MinIO. 
 - `MINIO_PASSWORD`  
-  The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
+  The password for MinIO. accordingly.
 
 ### Redis
 
 - `REDIS_PORT`  
   The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
 - `REDIS_PASSWORD`  
-  The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
+  The password for Redis.
 
 ### RAGFlow
 
@@ -140,7 +138,7 @@ If you cannot download the RAGFlow Docker image, try the following mirrors.
 
 ## Service configuration
 
-[service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
+[service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
 
 ### `ragflow`
 
@@ -164,13 +162,13 @@ If you cannot download the RAGFlow Docker image, try the following mirrors.
 
 ### `oauth`  
 
-The OAuth configuration for signing up or signing in to RAGFlow using a third-party account.  It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
+The OAuth configuration for signing up or signing in to RAGFlow using a third-party account.  It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
 
 - `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings](https://github.com/settings/developers) page to obtain your client_id and secret_key.
 
 ### `user_default_llm`  
 
-The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.  
+The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.  
 
 - `factory`: The LLM supplier. Available options:
   - `"OpenAI"`

diff --git a/docs/guides/develop/launch_ragflow_from_source.md b/docs/guides/develop/launch_ragflow_from_source.md
@@ -64,13 +64,13 @@ docker compose -f docker/docker-compose-base.yml up -d
 
 ### Update `host` and `port` Settings for Third-party Services
 
-1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`:
+1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml.template** to `127.0.0.1`:
 
    ```
    127.0.0.1       es01 infinity mysql minio redis
    ```
 
-2. In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
+2. In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
 
 ### Launch the RAGFlow Backend Service
 

diff --git a/docs/guides/llm_api_key_setup.md b/docs/guides/llm_api_key_setup.md
@@ -19,7 +19,7 @@ If you find your online LLM is not on the list, don't feel disheartened. The lis
 
 You have two options for configuring your model API key:
 
-- Configure it in **service_conf.yaml** before starting RAGFlow.
+- Configure it in **service_conf.yaml.template** before starting RAGFlow.
 - Configure it on the **Model Providers** page after logging into RAGFlow.
 
 ### Configure model API key before starting up RAGFlow
@@ -37,7 +37,7 @@ You have two options for configuring your model API key:
 ### Configure model API key after logging into RAGFlow
 
 :::caution WARNING
-After logging into RAGFlow, configuring your model API key through the **service_conf.yaml** file will no longer take effect.
+After logging into RAGFlow, configuring your model API key through the **service_conf.yaml.template** file will no longer take effect.
 :::
 
 After logging into RAGFlow, you can *only* configure API Key on the **Model Providers** page: