Skip to content

Commit 0c95a33

Browse files
gschmutzKevinHuShyuzhichang
authored
Dynamically create the service_conf.yaml file by replacing environment variables from .env (infiniflow#3341)
### What problem does this PR solve? This pull request implements the feature mentioned in infiniflow#3322. Instead of manually having to edit the `service_conf.yaml` file when changes have been made to `.env` and mapping it into the docker container at runtime, a template file is used and the values replaced by the environment variables from the `.env` file when the container is started. ### Type of change - [X] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Kevin Hu <[email protected]> Co-authored-by: Zhichang Yu <[email protected]>
1 parent 7274420 commit 0c95a33

12 files changed

+122
-35
lines changed

‎Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"
129129

130130
ENV PYTHONPATH=/ragflow/
131131

132+
COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
132133
COPY docker/entrypoint.sh ./entrypoint.sh
133134
RUN chmod +x ./entrypoint.sh
134135

‎Dockerfile.scratch.oc9

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ RUN conda run -n py11 python -m nltk.downloader wordnet
5353
ENV PYTHONPATH=/ragflow/
5454
ENV HF_ENDPOINT=https://hf-mirror.com
5555

56+
COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
5657
ADD docker/entrypoint.sh ./entrypoint.sh
5758
RUN chmod +x ./entrypoint.sh
5859

‎Dockerfile.slim

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"
122122

123123
ENV PYTHONPATH=/ragflow/
124124

125+
COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
125126
COPY docker/entrypoint.sh ./entrypoint.sh
126127
RUN chmod +x ./entrypoint.sh
127128

‎README.md

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ releases! 🌟
216216
5. In your web browser, enter the IP address of your server and log in to RAGFlow.
217217
> With the default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default
218218
HTTP serving port `80` can be omitted when using the default configurations.
219-
6. In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM factory in `user_default_llm` and update
219+
6. In [service_conf.yaml.template](./docker/service_conf.yaml.template), select the desired LLM factory in `user_default_llm` and update
220220
the `API_KEY` field with the corresponding API key.
221221
222222
> See [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) for more information.
@@ -229,16 +229,11 @@ When it comes to system configurations, you will need to manage the following fi
229229
230230
- [.env](./docker/.env): Keeps the fundamental setups for the system, such as `SVR_HTTP_PORT`, `MYSQL_PASSWORD`, and
231231
`MINIO_PASSWORD`.
232-
- [service_conf.yaml](./docker/service_conf.yaml): Configures the back-end services.
233-
- [docker-compose.yml](./docker/docker-compose.yml): The system relies
234-
on [docker-compose.yml](./docker/docker-compose.yml) to start up.
235-
236-
You must ensure that changes to the [.env](./docker/.env) file are in line with what are in the [service_conf.yaml](./docker/service_conf.yaml) file.
232+
- [service_conf.yaml.template](./docker/service_conf.yaml.template): Configures the back-end services. The environment variables in this file will be automatically populated when the Docker container starts. Any environment variables set within the Docker container will be available for use, allowing you to customize service behavior based on the deployment environment.
233+
- [docker-compose.yml](./docker/docker-compose.yml): The system relies on [docker-compose.yml](./docker/docker-compose.yml) to start up.
237234
238235
> The [./docker/README](./docker/README.md) file provides a detailed description of the environment settings and service
239-
> configurations, and you are REQUIRED to ensure that all environment settings listed in
240-
> the [./docker/README](./docker/README.md) file are aligned with the corresponding configurations in
241-
> the [service_conf.yaml](./docker/service_conf.yaml) file.
236+
> configurations which can be used as `${ENV_VARS}` in the [service_conf.yaml.template](./docker/service_conf.yaml.template) file.
242237
243238
To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80`
244239
to `<YOUR_SERVING_PORT>:80`.
@@ -293,11 +288,11 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
293288
docker compose -f docker/docker-compose-base.yml up -d
294289
```
295290
296-
Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`:
291+
Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/.env** to `127.0.0.1`:
297292
```
298293
127.0.0.1 es01 infinity mysql minio redis
299294
```
300-
In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
295+
In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
301296
302297
4. If you cannot access HuggingFace, set the `HF_ENDPOINT` environment variable to use a mirror site:
303298

‎docker/.env

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
# The version of Elasticsearch.
22
STACK_VERSION=8.11.3
33

4+
# The hostname where the Elasticsearch service is exposed
5+
ES_HOST=es01
6+
47
# The port used to expose the Elasticsearch service to the host machine,
58
# allowing EXTERNAL access to the service running inside the Docker container.
69
ES_PORT=1200
@@ -27,10 +30,16 @@ INFINITY_PSQL_PORT=5432
2730
# The password for MySQL.
2831
# When updated, you must revise the `mysql.password` entry in service_conf.yaml.
2932
MYSQL_PASSWORD=infini_rag_flow
33+
# The hostname where the MySQL service is exposed
34+
MYSQL_HOST=mysql
35+
# The database of the MySQL service to use
36+
MYSQL_DBNAME=rag_flow
3037
# The port used to expose the MySQL service to the host machine,
3138
# allowing EXTERNAL access to the MySQL database running inside the Docker container.
3239
MYSQL_PORT=5455
3340

41+
# The hostname where the MySQL service is exposed
42+
MINIO_HOST=minio
3443
# The port used to expose the MinIO console interface to the host machine,
3544
# allowing EXTERNAL access to the web-based console running inside the Docker container.
3645
MINIO_CONSOLE_PORT=9001
@@ -44,6 +53,8 @@ MINIO_USER=rag_flow
4453
# When updated, you must revise the `minio.password` entry in service_conf.yaml accordingly.
4554
MINIO_PASSWORD=infini_rag_flow
4655

56+
# The hostname where the Redis service is exposed
57+
REDIS_HOST=redis
4758
# The port used to expose the Redis service to the host machine,
4859
# allowing EXTERNAL access to the Redis service running inside the Docker container.
4960
REDIS_PORT=6379

‎docker/README.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ The [.env](./.env) file contains important environment variables for Docker.
2727
- `ES_PORT`
2828
The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `1200`.
2929
- `ELASTIC_PASSWORD`
30-
The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
30+
The password for Elasticsearch.
3131

3232
### Kibana
3333

@@ -46,7 +46,7 @@ The [.env](./.env) file contains important environment variables for Docker.
4646
### MySQL
4747

4848
- `MYSQL_PASSWORD`
49-
The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
49+
The password for MySQL.
5050
- `MYSQL_PORT`
5151
The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
5252

@@ -57,16 +57,16 @@ The [.env](./.env) file contains important environment variables for Docker.
5757
- `MINIO_PORT`
5858
The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
5959
- `MINIO_USER`
60-
The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
60+
The username for MinIO.
6161
- `MINIO_PASSWORD`
62-
The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
62+
The password for MinIO.
6363

6464
### Redis
6565

6666
- `REDIS_PORT`
6767
The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
6868
- `REDIS_PASSWORD`
69-
The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
69+
The password for Redis.
7070

7171
### RAGFlow
7272

@@ -119,7 +119,7 @@ The [.env](./.env) file contains important environment variables for Docker.
119119

120120
## 🐋 Service configuration
121121

122-
[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
122+
[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. In a dockerized setup, this file is automatically created based on the [service_conf.yaml.template](./service_conf.yaml.template) file (replacing all environment variables by their values).
123123

124124
- `ragflow`
125125
- `host`: The API server's IP address inside the Docker container. Defaults to `0.0.0.0`.
@@ -139,11 +139,11 @@ The [.env](./.env) file contains important environment variables for Docker.
139139
- `host`: The MinIO serving IP *and* port inside the Docker container. Defaults to `minio:9000`.
140140

141141
- `oauth`
142-
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
142+
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
143143
- `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings page](https://github.com/settings/developers) to obtain your client_id and secret_key.
144144

145145
- `user_default_llm`
146-
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
146+
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
147147
- `factory`: The LLM supplier. Available options:
148148
- `"OpenAI"`
149149
- `"DeepSeek"`

‎docker/docker-compose-gpu.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@ services:
1616
- 80:80
1717
- 443:443
1818
volumes:
19-
- ./service_conf.yaml:/ragflow/conf/service_conf.yaml
2019
- ./ragflow-logs:/ragflow/logs
2120
- ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
2221
- ./nginx/proxy.conf:/etc/nginx/proxy.conf

‎docker/entrypoint.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,12 @@
11
#!/bin/bash
22

3+
# replace env variables in the service_conf.yaml file
4+
rm -rf /ragflow/conf/service_conf.yaml
5+
while IFS= read -r line || [[ -n "$line" ]]; do
6+
# Use eval to interpret the variable with default values
7+
eval "echo \"$line\"" >> /ragflow/conf/service_conf.yaml
8+
done < /ragflow/conf/service_conf.yaml.template
9+
310
# unset http proxy which maybe set by docker daemon
411
export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
512

‎docker/service_conf.yaml.template

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
ragflow:
2+
host: ${RAGFLOW_HOST:-0.0.0.0}
3+
http_port: 9380
4+
mysql:
5+
name: '${MYSQL_DBNAME:-rag_flow}'
6+
user: '${MYSQL_USER:-root}'
7+
password: '${MYSQL_PASSWORD:-infini_rag_flow}'
8+
host: '${MYSQL_HOST:-mysql}'
9+
port: 3306
10+
max_connections: 100
11+
stale_timeout: 30
12+
minio:
13+
user: '${MINIO_USER:-rag_flow}'
14+
password: '${MINIO_PASSWORD:-infini_rag_flow}'
15+
host: '${MINIO_HOST:-minio}:9000'
16+
es:
17+
hosts: 'http://${ES_HOST:-es01}:9200'
18+
username: '${ES_USER:-elastic}'
19+
password: '${ES_PASSWORD:-infini_rag_flow}'
20+
redis:
21+
db: 1
22+
password: '${REDIS_PASSWORD:-infini_rag_flow}'
23+
host: '${REDIS_HOST:-redis}:6379'
24+
25+
# postgres:
26+
# name: '${POSTGRES_DBNAME:-rag_flow}'
27+
# user: '${POSTGRES_USER:-rag_flow}'
28+
# password: '${POSTGRES_PASSWORD:-infini_rag_flow}'
29+
# host: '${POSTGRES_HOST:-postgres}'
30+
# port: 5432
31+
# max_connections: 100
32+
# stale_timeout: 30
33+
# s3:
34+
# endpoint: 'endpoint'
35+
# access_key: 'access_key'
36+
# secret_key: 'secret_key'
37+
# region: 'region'
38+
# azure:
39+
# auth_type: 'sas'
40+
# container_url: 'container_url'
41+
# sas_token: 'sas_token'
42+
# azure:
43+
# auth_type: 'spn'
44+
# account_url: 'account_url'
45+
# client_id: 'client_id'
46+
# secret: 'secret'
47+
# tenant_id: 'tenant_id'
48+
# container_name: 'container_name'
49+
# user_default_llm:
50+
# factory: 'Tongyi-Qianwen'
51+
# api_key: 'sk-xxxxxxxxxxxxx'
52+
# base_url: ''
53+
# oauth:
54+
# github:
55+
# client_id: xxxxxxxxxxxxxxxxxxxxxxxxx
56+
# secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
57+
# url: https://github.com/login/oauth/access_token
58+
# feishu:
59+
# app_id: cli_xxxxxxxxxxxxxxxxxxx
60+
# app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
61+
# app_access_token_url: https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal
62+
# user_access_token_url: https://open.feishu.cn/open-apis/authen/v1/oidc/access_token
63+
# grant_type: 'authorization_code'
64+
# authentication:
65+
# client:
66+
# switch: false
67+
# http_app_key:
68+
# http_secret_key:
69+
# site:
70+
# switch: false
71+
# permission:
72+
# switch: false
73+
# component: false
74+
# dataset: false

‎docs/configurations.md

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,9 @@ Configurations for installing RAGFlow via Docker.
1212
When it comes to system configurations, you will need to manage the following files:
1313

1414
- [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env): Contains important environment variables for Docker.
15-
- [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
15+
- [service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor. Upon container startup, the `service_conf.yaml` file will be generated based on this template file. This process replaces any environment variables within the template, allowing for dynamic configuration tailored to the container's environment.
1616
- [docker-compose.yml](https://github.com/infiniflow/ragflow/blob/main/docker/docker-compose.yml): The Docker Compose file for starting up the RAGFlow service.
1717

18-
You must ensure that changes to the [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file are in line with what are in the [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) file.
19-
2018
To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80`
2119
to `<YOUR_SERVING_PORT>:80`.
2220

@@ -47,7 +45,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
4745
- `ES_PORT`
4846
The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `1200`.
4947
- `ELASTIC_PASSWORD`
50-
The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
48+
The password for Elasticsearch.
5149

5250
### Kibana
5351

@@ -66,7 +64,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
6664
### MySQL
6765

6866
- `MYSQL_PASSWORD`
69-
The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
67+
The password for MySQL.
7068
- `MYSQL_PORT`
7169
The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
7270

@@ -77,16 +75,16 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
7775
- `MINIO_PORT`
7876
The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
7977
- `MINIO_USER`
80-
The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
78+
The username for MinIO.
8179
- `MINIO_PASSWORD`
82-
The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
80+
The password for MinIO. accordingly.
8381

8482
### Redis
8583

8684
- `REDIS_PORT`
8785
The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
8886
- `REDIS_PASSWORD`
89-
The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
87+
The password for Redis.
9088

9189
### RAGFlow
9290

@@ -140,7 +138,7 @@ If you cannot download the RAGFlow Docker image, try the following mirrors.
140138

141139
## Service configuration
142140

143-
[service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
141+
[service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
144142

145143
### `ragflow`
146144

@@ -164,13 +162,13 @@ If you cannot download the RAGFlow Docker image, try the following mirrors.
164162

165163
### `oauth`
166164

167-
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
165+
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
168166

169167
- `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings](https://github.com/settings/developers) page to obtain your client_id and secret_key.
170168

171169
### `user_default_llm`
172170

173-
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
171+
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
174172

175173
- `factory`: The LLM supplier. Available options:
176174
- `"OpenAI"`

‎docs/guides/develop/launch_ragflow_from_source.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,13 +64,13 @@ docker compose -f docker/docker-compose-base.yml up -d
6464

6565
### Update `host` and `port` Settings for Third-party Services
6666

67-
1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`:
67+
1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml.template** to `127.0.0.1`:
6868

6969
```
7070
127.0.0.1 es01 infinity mysql minio redis
7171
```
7272

73-
2. In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
73+
2. In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
7474

7575
### Launch the RAGFlow Backend Service
7676

‎docs/guides/llm_api_key_setup.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ If you find your online LLM is not on the list, don't feel disheartened. The lis
1919

2020
You have two options for configuring your model API key:
2121

22-
- Configure it in **service_conf.yaml** before starting RAGFlow.
22+
- Configure it in **service_conf.yaml.template** before starting RAGFlow.
2323
- Configure it on the **Model Providers** page after logging into RAGFlow.
2424

2525
### Configure model API key before starting up RAGFlow
@@ -37,7 +37,7 @@ You have two options for configuring your model API key:
3737
### Configure model API key after logging into RAGFlow
3838

3939
:::caution WARNING
40-
After logging into RAGFlow, configuring your model API key through the **service_conf.yaml** file will no longer take effect.
40+
After logging into RAGFlow, configuring your model API key through the **service_conf.yaml.template** file will no longer take effect.
4141
:::
4242

4343
After logging into RAGFlow, you can *only* configure API Key on the **Model Providers** page:

0 commit comments

Comments
 (0)