diff --git a/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md b/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md index 6040b0e45b2a6..aae7d94acbafc 100644 --- a/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md +++ b/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md @@ -13,7 +13,7 @@ chapter = true 在解析并加载 YAML 文件为 ShardingSphere 的元数据后， 会再次通过[模式配置](../../../java-api/mode)的相关配置决定下一步行为。讨论两种情况： 1. 元数据持久化仓库中不存在 ShardingSphere 的元数据，本地元数据将被存储到元数据持久化仓库。 -2. 元数据持久化仓库中已存在与本地元数据不同的 ShardingSphere 的元数据，本地元数据将被元数据持久化仓库的元数据覆盖。 +2. 元数据持久化仓库中已存在 ShardingSphere 的元数据，无论是否与本地元数据相同，本地元数据将被元数据持久化仓库的元数据覆盖。 对元数据持久化仓库的配置需参考[元数据持久化仓库](../../../../common-config/builtin-algorithm/metadata-repository)。 @@ -33,5 +33,39 @@ chapter = true 用例： - `jdbc:shardingsphere:absolutepath:/path/to/config.yaml` +### 从类路径中加载包含环境变量的配置文件 + +加载 classpath 中包含环境变量的 config.yaml 配置文件的 JDBC URL，通过 `jdbc:shardingsphere:classpath-environment:` 前缀识别。 +配置文件为 `xxx.yaml`，配置文件格式与 [YAML 配置](../../../yaml-config)基本一致。 +在涉及的 YAML 文件中，允许通过环境变量设置特定YAML属性的值，并配置可选的默认值。这常用于 Docker Image 的部署场景。 +环境变量的名称和其可选的默认值通过`::`分割，在最外层通过`$${`和`}`包裹。 + +讨论两种情况。 +1. 当对应的环境变量不存在时，此 YAML 属性的值将被设置为`::`右侧的默认值。 +2. 当对应的环境变量和`::`右侧的默认值均不存在时，此属性将被设置为空。 + +假设存在以下一组环境变量， +1. 存在环境变量`FIXTURE_JDBC_URL`为`jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`。 +2. 存在环境变量`FIXTURE_USERNAME`为`sa`。 + +则对于以下 YAML 文件的截取片段， +```yaml +ds_1: + dataSourceClassName: com.zaxxer.hikari.HikariDataSource + driverClassName: $${FIXTURE_DRIVER_CLASS_NAME::org.h2.Driver} + jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use} + username: $${FIXTURE_USERNAME::} + password: $${FIXTURE_PASSWORD::} +``` +此 YAML 截取片段将被解析为， +```yaml +ds_1: + dataSourceClassName: com.zaxxer.hikari.HikariDataSource + driverClassName: org.h2.Driver + jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL + username: sa + password: +``` + ### 其他实现 具体可参考 https://github.com/apache/shardingsphere-plugin 。 diff --git a/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md b/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md index a78f2579e2dfc..4244252b6d907 100644 --- a/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md +++ b/docs/document/content/user-manual/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md @@ -12,10 +12,11 @@ allows YAML configuration files to be fetched from multiple sources and File Sys After parsing and loading the YAML file into ShardingSphere's metadata, The next behavior will be determined again through the relevant configuration of [Mode Configuration](../../../java-api/mode). Discuss two situations: -1. ShardingSphere’s metadata does not exist in the metadata persistence warehouse, and local metadata will be stored in the metadata persistence warehouse. -2. Metadata of ShardingSphere that is different from local metadata already exists in the metadata persistence warehouse, and the local metadata will be overwritten by the metadata of the metadata persistence warehouse. +1. ShardingSphere’s metadata does not exist in the metadata repository, and local metadata will be stored in the metadata repository. +2. ShardingSphere’s metadata already exists in the metadata repository, regardless of whether it is the same as the local metadata, +the local metadata will be overwritten by the metadata of the metadata repository. -For the configuration of the metadata persistence warehouse, please refer to [Metadata Persistence Warehouse](../../../../common-config/builtin-algorithm/metadata-repository). +For the configuration of the metadata repository, please refer to [Metadata Repository](../../../../common-config/builtin-algorithm/metadata-repository). ## URL configuration @@ -33,5 +34,40 @@ The configuration file is `xxx.yaml`, and the configuration file format is consi Example: - `jdbc:shardingsphere:absolutepath:/path/to/config.yaml` +### Load configuration file containing environment variables from classpath + +JDBC URL to load the config.yaml configuration file that contains environment variables in classpath, identified by the `jdbc:shardingsphere:classpath-environment:` prefix. +The configuration file is `xxx.yaml`, and the configuration file format is basically the same as [YAML configuration](../../../yaml-config). +Allows setting the value of specific YAML properties via environment variables and configuring optional default values in the involved YAML files. +This is commonly used in Docker Image deployment scenarios. +The name of an environment variable and its optional default value are separated by `::` and wrapped in the outermost layer by `$${` and `}`. + +Discuss two situations. +1. When the corresponding environment variable does not exist, the value of this YAML attribute will be set to the default value on the right side of `::`. +2. When the corresponding environment variable and the default value on the right side of `::` do not exist, this property will be set to empty. + +Assume that the following set of environment variables exists, +1. The existing environment variable `FIXTURE_JDBC_URL` is `jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`. +2. The existing environment variable `FIXTURE_USERNAME` is `sa`. + +Then for the intercepted fragment of the following YAML file, +```yaml +ds_1: + dataSourceClassName: com.zaxxer.hikari.HikariDataSource + driverClassName: $${FIXTURE_DRIVER_CLASS_NAME::org.h2.Driver} + jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use} + username: $${FIXTURE_USERNAME::} + password: $${FIXTURE_PASSWORD::} +``` +This YAML snippet will be parsed as, +```yaml +ds_1: + dataSourceClassName: com.zaxxer.hikari.HikariDataSource + driverClassName: org.h2.Driver + jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL + username: sa + password: +``` + ### Other implementations For details, please refer to https://github.com/apache/shardingsphere-plugin . diff --git a/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/AbstractClasspathURLProvider.java b/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/AbstractClasspathURLProvider.java new file mode 100644 index 0000000000000..ce9cc17526092 --- /dev/null +++ b/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/AbstractClasspathURLProvider.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath; + +import com.google.common.base.Preconditions; +import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider; + +import java.io.InputStream; + +/** + * Abstract classpath URL provider. + */ +public abstract class AbstractClasspathURLProvider implements ShardingSphereURLProvider { + + String getConfigurationFile(final String url, final String urlPrefix, final String typePrefix) { + String configuredFile = url.substring(urlPrefix.length(), url.contains("?") ? url.indexOf('?') : url.length()); + String file = configuredFile.substring(typePrefix.length()); + Preconditions.checkArgument(!file.isEmpty(), "Configuration file is required in ShardingSphere URL."); + return file; + } + + InputStream getResourceAsStream(final String resource) { + InputStream result = Thread.currentThread().getContextClassLoader().getResourceAsStream(resource); + result = null == result ? Thread.currentThread().getContextClassLoader().getResourceAsStream("/" + resource) : result; + if (null != result) { + return result; + } + throw new IllegalArgumentException(String.format("Can not find configuration file `%s`.", resource)); + } +} diff --git a/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/ClasspathURLProvider.java b/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathURLProvider.java similarity index 60% rename from jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/ClasspathURLProvider.java rename to jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathURLProvider.java index d34e395c37c62..4f9e9fd9e8abc 100644 --- a/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/ClasspathURLProvider.java +++ b/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathURLProvider.java @@ -15,12 +15,10 @@ * limitations under the License. */ -package org.apache.shardingsphere.driver.jdbc.core.driver.spi; +package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath; -import com.google.common.base.Preconditions; import com.google.common.base.Strings; import lombok.SneakyThrows; -import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider; import java.io.BufferedReader; import java.io.IOException; @@ -31,21 +29,19 @@ /** * Classpath URL provider. */ -public final class ClasspathURLProvider implements ShardingSphereURLProvider { +public final class ClasspathURLProvider extends AbstractClasspathURLProvider { - private static final String CLASSPATH_TYPE = "classpath:"; + private static final String TYPE_PREFIX = "classpath:"; @Override public boolean accept(final String url) { - return !Strings.isNullOrEmpty(url) && url.contains(CLASSPATH_TYPE); + return !Strings.isNullOrEmpty(url) && url.contains(TYPE_PREFIX); } @Override @SneakyThrows(IOException.class) public byte[] getContent(final String url, final String urlPrefix) { - String configuredFile = url.substring(urlPrefix.length(), url.contains("?") ? url.indexOf('?') : url.length()); - String file = configuredFile.substring(CLASSPATH_TYPE.length()); - Preconditions.checkArgument(!file.isEmpty(), "Configuration file is required in ShardingSphere URL."); + String file = getConfigurationFile(url, urlPrefix, TYPE_PREFIX); try ( InputStream stream = getResourceAsStream(file); BufferedReader reader = new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) { @@ -59,13 +55,4 @@ public byte[] getContent(final String url, final String urlPrefix) { return builder.toString().getBytes(StandardCharsets.UTF_8); } } - - private InputStream getResourceAsStream(final String resource) { - InputStream result = Thread.currentThread().getContextClassLoader().getResourceAsStream(resource); - result = null == result ? Thread.currentThread().getContextClassLoader().getResourceAsStream("/" + resource) : result; - if (null != result) { - return result; - } - throw new IllegalArgumentException(String.format("Can not find configuration file `%s`.", resource)); - } } diff --git a/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProvider.java b/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProvider.java new file mode 100644 index 0000000000000..036086bc5373b --- /dev/null +++ b/jdbc/core/src/main/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProvider.java @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath; + +import com.google.common.base.Strings; +import lombok.SneakyThrows; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +/** + * Classpath with environment variables URL provider. + */ +public final class ClasspathWithEnvironmentURLProvider extends AbstractClasspathURLProvider { + + private static final String TYPE_PREFIX = "classpath-environment:"; + + private static final String KEY_VALUE_SEPARATOR = "::"; + + @SuppressWarnings("RegExpRedundantEscape") + private static final Pattern PATTERN = Pattern.compile("\\$\\$\\{(.+::.*)\\}$"); + + @Override + public boolean accept(final String url) { + return !Strings.isNullOrEmpty(url) && url.contains(TYPE_PREFIX); + } + + @Override + @SneakyThrows(IOException.class) + public byte[] getContent(final String url, final String urlPrefix) { + String file = getConfigurationFile(url, urlPrefix, TYPE_PREFIX); + try ( + InputStream stream = getResourceAsStream(file); + BufferedReader reader = new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) { + StringBuilder builder = new StringBuilder(); + String line; + while (null != (line = reader.readLine())) { + if (!line.startsWith("#")) { + line = replaceEnvironmentVariables(line); + builder.append(line).append('\n'); + } + } + return builder.toString().getBytes(StandardCharsets.UTF_8); + } + } + + private String replaceEnvironmentVariables(final String line) { + Matcher matcher = PATTERN.matcher(line); + if (matcher.find()) { + StringBuffer modifiedLine = new StringBuffer(); + String[] envNameAndDefaultValue = matcher.group(1).split(KEY_VALUE_SEPARATOR, 2); + String envName = envNameAndDefaultValue[0]; + String envValue = getEnvironmentVariables(envName); + if (Strings.isNullOrEmpty(envValue) && envNameAndDefaultValue[1].isEmpty()) { + matcher.appendReplacement(modifiedLine, ""); + return modifiedLine.substring(0, modifiedLine.length() - 1); + } + if (Strings.isNullOrEmpty(envValue)) { + envValue = envNameAndDefaultValue[1]; + } + matcher.appendReplacement(modifiedLine, envValue); + return modifiedLine.toString(); + } + return line; + } + + /** + * This method is only used for mocking environment variables in unit tests and should not be used under any circumstances. + * + * @param name the name of the environment variable + * @return the string value of the variable, or null if the variable is not defined in the system environment + */ + String getEnvironmentVariables(final String name) { + return System.getenv(name); + } +} diff --git a/jdbc/core/src/main/resources/META-INF/services/org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider b/jdbc/core/src/main/resources/META-INF/services/org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider index df99d5c8e8e50..e1f3b18d8b5d0 100644 --- a/jdbc/core/src/main/resources/META-INF/services/org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider +++ b/jdbc/core/src/main/resources/META-INF/services/org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider @@ -16,4 +16,5 @@ # org.apache.shardingsphere.driver.jdbc.core.driver.spi.AbsolutePathURLProvider -org.apache.shardingsphere.driver.jdbc.core.driver.spi.ClasspathURLProvider +org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathURLProvider +org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathWithEnvironmentURLProvider diff --git a/jdbc/core/src/test/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProviderTest.java b/jdbc/core/src/test/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProviderTest.java new file mode 100644 index 0000000000000..09fe9e14e2eee --- /dev/null +++ b/jdbc/core/src/test/java/org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProviderTest.java @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath; + +import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager; +import org.junit.jupiter.api.Test; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +public class ClasspathWithEnvironmentURLProviderTest { + + @Test + void assertReplaceEnvironmentVariables() { + final String urlPrefix = "jdbc:shardingsphere:"; + ClasspathWithEnvironmentURLProvider spy = spy(new ClasspathWithEnvironmentURLProvider()); + when(spy.getEnvironmentVariables("FIXTURE_JDBC_URL")).thenReturn("jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL"); + when(spy.getEnvironmentVariables("FIXTURE_USERNAME")).thenReturn("sa"); + byte[] actual = spy.getContent("jdbc:shardingsphere:classpath-environment:config/driver/foo-driver-environment-variables-fixture.yaml", urlPrefix); + byte[] actualOrigin = ShardingSphereURLManager.getContent("jdbc:shardingsphere:classpath:config/driver/foo-driver-fixture.yaml", urlPrefix); + assertThat(actual.length, is(999)); + assertThat(actual, is(actualOrigin)); + } +} diff --git a/jdbc/core/src/test/resources/config/driver/foo-driver-environment-variables-fixture.yaml b/jdbc/core/src/test/resources/config/driver/foo-driver-environment-variables-fixture.yaml new file mode 100644 index 0000000000000..36b2aaf73c2ef --- /dev/null +++ b/jdbc/core/src/test/resources/config/driver/foo-driver-environment-variables-fixture.yaml @@ -0,0 +1,58 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# After `ShardingSphereURLManager.getContent`, this file should be equivalent to `foo-driver-fixture.yaml` in the same folder. +databaseName: foo_driver_fixture_db + +dataSources: + ds_0: + dataSourceClassName: com.zaxxer.hikari.HikariDataSource + driverClassName: org.h2.Driver + jdbcUrl: jdbc:h2:mem:foo_ds_0;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL + username: sa + password: + ds_1: + dataSourceClassName: com.zaxxer.hikari.HikariDataSource + driverClassName: $${FIXTURE_DRIVER_CLASS_NAME::org.h2.Driver} + jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use} + username: $${FIXTURE_USERNAME::} + password: $${FIXTURE_PASSWORD::} + +rules: + - !SHARDING + autoTables: + t_order: + actualDataSources: ds_0,ds_1 + shardingStrategy: + standard: + shardingColumn: order_id + shardingAlgorithmName: auto_mod + keyGenerateStrategy: + column: user_id + keyGeneratorName: snowflake + shardingAlgorithms: + auto_mod: + type: HASH_MOD + props: + sharding-count: 2 + + keyGenerators: + snowflake: + type: SNOWFLAKE + +props: + sql-show: true