Skip to content

Commit

Permalink
Use KAFKA_ as prefix for environment Kafka config (#209)
Browse files Browse the repository at this point in the history
  • Loading branch information
@philipp94831
philipp94831 authored Jul 23, 2024
1 parent 632b7fe commit be8ffc8
Show file tree
Hide file tree
Showing 14 changed files with 50 additions and 50 deletions.
2 changes: 1 addition & 1 deletion charts/producer-app-cleanup-job/templates/job.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ spec:
- name: ENV_PREFIX
value: {{ .Values.configurationEnvPrefix }}_
{{- range $key, $value := .Values.streams.config }}
- name: {{ printf "STREAMS_%s" $key | replace "." "_" | upper | quote }}
- name: {{ printf "KAFKA_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if hasKey .Values.streams "brokers" }}
Expand Down
20 changes: 10 additions & 10 deletions charts/producer-app/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,16 @@ Alternatively, a YAML file that specifies the values for the parameters can be p

### Other

| Parameter | Description | Default |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `configurationEnvPrefix` | Prefix for environment variables to use that should be parsed as command line arguments. | `APP` |
| `commandLine` | Map of command line arguments passed to the producer app. | `{}` |
| `debug` | Configure logging to debug | `false` |
| `env` | Custom environment variables | `{}` |
| `secrets` | Custom secret environment variables. Prefix with `configurationEnvPrefix` in order to pass secrets to command line or prefix with `STREAMS_` to pass secrets to Kafka Streams configuration. E.g., `APP_MY_PARAM` would be passed as `--my-param` and `STREAMS_MAX_POLL_TIMEOUT_MS` would be translated to `max.poll.timeout.ms`. | `{}` |
| `secretRefs` | Inject existing secrets as environment variables. Map key is used as environment variable name. Value consists of secret `name` and `key`. | `{}` |
| `secretFilesRefs` | Mount existing secrets as volumes | `[]` |
| `files` | Map of files to mount for the app. File will be mounted as `$value.mountPath/$key`. `$value.content` denotes file content (recommended to be used with `--set-file`). | `{}` |
| Parameter | Description | Default |
|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
| `configurationEnvPrefix` | Prefix for environment variables to use that should be parsed as command line arguments. | `APP` |
| `commandLine` | Map of command line arguments passed to the producer app. | `{}` |
| `debug` | Configure logging to debug | `false` |
| `env` | Custom environment variables | `{}` |
| `secrets` | Custom secret environment variables. Prefix with `configurationEnvPrefix` in order to pass secrets to command line or prefix with `KAFKA_` to pass secrets to Kafka Streams configuration. E.g., `APP_MY_PARAM` would be passed as `--my-param` and `KAFKA_MAX_POLL_TIMEOUT_MS` would be translated to `max.poll.timeout.ms`. | `{}` |
| `secretRefs` | Inject existing secrets as environment variables. Map key is used as environment variable name. Value consists of secret `name` and `key`. | `{}` |
| `secretFilesRefs` | Mount existing secrets as volumes | `[]` |
| `files` | Map of files to mount for the app. File will be mounted as `$value.mountPath/$key`. `$value.content` denotes file content (recommended to be used with `--set-file`). | `{}` |

### JVM

Expand Down
2 changes: 1 addition & 1 deletion charts/producer-app/templates/pod.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ spec:
- name: ENV_PREFIX
value: {{ .Values.configurationEnvPrefix }}_
{{- range $key, $value := .Values.streams.config }}
- name: {{ printf "STREAMS_%s" $key | replace "." "_" | upper | quote }}
- name: {{ printf "KAFKA_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if hasKey .Values.streams "brokers" }}
Expand Down
2 changes: 1 addition & 1 deletion charts/streams-app-cleanup-job/templates/job.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ spec:
- name: ENV_PREFIX
value: {{ .Values.configurationEnvPrefix }}_
{{- range $key, $value := .Values.streams.config }}
- name: {{ printf "STREAMS_%s" $key | replace "." "_" | upper | quote }}
- name: {{ printf "KAFKA_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if hasKey .Values.streams "brokers" }}
Expand Down
20 changes: 10 additions & 10 deletions charts/streams-app/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,16 +67,16 @@ Alternatively, a YAML file that specifies the values for the parameters can be p

### Other

| Parameter | Description | Default |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `configurationEnvPrefix` | Prefix for environment variables to use that should be parsed as command line arguments. | `APP` |
| `commandLine` | Map of command line arguments passed to the streams app. | `{}` |
| `debug` | Configure logging to debug | `false` |
| `env` | Custom environment variables | `{}` |
| `secrets` | Custom secret environment variables. Prefix with `configurationEnvPrefix` in order to pass secrets to command line or prefix with `STREAMS_` to pass secrets to Kafka Streams configuration. E.g., `APP_MY_PARAM` would be passed as `--my-param` and `STREAMS_MAX_POLL_TIMEOUT_MS` would be translated to `max.poll.timeout.ms`. | `{}` |
| `secretRefs` | Inject existing secrets as environment variables. Map key is used as environment variable name. Value consists of secret `name` and `key`. | `{}` |
| `secretFilesRefs` | Mount existing secrets as volumes | `[]` |
| `files` | Map of files to mount for the app. File will be mounted as `$value.mountPath/$key`. `$value.content` denotes file content (recommended to be used with `--set-file`). | `{}` |
| Parameter | Description | Default |
|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
| `configurationEnvPrefix` | Prefix for environment variables to use that should be parsed as command line arguments. | `APP` |
| `commandLine` | Map of command line arguments passed to the streams app. | `{}` |
| `debug` | Configure logging to debug | `false` |
| `env` | Custom environment variables | `{}` |
| `secrets` | Custom secret environment variables. Prefix with `configurationEnvPrefix` in order to pass secrets to command line or prefix with `KAFKA_` to pass secrets to Kafka Streams configuration. E.g., `APP_MY_PARAM` would be passed as `--my-param` and `KAFKA_MAX_POLL_TIMEOUT_MS` would be translated to `max.poll.timeout.ms`. | `{}` |
| `secretRefs` | Inject existing secrets as environment variables. Map key is used as environment variable name. Value consists of secret `name` and `key`. | `{}` |
| `secretFilesRefs` | Mount existing secrets as volumes | `[]` |
| `files` | Map of files to mount for the app. File will be mounted as `$value.mountPath/$key`. `$value.content` denotes file content (recommended to be used with `--set-file`). | `{}` |

### JMX Configuration

Expand Down
6 changes: 3 additions & 3 deletions charts/streams-app/templates/deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ spec:
- name: ENV_PREFIX
value: {{ .Values.configurationEnvPrefix }}_
{{- range $key, $value := .Values.streams.config }}
- name: {{ printf "STREAMS_%s" $key | replace "." "_" | upper | quote }}
- name: {{ printf "KAFKA_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- range .Values.ports }}
Expand All @@ -97,12 +97,12 @@ spec:
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: STREAMS_APPLICATION_SERVER
- name: KAFKA_APPLICATION_SERVER
value: "$(POD_IP):{{ .containerPort }}"
{{- end }}
{{- end }}
{{- if .Values.streams.staticMembership }}
- name: STREAMS_GROUP_INSTANCE_ID
- name: KAFKA_GROUP_INSTANCE_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
Expand Down
4 changes: 2 additions & 2 deletions streams-bootstrap-cli/src/main/java/com/bakdata/kafka/KafkaApplication.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,8 @@ public static int startApplicationWithoutExit(final KafkaApplication<?, ?, ?, ?,
}

private static String[] addEnvironmentVariablesArguments(final String[] args) {
Preconditions.checkArgument(!ENV_PREFIX.equals(EnvironmentStreamsConfigParser.PREFIX),
"Prefix '" + EnvironmentStreamsConfigParser.PREFIX + "' is reserved for Streams config");
Preconditions.checkArgument(!ENV_PREFIX.equals(EnvironmentKafkaConfigParser.PREFIX),
"Prefix '" + EnvironmentKafkaConfigParser.PREFIX + "' is reserved for Kafka config");
final List<String> environmentArguments = new EnvironmentArgumentsParser(ENV_PREFIX)
.parseVariables(System.getenv());
final Collection<String> allArgs = new ArrayList<>(environmentArguments);
Expand Down
2 changes: 1 addition & 1 deletion streams-bootstrap-core/src/main/java/com/bakdata/kafka/ConfiguredProducerApp.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ private static Map<String, Object> createBaseConfig(final KafkaEndpointConfig en
* </li>
* <li>
* Configs provided via environment variables (see
* {@link EnvironmentStreamsConfigParser#parseVariables(Map)})
* {@link EnvironmentKafkaConfigParser#parseVariables(Map)})
* </li>
* <li>
* Configs provided by {@link AppConfiguration#getKafkaConfig()}
Expand Down
2 changes: 1 addition & 1 deletion streams-bootstrap-core/src/main/java/com/bakdata/kafka/ConfiguredStreamsApp.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ private static Map<String, Object> createBaseConfig(final KafkaEndpointConfig en
* </li>
* <li>
* Configs provided via environment variables (see
* {@link EnvironmentStreamsConfigParser#parseVariables(Map)})
* {@link EnvironmentKafkaConfigParser#parseVariables(Map)})
* </li>
* <li>
* Configs provided by {@link AppConfiguration#getKafkaConfig()}
Expand Down
Loading

0 comments on commit be8ffc8

Please sign in to comment.