-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[confmap] - new feature flag for append merging strategy (#12097)
Koanf's default merging strategy currently overrides static values such as slices, numbers, and strings. However, lists of components should be treated as a special case. This pull request introduces a new command line option to allow for merging lists instead of discarding the existing ones. With this new merging strategy: - All the lists are merged rather than replaced. - The merging logic is name-aware, meaning that if components with the same name appear in both lists, they will only appear once in the final merged list. <!-- Issue number if applicable --> #### Link to tracking issue Related issues: - #8754 - #8394 - #10370 <!--Describe what testing was performed and which tests were added.--> #### Testing - Added <!--Describe the documentation added.--> #### Documentation - Added under readme. ##### Example Consider the following configs, ```yaml # main.yaml receivers: otlp/in: processors: batch: exporters: otlp/out: extensions: file_storage: service: pipelines: traces: receivers: [ otlp/in ] processors: [ batch ] exporters: [ otlp/out ] extensions: [ file_storage ] ``` ```yaml # extra_extension.yaml extensions: healthcheckv2: service: extensions: [ healthcheckv2 ] ``` If you run the collector with following command, ``` otelcol --config=main.yaml --config=extra_extension.yaml --feature-gates=-confmap.enableMergeAppendOption ``` then the final configuration after config resolution will look like following: ```yaml # main.yaml receivers: otlp/in: processors: batch: exporters: otlp/out: extensions: file_storage: healthcheckv2: service: pipelines: traces: receivers: [ otlp/in ] processors: [ batch ] exporters: [ otlp/out ] extensions: [ file_storage, healthcheckv2 ] ``` For backward compatibly, the default behaviour is **not** to merge lists. Users who want to explicitly merge lists can enable the command line option. Note: I’d appreciate your feedback on this 🙏
- Loading branch information
1 parent
72878b7
commit 6e82944
Showing
48 changed files
with
1,286 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Use this changelog template to create an entry for release notes. | ||
|
|
||
| # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' | ||
| change_type: enhancement | ||
|
|
||
| # The name of the component, or a single word describing the area of concern, (e.g. otlpreceiver) | ||
| component: confmap | ||
|
|
||
| # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). | ||
| note: Introduce a new feature flag to allow for merging lists instead of discarding the existing ones. | ||
|
|
||
| # One or more tracking issues or pull requests related to the change | ||
| issues: [8394, 8754, 10370] | ||
|
|
||
| # (Optional) One or more lines of additional information to render under the primary note. | ||
| # These lines will be padded with 2 spaces and then inserted directly into the document. | ||
| # Use pipe (|) for multiline entries. | ||
| subtext: | | ||
| You can enable this option via the command line by running following command: | ||
| otelcol --config=main.yaml --config=extra_config.yaml --feature-gates=-confmap.enableMergeAppendOption | ||
| # Optional: The change log or logs in which this entry should be included. | ||
| # e.g. '[user]' or '[user, api]' | ||
| # Include 'user' if the change is relevant to end users. | ||
| # Include 'api' if there is a change to a library API. | ||
| # Default: '[user]' | ||
| change_logs: [] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package confmap // import "go.opentelemetry.io/collector/confmap" | ||
|
|
||
| import ( | ||
| "reflect" | ||
| ) | ||
|
|
||
| func mergeAppend(src, dest map[string]any) error { | ||
| // mergeAppend recursively merges the src map into the dest map (left to right), | ||
| // modifying and expanding the dest map in the process. | ||
| // This function does not overwrite lists, and ensures that the final value is a name-aware | ||
| // copy of lists from src and dest. | ||
|
|
||
| for sKey, sVal := range src { | ||
| dVal, dOk := dest[sKey] | ||
| if !dOk { | ||
| // key is not present in destination config. Hence, add it to destination map | ||
| dest[sKey] = sVal | ||
| continue | ||
| } | ||
|
|
||
| srcVal := reflect.ValueOf(sVal) | ||
| destVal := reflect.ValueOf(dVal) | ||
|
|
||
| if destVal.Kind() != srcVal.Kind() { | ||
| // different kinds. Override the destination map | ||
| dest[sKey] = sVal | ||
| continue | ||
| } | ||
|
|
||
| switch srcVal.Kind() { | ||
| case reflect.Array, reflect.Slice: | ||
| // both of them are array. Merge them | ||
| dest[sKey] = mergeSlice(srcVal, destVal) | ||
| case reflect.Map: | ||
| // both of them are maps. Recursively call the mergeAppend | ||
| _ = mergeAppend(sVal.(map[string]any), dVal.(map[string]any)) | ||
| default: | ||
| // any other datatype. Override the destination map | ||
| dest[sKey] = sVal | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| func mergeSlice(src, dest reflect.Value) any { | ||
| slice := reflect.MakeSlice(src.Type(), 0, src.Cap()+dest.Cap()) | ||
| for i := 0; i < dest.Len(); i++ { | ||
| slice = reflect.Append(slice, dest.Index(i)) | ||
| } | ||
|
|
||
| for i := 0; i < src.Len(); i++ { | ||
| if isPresent(slice, src.Index(i)) { | ||
| continue | ||
| } | ||
| slice = reflect.Append(slice, src.Index(i)) | ||
| } | ||
| return slice.Interface() | ||
| } | ||
|
|
||
| func isPresent(slice reflect.Value, val reflect.Value) bool { | ||
| for i := 0; i < slice.Len(); i++ { | ||
| if slice.Index(i).Equal(val) { | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.