-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[elasticsearchexporter]: Add dynamic document id support for logs #37065
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good at a high level, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Q: do you expect the elasticsearch.document_id
to be indexed under attributes as well, or should it be ignored during serialization?
Haven't thought of that, but I think it makes sense to remove the field, as it only has meaning for the exporter. Also, |
I updated the code to remove the id field from the final document. Regarding the attribute name, I wonder if we should stick to some kind of semantic convention such as https://opentelemetry.io/docs/specs/semconv/database/elasticsearch/. I scrolled throught it but couldn't find an existing attribute for a document id. |
@@ -61,6 +61,9 @@ type Config struct { | |||
// fall back to pure TracesIndex, if 'elasticsearch.index.prefix' or 'elasticsearch.index.suffix' are not found in resource or attribute (prio: resource > attribute) | |||
TracesDynamicIndex DynamicIndexSetting `mapstructure:"traces_dynamic_index"` | |||
|
|||
// LogsDynamicID is used to configure the document id for logs. | |||
LogsDynamicID DynamicIndexSetting `mapstructure:"logs_dynamic_id"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LogsDynamicID DynamicIndexSetting `mapstructure:"logs_dynamic_id"` | |
LogsDynamicID bool `mapstructure:"logs_dynamic_id"` |
nit: This has nothing to do with Index, which makes the use of DynamicIndexSetting
slightly odd.
Also, personally I find this 1 level of nesting not very ergonomic. Unless we plan to add features to dynamic ID in the future e.g. logs_dynamic_id::attribute_name
to make the attribute name configurable, I feel like adding a layer just to have a ::enabled
bool not very useful. You can tell I'm already not very happy with *_dynamic_index::enabled
. But I'd be interested to know other codeowners' opinion on this.
Co-authored-by: Carson Ip <[email protected]>
Co-authored-by: Carson Ip <[email protected]>
Description
This PR adds a new config option
logs_dynamic_id
that when set to true reads theelasticsearch.document_id
attribute from each log record and uses it as the final document id in Elasticsearch. This is only implemented for logs but I can open subsequent PRs supporting metrics and traces akin to the*_dynamic_index
options.Fixes #36882
Testing
Added tests to verify that the document ID attribute can be read from the log record and that the _id is properly forwarded to Elasticsearch. Also asserted that when there is no doc id attribute the current behavior is retained.
Documentation
Updated the readme to mention the new
logs_dynamic_id
config option.