[META] Enable core dynamic configuration capabilities

[jackshirazi]

This is mainly for tracking, the intention is to enable these in the Otel agent itself rather than in our distribution. The list of features we'd like to be dynamically adjustable are:

• instrumentation (individually enabled/disabled)
• log level
• enabling/disabling the agent
• sending traces, metrics
• sampling percentage

[jackshirazi]

instrumentation (individually enabled/disabled)

This has been discussed in the Otel SIG with 4 approaches: suppression, sampling, re-configuring, retransforming

• Retransforming is done in the elastic agent, but is currently considered highly difficult in the otel agent
• Re-configuring is a long term goal (dynamically recreating the whole agent config and applying that) but would be a huge amount of work, so is not in scope here
• Sampling could work (there is already a jaeger dynamic sampler), but the instrumentation scope is not available to the sampler so it's not straightforward to do this on a per-instrumentation basis
• Suppression is being looked at

[jackshirazi]

enabling/disabling the agent

Note that the agent being enabled is not dynamic in the Elastic agent, and the equivalent OTEL_JAVAAGENT_ENABLED non-dynamic setting is already available. What this actually refers to is recording. Since the instrumentation enabling/disabling is likely to be suppression rather than instrumentation removal, recording on/off in this situation is equivalent to disabling all instrumentation and metrics, so it's dependent on the suppression for part of the feature. A simpler approach is to set sampling to 0, but this also doesn't target metrics, and would leave the agent doing a lot more work than turning all instrumentation off. Metric disablement needs to be considered

[jackshirazi]

sending traces, metrics

This is leaving the agent and instrumentation enabled, but not sending anything to the server (eg to allow log correlation, or trace propagation to continue, but with no data stored for this service). This should be easily done in the exporter?

[jackshirazi]

sampling percentage

Available as an extension, but we should customize/do our own implementation and add it as our sampler?

[jackshirazi]

Closing this, next steps covered in #300