[SDK] Better control of threads executed by opentelemetry-cpp

[marcalff]

Fixes #3174

Changes

This feature provides a way for applications, when configuring the SDK and exporters,
to participate in the execution path of internal opentelemetry-cpp threads.

The opentelemetry-cpp library provides the following:

• a new ThreadInstrumentation interface,
• new runtime options structures, to optionally configure the SDK:
  • BatchSpanProcessorRuntimeOptions
  • PeriodicExportingMetricReaderRuntimeOptions
  • BatchLogRecordProcessorRuntimeOptions
• new runtime options structures, to optionally configure the OTLP HTTP exporters:
  • OtlpHttpExporterRuntimeOptions
  • OtlpHttpMetricExporterRuntimeOptions
  • OtlpHttpLogRecordExporterRuntimeOptions
• new ThreadInstrumentation parameters, to optionally configure the CURL HttpClient
• new runtime options structures, to optionally configure the OTLP FILE exporters:
  • OtlpFileExporterRuntimeOptions
  • OtlpFileMetricExporterRuntimeOptions
  • OtlpFileLogRecordExporterRuntimeOptions
• new runtime options structure, to optionally configure the OTLP FILE client:
  • OtlpFileClientRuntimeOptions

Using the optional runtime options structures,
an application can subclass the ThreadInstrumentation interface,
and be notified of specific events of interest during the execution of an internal opentelemetry-cpp thread.

This allows an application to call, for example:

• pthread_setaffinity_np(), see related Add BatchSpanProcessor option to set CPU affinity on the worker thread #1822
• setns(), to control the network namespace used by HTTP CURL connections
• pthread_setname_np(), for better observability from the operating system
• many more specific apis, as needed

See the documentation for ThreadInstrumentation for details.

A new example program, example_otlp_instrumented_http, shows how to use the feature,
and add application logic in the thread execution code path.

For significant contributions please make sure you have completed the following items:

• CHANGELOG.md updated for non-trivial changes
• Unit tests have been added
• Changes in public API reviewed

[netlify]

✅ Deploy Preview for opentelemetry-cpp-api-docs canceled.

Name	Link
🔨 Latest commit	3b6699f
🔍 Latest deploy log	https://app.netlify.com/sites/opentelemetry-cpp-api-docs/deploys/6764598559d5a500085e0cd3

[codecov]

Codecov Report

Attention: Patch coverage is 38.09524% with 52 lines in your changes missing coverage. Please review.

Project coverage is 87.48%. Comparing base (4998eb1) to head (3b6699f).

Files with missing lines	Patch %	Lines
sdk/src/logs/batch_log_record_processor.cc	30.77%	18 Missing ⚠️
...metrics/export/periodic_exporting_metric_reader.cc	43.75%	18 Missing ⚠️
sdk/src/trace/batch_span_processor.cc	38.47%	16 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3175      +/-   ##
==========================================
- Coverage   88.16%   87.48%   -0.68%     
==========================================
  Files         198      198              
  Lines        6224     6298      +74     
==========================================
+ Hits         5487     5509      +22     
- Misses        737      789      +52     

Files with missing lines	Coverage Δ
...pentelemetry/sdk/logs/batch_log_record_processor.h	100.00% <ø> (ø)
...ude/opentelemetry/sdk/trace/batch_span_processor.h	100.00% <ø> (ø)
sdk/src/trace/batch_span_processor.cc	85.00% <38.47%> (-9.11%)	⬇️
sdk/src/logs/batch_log_record_processor.cc	80.87% <30.77%> (-8.42%)	⬇️
...metrics/export/periodic_exporting_metric_reader.cc	70.46% <43.75%> (-11.27%)	⬇️

[owent]

Do you think we can implement a worker pool to share threads among multiple exporters and processors? Since most of them simply wait for a timeout, there is no need to create a separate thread for each component.

[marcalff]

Do you think we can implement a worker pool to share threads among multiple exporters and processors? Since most of them simply wait for a timeout, there is no need to create a separate thread for each component.

For some applications, running in a multi threaded environment, it is critical to control how a thread executes in finer details.

For example, I have:

• (a) an OTLP HTTP trace exporter
• (b) several OTLP HTTP metric exporters
• (c) an OTLP HTTP log exporter

When the exporter uses CURL to post to the endpoint, it makes a TCP/IP connection.

(a), (b) and (c) do need to use different TCP/IP stacks, or different named networks, to connect to their respective endpoints.

This is achieved by calling setns(fd, CLONE_NEWNET) in each thread, where fd is a file descriptor describing the named network to use.

Having dedicated threads (as currently) makes this easier.

If execution of each exporter was to be multiplexed and executed in the same worker thread, the context will need to change for each exporter, calling setns() many more time, and introducing more complexity to counter the effect of a pool worker thread, so it won't help but instead get in the way.

setns() is just an example, there can be other use cases, like binding to a CPU, etc.

Overall, the need is for the main application to inject arbitrary code in the opentelemetry-cpp execution code path, for opentelemetry threads.

[owent]

Could please also add this instrumentation for OTLP file exporters?

[marcalff]

Could please also add this instrumentation for OTLP file exporters?

Sure, implemented.

Every place where opentelemetry-cpp starts a new std::thread is now instrumented.

[marcalff]

An example of using this feature, to set thread names to the operating system on Linux:

ps -Lp 199922 -o pid,tid,comm

    PID     TID COMMAND
 199922  199922 (redacted)
 ...
 199922  201152 otel_bsp-1
 199922  201153 otel_pmr-1
 199922  201154 otel_pmr-2
 199922  201157 otel_blrp-1
 199922  201163 otlp_traces
 199922  201165 otlp_logs
 199922  201179 otlp_metrics

In order, threads are:

• the application main thread
• the batch span processor
• the periodic exporting meter reader (2 instances)
• the batch log record processor
• a trace otlp http curl thread
• a logs otlp http curl thread
• a metrics otlp http curl thread

[owent]

LGTM and thanks.