perf(telemetry): batch telemetry events into one request

[mabdinur]

Previously, ddtrace sent a separate synchronous request for each telemetry event (e.g., app-started, config-changed, dependencies-loaded). In the worst case, we would have sent over 16 requests per minute (6 log events, 6 metric events, 1 configuration event, 1 app started event, etc.) but the typical number was 6–10 per minute.

With this change we will batch all telemetry events queued in the telemetry interval (default: 60 seconds) into on e request using the message-batch request type. This will reduce overhead by sending instrumentation telemetry events.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

benchmarks/ddtrace_run/scenario.py                                      @DataDog/apm-core-python
benchmarks/startup/config.yaml                                          @DataDog/apm-core-python
ddtrace/internal/telemetry/writer.py                                    @DataDog/apm-core-python
tests/conftest.py                                                       @DataDog/apm-core-python
tests/telemetry/test_telemetry.py                                       @DataDog/apm-python
tests/telemetry/test_writer.py                                          @DataDog/apm-python

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 237 ± 3 ms.

The average import time from base is: 239 ± 3 ms.

The import time difference between this PR and base is: -2.2 ± 0.1 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.975 ms (0.83%)

ddtrace.bootstrap.sitecustomize 1.305 ms (0.55%)

ddtrace.bootstrap.preload 1.305 ms (0.55%)

ddtrace.internal.products 1.305 ms (0.55%)

ddtrace.internal.remoteconfig.client 0.656 ms (0.28%)

ddtrace 0.670 ms (0.28%)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-05-09 17:38:19

Comparing candidate commit 8a2456a in PR branch munir/batch-all-telemetry-requests with baseline commit 8fa23a4 in branch main.

Found 2 performance improvements and 1 performance regressions! Performance is the same for 425 metrics, 6 unstable metrics.

scenario:iast_aspects-ospathsplitdrive_aspect

• 🟥 execution_time [+554.523ns; +613.524ns] or [+15.335%; +16.967%]

scenario:telemetryaddmetric-flush-100-metrics

• 🟩 execution_time [-22.531µs; -20.048µs] or [-11.158%; -9.928%]

scenario:telemetryaddmetric-flush-1000-metrics

• 🟩 execution_time [-242.363µs; -218.756µs] or [-10.082%; -9.100%]

[mabdinur]

Blocked by: #13391