chore(benchmarks): combine ddtrace-run and startup benchmarks

[mabdinur]

ddtrace-run benchmarks are broken, this PR removes this benchmarking scenario and combines it with the new startup scenario.

Issues with ddtrace-run benchmarks:

• telemetry variants do not enable/disable the generation of telemetry events. To record a telemetry event on application start up _DD_INSTRUMENTATION_TELEMETRY_TESTS_FORCE_APP_STARTED must be set to true. Otherwise telemetry events will be recorded 10 seconds after ddtrace instrumentation is applied
• http proxy does not mock tracer-agent connections. We set the trace api version to v0.4 but then mock requests to the v0.5 endpoint. This does not make sense.
• ddtrace-run benchmarks do not mock statsd connections. So the baseline that attempts to disable all http/udp/unix connections is inaccurate

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

benchmarks/startup/config.yaml                                          @DataDog/apm-core-python
benchmarks/startup/scenario.py                                          @DataDog/apm-core-python

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 232 ± 2 ms.

The average import time from base is: 237 ± 4 ms.

The import time difference between this PR and base is: -4.6 ± 0.1 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 2.292 ms (0.99%)

ddtrace.bootstrap.sitecustomize 1.564 ms (0.67%)

ddtrace.bootstrap.preload 1.564 ms (0.67%)

ddtrace.internal.products 1.564 ms (0.67%)

ddtrace.internal.remoteconfig.client 0.700 ms (0.30%)

ddtrace 0.727 ms (0.31%)

ddtrace._logger 0.054 ms (0.02%)

logging 0.054 ms (0.02%)

traceback 0.054 ms (0.02%)

contextlib 0.054 ms (0.02%)

[datadog-datadog-prod-us1]

Datadog Summary

✅ Code Quality    ✅ Code Security    ✅ Dependencies    ❌ Pipelines

Next Steps

The following jobs failed due to code errors:

System Tests / system-tests (django-py3.13, other)

1 failed test. AssertionError: Invalid IAST data for span: {'sources': [{'origin': 'http.request.parameter', 'name': 'url', 'redacted': True, 'pattern': 'abcdefghijklmnopqrstuvwxyzABCDE'}], 'vulnerabilities': [{'type': 'SSRF', 'evidence': {'valueParts': [{'source': 0, 'value': 'http://weblog:7777/?token='}, {'redacted': True, 'source': 0, 'pattern': 'ABCDE'}]}, 'location': {'spanId': 2256514972871161777, 'path': 'app/urls.py', 'line': 712, 'method': 'make_distant_call', 'class': ''}, 'hash': 1795483706, 'stackId': '1'}]}

System Tests / system-tests (fastapi, other)

1 failed test. Assertion Error: Invalid IAST data for span: {'sources': [{'origin': 'http.request.parameter', 'name': 'url', 'redacted': True, 'pattern': 'abcdefghijklmnopqrstuvwxyzABCDE'}], 'vulnerabilities': [{'type': 'SSRF', 'evidence': {'valueParts': [{'source': 0, 'value': 'http://weblog:7777/?token='}, {'redacted': True, 'source': 0, 'pattern': 'ABCDE'}]}, 'location': {'spanId': 3295642735528587381, 'path': 'main.py', 'line': 368, 'method': 'make_distant_call', 'class': ''}, 'hash': 2261779420, 'stackId': '1'}]}

System Tests / system-tests (uwsgi-poc, other)

1 failed test. AssertionError: Invalid IAST data for span: {'sources': [{'origin': 'http.request.parameter', 'name': 'url', 'redacted': True, 'pattern': 'abcdefghijklmnopqrstuvwxyzABCDE'}], 'vulnerabilities': [{'type': 'SSRF', 'evidence': {'valueParts': [{'source': 0, 'value': 'http://weblog:7777/?token='}, {'redacted': True, 'source': 0, 'pattern': 'ABCDE'}]}, 'location': {'spanId': 11476412351368578441, 'path': 'app.py', 'line': 518, 'method': 'make_distant_call', 'class': ''}, 'hash': 3581058908, 'stackId': '1'}]}

... and 4 more. View all.

The following jobs failed due to platform errors:

Require Checklist / require-checklist

Incomplete task checklist: PR author and reviewer criteria not completed as per requirements.

---

Was this helpful? Give us feedback!

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-05-12 18:11:31

Comparing candidate commit 4c9f7dd in PR branch munir/fix-ddtrace-run-benchmarks with baseline commit 53c50e8 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 388 metrics, 5 unstable metrics.

scenario:iast_aspects-ospathdirname_aspect

• 🟥 execution_time [+740.862ns; +811.509ns] or [+15.977%; +17.501%]