[Heartbeat] Cross linking APM with Synthetics

[vigneshshanmugam]

Describe the enhancement:

As part of the Crosslinking effort in associating the trace from the Synthetics waterfall to the relevant APM data and vice versa, we need to inject specific information from the Heartbeat side to the underlying Synthetics agent to propagate the information to the downstream services

Meta issue - https://github.com/elastic/apm-dev/issues/1024

Implementation details

To be able to associate a synthetics trace to the APM trace details and also make this account for the future proofing needs, We need to propagate the following information.

1. ELASTIC_SYNTHETICS_MONITOR_ID - monitor id of the synthetics monitor
2. ELASTIC_SYNTHETICS_TRACE_ID - checkgroup/exec that begins the synthetics journey, distinct for HTTP monitors
3. ELASTIC_SYNTHETICS_MONITOR_LOCATION - location where the synthetics monitor is run from
4. ELASTIC_SYNTHETICS_MONITOR_TYPE - type of the synthetics monitor (browser, http, tcp, etc)
5. Sampling decision - Transaction sampling rate with default being all are sampled.

Relevant PR on the synthetics agent side for browser monitors - elastic/synthetics#595

HTTP monitors

• Heartbeat would do the heavy lifting for the HTTP based monitors and add the baggage and tracestate headers as the synthetics agent is not involved for these tests.

Config Options

• trace_urls - urls where relevant headers would be propagated to the downstream services via the baggage header .
• sample_rate - sampling decision that will be propagated via the tracestate header.

heartbeat.monitors: 
  - type: lightweight
     apm:
        sample_rate: 1 //ranges from [0,1] 

  - type: browser
     apm:
        trace_urls: ["**/*", "api/**"] // glob or RegExp patterns
        sample_rate: 0.5

NOTE: It will be APM agents responsibility to add these baggage and tracestate information to all the transactions and error data which will be indexed to ES by the APM server.

Relevant Issues on APM

• Baggage Header - https://github.com/elastic/apm-dev/issues/994
• Tracestate Spec - https://github.com/elastic/apm/blob/main/specs/agents/tracing-distributed-tracing.md#tracestate

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!