Develop strategy for asynchronous Resources and Entities

[dyladan]

This comes from JS, but may affect other language which either cannot or don't want to block startup. In JS for example, it is impossible to block for a web request.

While working on the entity prototype an issue surfaced in JS which has been around in resource for a while. Some resources require asynchronous work to collect such as reading from the filesystem or making a web request to a platform provider API like AWS or kubernetes. Because JS can't block, these resources may not be resolved when the first spans are started. In the past, JS has solved this by allowing some attributes to be asynchronous (represented as a promise), and it is the responsibility of each exporter to await them. After the first export, the issue is resolved because all attributes are now resolved and Resource is never changed. This affects web requests the most by far. File system APIs have synchronous versions in NodeJS which most resource detectors use. One particularly weird example is the AWS Lambda function ID, which isn't actually acquired from an API but comes in with the first request.

This works fine, except that SpanProcessors don't have access to any attributes which are not yet resolved before the first export. The SpanProcessor spec states specifically that its methods are called synchronously with span start and end. This is important to avoid race conditions (if a span processor was async and modified the span, the instrumentation's synchronous modifications after span start would be lost). That means that the span processor can't simply await the resource attributes. Additionally, with Entities now being thrown into the mix it is possible that this could now happen many times over the lifetime of a process.

This was fine when there was only a single resource, but with entities we now need to be able to compare attributes to determine if two entities have the same identity. Async attributes makes this impossible because you can't know if { id: abc } is the same as { id: Promise<string> } until the promise resolves in the export pipeline.

In the current entity prototype, I sidestepped this a bit by disallowing async identifying attributes in entities. Descriptive attributes are fine because the priority has to do with detection order, not resolved value. I think we should discuss this further, at least in the entity SIG, but it would be nice to solve this generally as well.

Potential Solutions

Some of these depend on in-process spec or OTEPs

Update Resource after resolution (depends: ResourceProvider)

1. Create resource/entities with only sync fields
2. When an entity or resource resolves, the ResourceProvider will provide a new resource

This has the potential for telemetry emitted before the resolution to miss a resource or entity. In some cases (like lambda) this may be the only export.

Allow async descriptive attributes

Because these attribute values don't need to be compared in the SDK, they can be safely asynchronous. Identifying attributes in this case MUST be synchronous. This would disallow identifying attributes from at least web requests. It would raise the question of what async/sync attributes mean, because in JS you can block on filesystem but in some other runtime/language you may not be able to.