Enable OpenTelemetry support

[asalkeld]

Support opentelemetry.io

This is an industry standard and is supported by all metric/tracing vendors.

Use cases

1. none - tracing can hit performance, you might just need it off.
2. enable only 'auto' metrics from the membrane
3. enable 'auto' and custom metrics/traces
4. optionally create a cloud native telemetry resource (cloudwatch log group or similar)

Allow access to all collectors

https://opentelemetry.io/registry/?language=collector&component=exporter

Setup auto tracing/metrics

1. the tracing will be default off

2. auto tracing

in the stack.yaml

config:
  funcX:
    telemetry: <path-to opentelemetry config>

what this will do:

• insert the collector into the function container
• configure the collector with the above config
• create a default traceProvider
• all membrane calls will be instrumented

3. enable 'auto' and custom metrics/traces

The only required sdk part below is resources.NewTracerProvider() which initializes
the correct exporter and resource.

	ctx := context.Background()

	tp, err := resources.NewTracerProvider(ctx)
	if err != nil {
		fmt.Printf("error creating tracer provider: %v", err)
	}

	defer func(ctx context.Context) {
		err := tp.Shutdown(ctx)
		if err != nil {
			fmt.Printf("error shutting down tracer provider: %v", err)
		}
	}(ctx)

	otel.SetTracerProvider(tp)
  // the framework will then use otel.GetTraceProvider() for it's tracing in case the user has done the above.

  // custom tracing
  newCtx, span := otel.Tracer("work").Start(ctx, "Run")
  work(newCtx)
  span.End()

4. optionally create a cloud native telemetry resource (cloudwatch LogGroup or similar)

resources.NewNativeTelemetry("name")

If this is used, then there is no need to configure the stack.yaml as this will be populated with the results
of creating the resource.

open questions

The opentelemetry config is quite big. it would be nice to not have to specify all of it.
The part we really need provided is the exporters, see: https://opentelemetry.io/docs/collector/configuration/

[asalkeld]

some references:

• https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.61.0/exporter/awscloudwatchlogsexporter
• https://opentelemetry.io/docs/collector/configuration/
• https://github.com/open-telemetry/opentelemetry-go/blob/main/example/otel-collector/main.go