Metrics being lost

[ejsmith]

I have my .NET app completely configured to send traces, logs and metrics using otel protocol to NewRelic's otel endpoint. My logs and traces are there and a lot of the metrics are showing up too, but I have some histograms in my app that just don't show in in NewRelic. Does anyone know if this is a current limitation due to the protocol not being stabilized?

[ejsmith]

Also, does anyone know of like an in memory otel server that would make verifying the app is configured correctly and just show maybe like last X number of logs, traces and metrics?

[cijothomas]

The idea with "ZPages" was that..
Also there is "InMemoryExporter".
Or you can use PrometheusExporter, which will make metrics available at a local endpoint, which you can query from browser to see metrics...

[ejsmith]

I'm surprised someone hasn't made an otel server with some html views that show the most recent data and doesn't require a database or anything else. Seems like it would be useful.

[cijothomas]

Thats "Zpages". https://github.com/open-telemetry/opentelemetry-specification/blob/main/experimental/trace/zpages.md
Note the word "experimental" in zpages spec..

[alanwest]

My logs and traces are there and a lot of the metrics are showing up too, but I have some histograms in my app that just don't show in in NewRelic.

Currently, New Relic does not support Histograms aggregated using cumulative temporality. The SDK defaults to cumulative temporality. To make things work with New Relic, you can configure the SDK to use delta temporality like so:

.AddOtlpExporter((exporterOptions, metricReaderOptions) =>
{
    // Note that the means by which you perform this configuration may change a little
    // by the time .NET releases a stable metric SDK.
    metricReaderOptions.Temporality = AggregationTemporality.Delta;
});

You can look for any ingest errors with New Relic with the following query:

FROM NrIntegrationError SELECT *

If cumulative aggregation temporality turns out to be your issue, you'll see errors that indicate this.

[ejsmith]

Thank you!

[Aaronontheweb]

DataDog has this issue as well, I believe.

[ejsmith]

Now I am reporting directly to Elasticsearch APM and I am missing these metrics again. Guess I'll have to try changing that Temporality setting back to default and see if they like it that way.