[APM ]APM "Services > Metrics" Tab doesn't work if embedded dashboard includes ES|QL visualizations

[JonasKunz]

Kibana version: Tested on 9.1.0

Elasticsearch version: 9.1.0

Describe the bug:

I've been working on adding a new, Otel-native dashboard for Java for the Services > Metrics observability tab-
Due to requiring multi-step aggregations, I chose to use ES|QL for some of the panels.

Unfortunately it looks like the metrics tab can't deal with dashboards containing ES|QL visualizations. When going to the tab, the dashboard remains empty and the following error shows up:

If I remove all ES|QL visualizations from the dashboard, the error disappears and the dashboard loads.

At the same time importing the dashboard with the ES|QL visualizations and linking it from the Service > Dashboards tab works without problems.

Steps to reproduce:

Add the following file as embedded dashboard for the metrics tab:

otel_native-edot-java.json

Ingest some otel data and navigate to the tab metrics tab of the service.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[jennypavlova]

To render the dashboards, we use the DashboardRenderer here. The dashboard files are in the dashboards folder.

Comparing to the usage in the Dashboards tab in case of the metrics we have the logic to map the layers here. Then comparing to the other dashboards we have textBased ({"datasourceStates\":{"textBased":{"layers":,.......) in the java dashboard file linked and not formBased like for example in the otel_native-edot-python.json file - I am not familiar with the dashboard content: @JonasKunz Is that something we can change to match the previous format and try again? Or is this somehow related to the ES|QL visualizations?

Edit Some follow-up questions: What is the difference between textBased and formBased in those dashboards context? Can we have both at the same time, or is it always one of those?
As a solution from our side, we can add a fallback to check for textBased in case formBased is not present in the datasourceStates, which should fix the issue, but I am trying to understand the context here first

[JonasKunz]

I'm not aware of the underlying JSON schema. The dashboard was simply built via kibana and then exported via the "Saved Objects" settings. However, when removing the ES|QL visualizations from the dashboard everything works. So my guess is that the presence of those changes the serialization somehow?

I followed this internal guide for updating the dashboard.

[jennypavlova]

The dashboard was simply built via kibana and then exported via the "Saved Objects" settings.

@JonasKunz thanks for answering! My guess here is that those textBased and formBased are both valid cases and the second one is used when ES|QL visualization is used - that's what I am trying to understand, and as I mentioned, I am not familiar with the exported dashboards; that's why I asked 😅

So my guess is that the presence of those changes the serialization somehow?

I think the issue is in the logic to map the layers here we have and It probably worked fine before because we didn't have ES|QL visualizations. So what I will try now is to verify if the fix I have in mind solves this issue using the file you attached.

[jennypavlova]

@JonasKunz I opened a draft: Can you please check if it works for the service you instrumented?
I have only a remote cluster with OTel data atm (no EDOT, unfortunately), and when I changed the index (because I saw an error that the index was not available), I was able to see the first visualization with some twicks

I can try to set up an env with java edot next week to check it as well. If you have specific requirements on how to do it, please let me know.