Create an AppDynamics plugin

[EmrysMyrddin]

Context

AppDynamics is a Cisco technology for monitoring, trancing and error reporting.

It has automated Node instrumentation, and works with most used packages. In particular, it offers instrumentation of the graphql library. But since Mesh doesn't rely on it, it doesn't work automatically. We have to make a plugin to integrate with it.

Solutions

I have identified 2 ways we could integrate with AppDynamics:

• use our OTEL plugin and configure AppDynamics to injest OTEL metrics
• implement a custom plugin using AppDynamics Node API.

OTEL integration

This solution would have the huge advantage to require less work on our end, both in upfront development time and maintenance.

Based what is claimed in the documentation, it is supposed to "Just Work™️" after enabling the OTEL integration in the options. It is supposed to then correlate OTEL metrics and AppDynamics specific data and merge everything correctly.

The main drawback of this is that it requires to run an additional component. Users will have to setup an OTEL Collector, and configure it to export metrics to AppDynamics.

While the approach is a good idea and perhaps something we could explore afterward, I think the drawback of requiring a running OTEL Collector is too much to ask. Users going for this solution are probably doing so for it's ease of setup. And it can be difficult to meet this requirement in some env, like if the gateway is on the Edge.

Custom plugin using Node agent API

The second solution is to create a plugin from scratch, relying on AppDynamics Node Agent API.

From my understanding, there is 2 main concepts that are important for us:

• Transactions: can be started manually (with an optional correlation information), represent an execution span.
  • It seems we can attach some data to it using "snapshots", not sure what it is yet.
  • Seems a bit tricky to use, will probably require a lot of testing. Namely, it seems you have to manually "re-join" a transaction each time you cross async boundaries.
• Exit Call: represents what they name "downstream" requests.
  • It can be whatever external services
  • It can be of any kind (HTTP, Database, ...)
  • That's how we will have to track our subgraph queries
  • Since we rely on node:http and node:https for our requests, it is supposed to be automatically handled
  • Otherwise, we will have to manually handle the creation of correlation information and propagate them through headers

The documentation is very sparse, so it will probably require a lot of trial and error to make it work.

They do not provide any testing tool, so it will probably be very hard to test this plugin.

They do not provide any types, we will have to write our own. Perhaps we could then contribute them back to the DefinitelyTyped repo.

Propagation of transactions

In AppDynamics, correlation and propagation is supposed to be automatic. They instrument most used libraries to automatically add correlation info to outgoing calls, and detect them in ingoing calls (like HTTP requests).

We will probably have to manually check for those correlation information in a Yoga plugin so that transactions are correctly propagated in subgraphs if they are using Yoga.

Most of the work should be done automatically, it's just a matter of finding out how correlation data is transmitted (which HTTP header is used).

For other subgraph server library, they will have to either be automatically handled by AppDynamics (like graphql-http) or have a dedicated integration.

[EmrysMyrddin]

After creating a POC, it appears the best solution today would be to make OTEL integration work with AppDynamics.

Creating an integration from scratch seems complex and would require relying on AppDynamics Node SDK internals, which would be very fragile over time and difficult to maintain.

As of today, I didn't managed to have AppDynamics injesting OTEL spans. An example repository can be found here: https://github.com/the-guild-org/appdynamics-example

This repository contains a working Hive Gateway with a basic OTEL setup sending spans to AppDynamics. We can see in the logs of the OTEL Collector that spans are received from the Hive Gateway instance, are well processed with all required AppDynamics data. So it's not clear why AppDynamics doesn't injest those spans.