exporter endpoint svc to check for gpu health

[spraveenio]

No description provided.

[y2kenny-amd]

GPU health is core part of the plugin and therefore it should not require an external dependency to the exporter in order to function (a standalone install of the plugin must continue to work.) Since the plugin has direct access to the GPU like the labeller, health status should be read directly from the GPU (or via relevant library like libsmi) instead of via rpc.

[sajmera-pensando]

GPU health is core part of the plugin and therefore it should not require an external dependency to the exporter in order to function (a standalone install of the plugin must continue to work.) Since the plugin has direct access to the GPU like the labeller, health status should be read directly from the GPU (or via relevant library like libsmi) instead of via rpc.

Thanks @y2kenny-amd for the review. With this PR as well, a standalone install of the plugin will keep working. If the health information from device-metrics-exporter is not available, then device plugin will fall back to the existing basic health checking functionality.

K8s device-plugin's documentation does not say that device-plugin has to calculate the health. It just provides the API to mark the device as unhealthy. The health calculation could happen in any external component which has access to the data needed to find out whether the GPU is healthy or not. We are planning to implement the health calculation functionality in the device-metrics-exporter because it has access to all the metrics of the GPUs, but theoretically it could also happen in any external analytics systems like a Prometheus time-series database that has access to all the gpu metrics and can predict gpu health based on current metrics.

[spraveenio]

GPU health is core part of the plugin and therefore it should not require an external dependency to the exporter in order to function (a standalone install of the plugin must continue to work.) Since the plugin has direct access to the GPU like the labeller, health status should be read directly from the GPU (or via relevant library like libsmi) instead of via rpc.

Yes stand alone keeps working without any change in behavior. This extends when there is metrics exporter is running on the node. To mark each and every gpu healthy or unhealthy.