Proposal Draft : Utilize the Blockless Mesh Network to Monitor the Pod Health of Akash Providers

[dmikey]

Seeking: $7,500 (USD)
Timeline: 4 Weeks
Result: Publicly Index Data on The Graph for Akash Provider Health History, with in depth results of actual Pod Health.

Summary

The Blockless Network is in a unique position to leverage its highly scalable Node Network that can seamlessly deploy into Akash Network with a small foot print. It can benchmark and monitor pod health, and report back to the network that benchmarks succeeded. In addition, since Blockless is by nature multi-senate, the compute will not be idle and empty compute can be booked when the benchmark is not running to measure Node Health.

Why is this needed?

While deploying across Akash's various providers, we've had a daunting task to homogenize the network, to fit within the topology where the deployment system can vary widely even in base response. From specific environment attributes (CPU type, missing instruction sets) to various performance impact (Providers that are booked with Miners, do not perform well for general purpose leasing in a multi-tenet situation)

What determines Provider Health

There are a number of ways to gauge provider health. This definition should be fluid enough to evolve over time. One thing is for sure, that a number of different qualities need to be enumerated.

• Avg Response Time
• Avg Connection Speeds
• Avg Execution Times
• Storage or Memory Performance (writing and reading speeds)

The tests should indicate potential real world performance, when an application is deployed to a POD so all health will be reported as a tenet of the provider.

Provider Health will be a relative ranking system across all providers. There maybe ties based on the matrix, but potential to always have a clear "Best" performer.

Why is Blockless Uniquely positioned to Accomplish this?

Blockless is at a maturity state, where its scaling and health reporting can be immediately impactful for Akash. We feel that we need to have this kind of visibility in the underlying network to identify, respond, and scale with these issues. By building a provider health application, we can increase visibility into the performance and impact that providers have with providing quality compute - without wasting compute cycle to synthetic benchmarks, Blockless can use real compute scenarios, to respond back with health quality.

What Does Funding Look Like?

We are seeking $7,500 (USD) to develop this solution. It can be completely opt in by operators simply deploying a Blockless Pod into their Akash setup. They can earn while providing FaaS infrastructure while participating in Provider Health.

What Does Timeline Look Like?

We are looking at building and deploying the enhancements needed to enumerate the provider tracking across the network in 4 weeks.

Week 1.

We will build the Blockless FaaS that will gather pod health and execute with a Blockless trigger.

Week 2.

We will enhance our node software to gather metrics around real world scenarios for health reporting, and reference those to providers through annotation enhancements.

Week 3.

We will deploy enhanced nodes using the funding in the draft, to scale nodes across the Akash network.

Week 4.

We will release a UI that infers a leaderboard style to rank and distribute quality reporting back for providers.

Outcome

This solution will be released Open Source. Blockless Network is built on WASM technology and can operate in any WASI compatible environment. The data will be indexed to The Graph and available to all who want to incorporate it into Akash Projects.

[shimpa1]

hi!
I love the idea of homogenizing provider qualities! I believe this will bring a lot of value to the network as currently there is no way for a tenant to have an idea which provider offers what in exchange for their AKT.
These results can potentially be used as a provider score for other projects.

I do have a few questions:

1. Do you plan on integrating the results that the Monitor Pod has gathered into the Akash Console? This would make a lot of sense, so that the tenants can see the provider score before accepting bids and/or browse providers, similar to what Cloudmos currenty offers but in more detail.
2. What metrics do you plan on measuring? (CPU score, GPU score, ephemeral storage score, persistent storage score, internet access score, provider uptime, pod uptime, other metrics?
3. Please define "provider health" - does it mean pod uptime, provider uptime, provider performance, reliability or something else? Please provide more detail what will the result look like once everything has been deployed.

thank you!
Shimpa | Europlots

[dmikey]

Thanks for taking time to read! I've update the main proposal body to account for these questions, but to answer directly.

1. We'll leave that up to the community how the Data should be used. We'll index and publish it to "The Graph" so it has availability for any and all who want to use it.

2. Metrics will vary across different use cases to test real use scenarios, and generate a health score from that. How well does the POD perform, and how well as a tenet can I expect my dApp to perform on a specific provider.

3. Provider health will be a grading from 0 to 1. A provider may tie for a graded position. 0.70 as an example, but we can rank them in relative performance, while also recording the results for that.

I expect that this ranking and benchmarker will evolve over time! Hope this answers those questions. Thanks so much!

[cloud-j-luna]

Hey!

1. Is it possible to provide more detail (technical if possible) in each phase?
2. Could you provide more details in how you've reached the $7,500?
3. Have you considered making this part of the Akash Core protocol? As a use of the community fund I think it would be great to have these metrics being gathered by an Akash Network owned service and exported to different kinds of services such as Prometheus. Data could still be indexed to The Graph but the metrics exporting could be done at the Akash Core. There are tools that already do this and are open sourced. I would rather fund this as a new feature on the Akash Core protocol than an external project.

Thank you!

[anilmurty]

Thanks for writing this up @dmikey ! - very interesting and definitely useful -- but I think this can be teased apart a bit and also need more detail:

1. Let's talk about health metrics first. As you pointed out, there are multiple facets to provider health metrics:
   a. Metrics that provider need in order to be able to manage themselves and to provide SLAs/ SLOs (call these "provider monitoring metrics")
   b. Metrics that providers need to determine if they are or can be profitable (call these "provider profitability metrics")
   c. Metrics that tenants would like, to make a decision about whether to choose one provider or another (call these "provider uptime and tenant matching" metrics)
   d. Metrics that a third party (like a messari analyst) would like, to determine the overall health of Akash Network (of which provider growth and retention are one half - the other half being demand in terms of deployments). (Call this "provider growth metrics")

Are you looking to tackle all 4 (1a - 1d) or some subset?
Logistically: Couple of these things are being tackled with the content-moderation, provider-monitoring and analytics sigs and workgroups and could fit into those. We would need a lot of detail to be fleshed out (as we've done with the existing content-mod and analytics efforts that are in progress - check these PRDs:
https://github.com/akash-network/community/blob/main/wg-content-moderation/prd.md
https://github.com/akash-network/community/blob/main/sig-analytics/prd.md

2. The second portion of this is the technical details: Specifically, I'm a little unclear about how you will be able to get pod level metrics by looking at on-chain data. Is this going to be be achieved using an agent-based solution that would read the pod-level metrics and store the on-chain for querying later? Or something else?
   Separately, the idea of homogenizing (aka trying to put it in a similar format so it can be compared and potentially a single score can be assigned to a provider) requires that we first have a consistent way that provider attributes are set, audited and reported. This has been discussed and is something we are hoping to kick off as part of the "wg-provider-attributes" workgroup (https://github.com/akash-network/community/tree/main/wg-provider-attributes) -- I feel like that is a necessary prerequisite before we can truly homogenize and score providers.

[bmenzalji]

Hey @dmikey

warmly checking in on an update

[dmikey]

Thanks, I've been on the road the last month. Will circle back here with more!