Akash Network GPU Provider Incentives

[lechenghiskhan]

This in a update to the original proposal based on community feedback, posted on January 19th, 2023

Introduction

Akash is unable to attract new tenants due to a critical shortage of high-density GPUs, namely A100s.

A closer look at the data, however, shows that 91% of A100 GPUs are fully utilized as of block 14622914. This means no new tenants will find any A100 inventory suitable for meaningful workloads. Alleviating this supply constraint is critical for continued growth.

Qualitative feedback suggests we have several immediate use cases for these powerful, high-density GPUs:

1. Training and fine-tuning AI workloads like the Akash X Thumper foundational model,
2. Various AI inference applications across high-growth companies, and
3. Virtual world rendering and AAA-game hosting with Passage

As proposed in AKT 2.0, which received overwhelming support from the community, we propose funding provider incentives to overcome this shortage. Bolstered by proposals 211, 240, and 241, we ensured the Community Pool would be well-funded for this exact purpose.

Program Budget

In this initial pilot program, we will request $5,000,000. We estimate these incentives bring on over 1,000 A100s and are estimated to run for 120 days. The estimated start date will be February 2024.

Any non-distributed tokens will be returned to the community just as done here.

Approaches

We propose three methods to incentivize providers, all administered by the Pilot Program Administrator (PPA), which can be an individual or a team. The PPA will be elected by a vote within the steering committee. Anyone can apply to be the PPA, provided they meet the following criteria:

• Should have extensive experience in managing the community.
• Must possess deep expertise in managing Akash providers.
• Should be publicly known and respected within the Akash Community.
• Ideally should have contributed to the Akash open-source repositories.

Responsibilities:

• Commit time and resources towards administering the program.
• Disclose all payments to the public and maintain a strict record of all payments that are subject to 3rd party audit if necessary.
• Provide server management and support to participants.
• Provide responsible settlement services (AKT to USD) for participants if needed.
• Builds tools for verifying compute.

1. Committed Pool

The goal of the committed pool is to provide high-quality computing resources by engaging professional providers of compute, usually tier-2 or higher data center operators, who can commit to providing computing power to Akash for at least one year. The program is open to anyone, but it has strict requirements for quality, quantity, reliability, compliance, and support.

Information (pricing, specs, and availability) on all compute providers considered for this pool will be shared publicly. The rationale guiding provider selection will also be disclosed to the public.

Some advantages are:

• High-quality and reliable compute
• Cost-optimized (No overage)
• Akash Core Team has identified providers that are ready and can go live with short notice

Some disadvantages are:

• Fewer participants (providers)
• More centralized
• A smaller sample size means less data to learn from for building on-chain incentives
• Risk of obsolescence

Budget & Distribution

We propose allocating 40% of the Pilot budget that is approximately $3,500,000 with the following distribution (not final):

Chipset	Est. Price / Hr	Quantity	Total
H100	$2.49	83	$1,800,000
A100 80 GB	$1.69	54	$800,000
RTX A6000	$0.98	105	$900,000

2. Liquidity Mining Pool

In alignment with how DeFi applications and other DePIN projects (like Helium Network, Filecoin, and others) we propose a model in which providers supply GPUs to a homogeneous configuration pool. This pool allocates a fixed amount of tokens per epoch, distributed pro-rata to all participants.

Each provider's earnings will be distributed periodically, with a cadence that allows us enough time to check for accuracy, uptime, and other critical metrics. The source code and reports will be made publicly available for the sake of transparency.

These incentives will attract and retain providers on Akash, directly addressing the current GPU supply shortage. The program's subsidies are designed to ensure profitability for providers, secure meaningful quantities of GPUs in desired configurations, and provide technical support.

Some advantages are:

• More participants
• Increased awareness
• A larger sample size means more data to learn from for building on-chain incentives

Some disadvantages are:

• Unknown quality and reliability
• Greater volatility on compute availability
• Value-extraction by arbitrageurs (Inefficient use of funds)
• Greater risk
• Harder to answer tenant questions

Budget & Distribution

We propose allocating 40% of the Pilot budget which is approximately $750,000 with the following distribution (not final):

Chipset	Allocation	Value
Any Nvidia GPU with at least 24GB VRAM	100%	$750,000

3. Next-Generation R&D Pool

As AI models are evolving the GPU architecture and following suit. This model provides sufficient budget to the PPA to understand the dynamic demand curve by allowing for benchmarking and customer development

Budget & Distribution

We propose allocating 20% of the program budget, which is $750,000 and will be exclusively allocated to GPUs launched in 2024 or later. This will apply to both Nvidia and AMD GPUs. Some GPU models under consideration are Nvidia’s H200, B100, L40s, and AMD’s MI300 MI350.

Some advantages are:

• Greater flexibility
• Future-proofing
• Being able to be first-to-market on new GPUs

Some disadvantages are:

• Opportunity cost of not being able to use the funds immediately

[shimpa1]

Great incentive and definitely something Akash needs.
A few questions please;

• There's a number of different A100 models - the biggest difference being 40GB or 80GB of on-board memory. How will the incentives differ between these two models?
• What about H100? That is an even more desirable GPU.
  thank you!

[lechenghiskhan]

Great question!

H100s are in consideration, but the truth of the matter is price vs performance.
H100s are, on average, ~2x the cost of A100s, and many of the tenants we've talked to prefer the better price vs performance of A100s instead. In short: there is a larger market for A100s.

As far as the configuration, we think there should be a split between A100-80s and A100-40s with A100-80s being prioritized.
Furthermore, there will be other GPUs and CPU-centered configurations too! Their respective shares of the total incentives will be smaller relative to what's allocated to A100s, however.

Stay tuned for the detailed breakdown with all the GPU configurations.

[shimpa1]

A100-40GB is a third of the price of an A100-80GB and the performance gains are not that big, especially if the model fits into onboard memory (in that case there is zero difference in terms of performance). Just something to consider.

[gosuri]

The actual GPU (A100-80/40/H100) will be determined based on the demand we assess over the next month from qualitative feedback. This post is to open the discussion for the proposal and not to specific configurations.

[karnthis]

I am concerned by the short lead time for starting these incentives.

A100s are not simple to get, and new purchases are often significantly delayed. This will lead to not new resources entering the Akash marketplace but instead the movement of existing resources into the pool, resulting in significant payouts to these existing providers for probably the first 2 months of the expected 4 month duration.

What steps will be taken to ensure the incentives reward growth and not exploitation?

PS, I want to echo @shimpa1 's excellent questions.

[shimpa1]

Indeed the lead time is quite short. Definitely not long enough to figure out how to finance and source the GPUs in time.

[lechenghiskhan]

We (the Akash core team) aim to start these incentives next month, but we have been working with/speaking to tenants and providers every day since the launch of GPUs on Akash.

For further context, the demand/asks have focused on A100s more so than any other GPU, so we have a good degree of confidence in focusing on A100s vs other models. To be clear, we will have incentives allocated towards other GPU models like 4090s, V100s, A6000s, and others.

You're right that A100s are not simple to acquire, but we have several sources and will do our best to ensure we have a well-stocked inventory for the program's launch. We also have sources with H100s, so a reallocation towards other models is possible if we need to do so.

Thank you for diving into this subject and participating in the discussion!

[karnthis]

Could you clarify who "we" is in this context?

From the proposal, I was under the impression the incentives were intended to build the general GPU provider community and work collectively. From your above statement, it sounds like you are seeking funding for a limited pool you or another team already has ready to deploy and simply want extra subsidies.

If this is for the general community, will this "well-stocked inventory" be available for purchase by the community to deploy to the pool?

[gosuri]

We = The proposers and administrators responsible for distributing the funds.

The pilot program is available for anyone with the right GPUs to participate in and is administered by Overclock Labs. All aspects are open-sourced and will be verifiable by anyone.

OCL has done the legwork to identify potential providers that can quickly ramp up the supply we anticipate will be ready when the program starts.

The goal of the pilot is to understand optimal distribution based on demand and supply curve to feed into the larger incentive program that goes on-chain.

[gulekisimo]

How do you estimate the ROI?
What do you think provider retention will be after 120 days without incentives?

[lechenghiskhan]

ROI is a good question and a complex one.

The return here can be measured in so many ways; here are just some specific examples:

1. Marketing and PR. The size of this unprecedented incentive program will generate a lot of attention and exposure for Akash Network. It will attract new tenants, providers, analysts, and audiences unaware of Akash.
2. Greater credibility and scale. Akash Network will increase its resources by 10x or more with this program. This will give Akash Network the size and scale to serve workloads on a scale not possible on any other decentralized cloud. This further enhances Akash's credibility and standing.
3. Usage and community pool. The additional capacity will drive a ton of additional usage. This boost will dramatically increase the network's revenue and one of the sources for replenishing the community pool via the take rate.

Regarding provider retention after incentives:

1. Our current average utilization rate on all GPUs is ~44% and ~91% on A100 GPUs. All of this is being achieved with zero incentives. If current utilization holds for the additional GPUs brought on by the incentive program, retention will not be a challenge as provider profitability requirements are met.
2. A major focus is on highly sought-after GPUs like A100s, which means they'll be in high demand and Akash is where the goods will be.
3. Furthermore, the Akash Core Team will in be in direct contact with major providers and tenants, taking proactive measures to ensure the best possible experience to maximize retention. Ensuring a class-A experience will build loyalty and trust in Akash Network.
4. The community is one of Akash's greatest strengths. The Akash Network's vast and highly capable community has been speeding new users up the learning curve and getting past hurdles since the beginning and I fully expect this to continue and grow.

[BB83740]

Today with 65 GPU we are on 2K$/day so 1000 GPUs with 5M$ incentives mean 30K$/day, and for 300M$ that is times 60, meaning 1800K$ per day ?

Also are ncentives also for CPUs and storage ?

[lechenghiskhan]

The incentives will also include allocations for CPU-heavy configurations including memory and storage. The GPU-specific configurations will also include minimum requirements for CPU, memory, and storage. The program spec includes all of these details.

As far as what Akash Network's revenue will look like, it will be hard to predict and depend on a battery of variables like utilization rate, exact composition, participation rate, etc. The goal of the Akash Core Team is to drive full participation from providers and the highest utilization rate possible from tenants.

[BB83740]

Thanks. Can you please share where one can find the incentive program ?

Another thought :
If GPU is already 80% cheaper than AWS today, do you think it will make a big difference by giving incentives to both tenants and providers to reach say a 90% discount ? Is the 10% further discount (or more please correct me if I am wrong) à big deal ?

[lechenghiskhan]

The specifics haven't yet been finalized, but we will present the exact specs and configurations soon.

Getting community feedback here is an important part of informing how the specs/configurations will be structured.

[88plug]

Hello everyone,

I've been closely following the discussions around the Akash Network's GPU Provider Incentives, and I must say, it's shaping up to be a transformative initiative for our network. However, I'd like to propose an amendment that could potentially enhance the long-term stability and effectiveness of this program.

Proposed Amendment: Minimum One-Year Commitment with Quarterly Vesting

My proposed change aims to balance the need for long-term network stability with the practical realities of deploying high-end GPUs and HPC systems. This update ensures providers and their GPUs are not just passing ships but committed members of the Akash family. Here’s how it would work:

• One-Year Commitment: This timeframe is critical. High-end GPUs, especially models like A100s and H100s, require extensive setup, optimization, and integration into our network. New providers participating in the incentive program would be required to commit to a minimum of one year. This commitment is essential for ensuring that providers are not just transient participants but long-term partners in our network's growth.
• Quarterly Vesting Schedule: The incentives offered would be vested and distributed quarterly. This approach ensures a sustained contribution from the providers and avoids any potential short-term, speculative participation.

Why This Matters?

• Consistency in Network Growth: A one-year commitment from providers brings a predictable and stable growth trajectory to our network.
• Quality over Quantity: We prioritize providers who are in this for the long haul, ensuring a dedicated and reliable provider base.
• Balanced Ecosystem: This approach aligns provider incentives with the network's long-term objectives, fostering an ecosystem where all parties thrive together.

Importance of This Amendment:

• Technical Readiness and Investment: High-density GPU environments are complex and require more than just a month or two to be fully operational and optimized. A year-long commitment respects this setup period and ensures providers are adequately prepared to offer high-quality service.
• Stable Network Growth: We're fostering a network that values long-term relationships with providers. This amendment aims to attract dedicated partners, thereby ensuring a consistent and reliable growth trajectory.
• Quality of Service: By considering the technical and operational aspects, we're not just expanding our network; we're enhancing its capability to handle high-demand, high-performance tasks, ensuring Akash remains a leader in decentralized cloud services.

I firmly believe that this amendment could be a cornerstone in building a more robust and reliable Akash Network. Of course, this is a proposal, and the essence of our community lies in collaboration and open dialogue. I look forward to a constructive discussion and a brighter future for Akash Network!

Kind regards!

[anilmurty]

I like this idea but I do wonder if it will significantly limit the number of people that can or will want to participate. Definitely makes sense for a part of the program to be reserved for the 1y commit (and incentivized higher)

[ThanosNode]

If I'm plugging $40K (or "x" multiple) worth of GPU's into the network, I will need a minimum 1 year commitment, with a 2 - 3 year option (preferred). Supply follows demand, not the other way around. Business 101.

[anilmurty]

in general both supply and demand need to be booted up in concert. For higher end GPUs (A100s in particular) I'd argue that demand currently follows supply. We (core team and the Akash community) have literally seen people jump through hoops to get access to A100s and in many cases, non crypto folks also being willing to put up with crypto. This is part of what has driven the core team to take the initiative to work on this proposal now (even though it is something that has been discussed for a long time). TL; DR: not all compute is the same and market dynamics of years past are very different from the last year.

[ThanosNode]

I maintain hype based short-circuit market forces are not end runs for basic demand & supply. I would caution against using the last 18 months as a benchmark for measuring market dynamics......most of those assumptions post-pandemic are being reversed.

While the pandemic (mid/late stage) propelled demand, recent dip in growth hints at a return to pre-pandemic market dynamics. 35% YoY growth in 2020-22, IDC reported a slowdown to 17.8% in 2023, suggesting the initial surge was an artificial bump fueled by the sudden shift to remote work and digital acceleration. Adjusting to the post-pandemic landscape, even Gartner expects the market to stabilize around 14.3% annual growth over the next five years, reflecting a return to fundamentals of organic demand and supply.

Real needs and cost optimization applies to both sides of the marketplace. While the pandemic served as a catalyst, future success will hinge on delivering tangible value beyond the temporary bump.

[anilmurty]

the demand in 2023 has nothing to do with the pandemic - it has everything to do with the growth in AI. Anyways, this seems to be going off topic. The main point is it does make sense to have some portion of the GPU supply to be incentivized for longer term commits (just like all cloud providers do).

[pamanseau]

Dear Akash Network Community,

We appreciate the effort put forth in the recent proposal outlining the GPU incentive program. As HyperClouds, we are eager to contribute to the success of the Akash Network and believe in the potential of the proposed initiative. However, we have identified several concerns that we feel are crucial for the long-term viability and success of the program.

1. Lack of SLA (Service Level Agreement):
   One major concern is the absence of a Service Level Agreement (SLA) or Service Level Objective (SLO) mechanism on the Akash network. Without a clear framework, tenants may face challenges distinguishing between providers, ranging from home labs to tiered certified data centers. We propose the inclusion of an SLA/SLO mechanism, with penalties and refunds for providers not meeting a minimum uptime threshold, providing tenants with the assurance they need.

2. Pricing and A100 Availability:
   While we understand the intention to target an 80% discount pricing over hyper-scaler on-demand pricing, achieving this goal amidst the A100 shortage poses a significant challenge. The proposed pricing seems aggressive, especially when compared to SPOT pricing on major hyper-scalers or on-demand pricing of Vast, and FluidStack. Consideration should be given to utilizing community funds to compensate providers and ensure the feasibility of the target pricing and long-term commitment.

3. Reliability and Incentive Targets:
   Tenants investing significant time in AI workloads expect reliability. An SLA becomes essential as tenants may spend days or even weeks on training or fine-tuning tasks or have an SLO for inference. The proposed incentive structure should prioritize availability and stability to ensure a consistent and reliable service for tenants.

4. Resource Allocation and Limited-Time Offers:
   Providers, including HyperClouds, may have nodes with A100 GPUs that are utilized only part of the time. There is a need to address how resources can be efficiently shared or allocated to the Akash Network, especially for time-limited GPU-intensive jobs. Offering flexibility in resource allocation from providers.

5. Permanent Storage SLA Tiers:
   Permanent storage, a critical component in many workloads, should have clear SLA tiers. This ensures that tenants have a transparent understanding of the reliability and performance of the storage solutions offered by providers.

We believe that addressing these concerns will enhance the robustness of the proposed GPU incentive program and contribute to the overall success and growth of the Akash Network. We look forward to further discussions and collaboration to refine and optimize these aspects for the benefit of the entire community.

Best regards,

Pierre A. Manseau
HyperClouds.io

[kirinzhang]

Dear Akash Community,

I have the following inquiries:

1）Regarding the $5 million incentive fund, what is its source? from the treasury, or additional issuance of token?

2）Has there been a survey conducted on A100 providers to ensure that the implementation of the incentive program will attract a substantial number of A100 providers into the network, not merely due to a lack of awareness about the Akash platform?

3. Is the current usage rate of 91% for A100s based on a sufficient sample size? In the event that the incentive program is successful and a large number of A100s enter the Akash network, will the demand side be able to keep pace with this increase?

Best regards!

[gosuri]

1. Funds come from the community pool.
2. The demand for the A100 is quantified by several customer interviews; the program is designed to be flexible with regard to the exact chipset.
3. Satisfactory sample size from filled and unfulfilled demand.

[BB83740]

Dear Akash community,

Here are some of my thoughts on this matter. I hope it help us find the best incentive program.

1. Progressive reward for providers
   There should be progressive reward rules for providers based on their effective renting periods : the longer the period you offer your GPUs for rent, the higher the reward. For instance,  a A100 available 113 hours on a 168 hours week, get 67.2% of the reward.
   Full reward is only achieved with 100% provision of service. This way the incentive amount we can communicate on is significantly higher than just a fixed one (because not many providers can provide their hardware 100% of the time). This way it will sound even more attractive to new providers.

2. SLAs : a must have on the marketplace for tenants
   Again, the higher the SLAs and provision of service period, the higher the reward. I think provider reviews from previous tenants with stars from 1 to 5 must also clearly be displayed to help future tenants to chose.
   SLAs info and provider reliability should clearly be displayed on the market place.
   The higher the SLA, the more frequent a provider will rent his GPUs, the higher the incentive reward.

3. The most powerful GPUs (and CPUs) get a higher reward rate (evolutive A to E quality grades based on demand)
   Another parameter to consider is that demand is focused on powerful GPUs. Hence we should focus on better rewarding such providers than class C providers with less powerful GPUs.

4. Rewards based on the number of GPUs (quantity)
   The number of provided GPUs is also something to reward because it shows a strong provider commitment to the network.
   A provider willing to rent his A-grade GPU  should receive less reward than one who offers multiple C-grade GPUs. Demand is the ultimate referee to decide the reward amounts (dynamically).

[izrake]

Dear Akash Community,

I am adding my thoughts and would like to make some adjustments to the proposal where we can also focus on running incentives for the network activity driver, whether on the CPU or GPU side.

We all know the market around GPU is booming & as a community, we all need to work towards ensuring that we have sufficient supply created around it.
While I agree on the supply-side creation, on the demand side, we are seeing less growth in terms of the existing capacity utilisation in our network. As per Akashlytics, we currently have 174 GPUs which are there & out of which 74 are leased out, which is lower than 50% of the capacity utilization rate.

What if we bring in another GPU provider & ask them to lower the cost by providing the incentive? Regarding demand & supply economics, if there is enough demand, we should refrain from working toward incentivization to compete against the price. Instead, we should work towards building the experience with very optimal price competition.

How are we ensuring & believing that after giving a lower cost of GPU, how much percentage of industry will run towards using Akash?

If we recheck some metrics, Akash has good amount of CPU supply, but we have never been able to generate enough demand; even as we speak, there is a considerable demand for CPUs, which is untapped. Take the example of Spheron. We added our provider, made the necessary changes & now we have close to 600 active leases in just one month.

Let me place some numbers again here. The average CPU utilization of providers except Spheron is lower than 50% & most of their CPU, RAM & everything is just sitting IDLE doing nothing.

I want to ask you to consider the teams building an experience layer for the CPU, like Spheron & also the providers who are willing to add CPU capacity. If there is no interest around the same, that's also good, but I would love to hear some thoughts about it.

I still believe the community fund must be used with extreme caution by ensuring that not just supply but actual demand creation is also been created so that added GPUs are not switching off after 1 year of time frame.

Also, it would be better to place some numbers like if we put in $5M then how much value we are going to be generating for the network. What if the value is not generated, will those GPUs will be sold in the open market & brought in the amount back?

[gosuri]

Non-GPU supply is difficult to incentivize because the configurations are extremely heterogeneous. For example, the configuration required to run a node is very different from that needed to run an API server. If non-GPU compute were to be incentivized, which specific configuration would you incentivize, and what are some datapoints to increase confidence for demand sustainability?

Either way, true to Akash's open culture, the specific configurations to be incentivized will be posted for feedback before going live.

How are we ensuring and believing that after offering a lower cost for GPUs, what percentage of the industry will shift towards using Akash?

As we addressed in previous comments, we have quantifiable demand with high statistical confidence that improving the supply of high-density GPUs like the A100s and H100s leads to increased demand. The A100 utilization rate is now at over 90%.

[izrake]

Got it, does that mean, there is no incentives for providing the CPU & building around it leveraging Akash as chain (not the open source code)?

Also, it would be amazing to hear some real numbers & projections around utilisation in quantifiable number projection. The reason i am trying to understand this is because some of the companies are trying to do the same by having a high number of GPUs aggregated but when it's coming to real number of driving the demand they are failing because of certain regulations around GPUs & use of such infra over open data or private data.

Does that also been considered? If so, what's measure team is taking. Also, will these compute be used for open data or can also be used by private companies?

[gosuri]

Got it, does that mean, there is no incentives for providing the CPU & building around it leveraging Akash as chain (not the open source code)?

I just answered your question. Incentivizing non-GPU supply is possible if you can provide a homogeneous configuration and make a case for the demand - like compute, storage, memory, and bandwidth requirements.

Also, it would be amazing to hear some real numbers & projections around utilisation in quantifiable number projection.

Estimated demand projections will be posted well in advance of rolling out the program; the demand is mostly around high-density GPUs.

Does that also been considered? If so, what's measure team is taking. Also, will these compute be used for open data or can also be used by private companies?

I have to apologize; it could be the language barrier, but I'm having trouble understanding what you mean by this. Akash is used by private companies, you don't need to open your data.

[izrake]

I got the answer, @gosuri. Thank you for answering it.

[deathlessdd]

If akash cant bring in people with a100s maybe akash can help current providers that would like to buy a100s buy them, Since there very expensive to get, Id like to buy some but there just not in my budget to get. Bought the servers to hold a100s but these are also expensive, Just an idea though.

[BB83740]

Just a question : Why spent assets do not follow CPUs or memory records ?