Pre-draft discussion for contributions & becoming a new auditor

[piber-dev]

Introduction

Note: this is NOT a proposal draft yet, we strictly created this for discussion.

After having the idea of creating an automatic stat-based auditor, we heard news that Moultrie Audits are shutting down their services in December. We thought: "there's no better time than now to start," and recently started getting involved with different SIG's and talking to different community members. Talking about this and showing the powers of one of our potential services, we realized that not everything we wanted to create was new. In fact, a big portion of what we were trying to create is already done by Cloudmos' indexer. This is perfect! This means that efforts have already been made to create part of the services we want to bring to Akash, and that these pre-existing parts can be expanded, and to have services be building on the same proven-to-work technology as an ecosystem partner.

What we want to do

• We are looking to contribute to the Cloudmos indexer to expand their tasks to include examples such as: benchmarking, testing RPC nodes, tracking the lease uptime of providers, and other trackable metrics to include in their API. We would also want snapshots of this database to be made semi-regularly, if there is demand for this.
• We are looking to create our own services:
  • Stat-based, tier-based auditing service, where filtering API can be used on Cloudmos & signing can happen for CLI users. Example on the filtering this would allow.
  • General auditing, where we can have a general audit similar to how Overclock Labs sign their providers, and how Moultrie Audits signed providers. We create a set of criteria that we think are "good enough," and sign generally from this.
  • KYC-based auditing on providers and tenants. Details for this are TBD and to do the KYC, we would be working with companies that do this as a service.
  • Yet another Akash Stats Page, if there is demand for one. In the past weeks, some questions have not been able to be responded to due to data not being formatted in such a way.
    To our knowledge, this relates to: rich lists as a pie graph, the community pool growth over time, and provider filtering shown as a demo here.

We estimate that this will take 100 hours for benchmarking, 20 hours for testing RPC nodes, 120 hours for building the tier-based auditing system and API, 10 hours for the general auditing service after the tier-based auditing system is built, 100 hours for the KYC auditing on providers and tenants, including a business plan. We have also spent approximately 45 hours on building the protoype for the hackathon, video demos, and other preliminary work. We are not counting any time for SIG calls or Office Hours calls. In this calculation we do not include the stats page, because we have no idea if the community wants this yet.

What we want help with

We want help from Cloudmos understanding what knowledge it would take to create the tasks. While it looks pretty straight forward to create a task, we want to make sure that we do it properly in their way and introduce it as properly as they would do themselves. If it isn't straight forward, adjustments might be made to the draft proposal (and later proposal). We also want to discuss with Cloudmos if they could introduce filtering if we supply an API for it, or if they want to build a replica within their frontend.

We want help from Praetor App to create a way where whitelists can be shared by auditors and formatted in a good way. Is this a good format for whitelists, or should the standard include more details? Should tenants be tracked as much as providers, in terms of their frequency of deployments & their active deployments? But if this were to happen, strategies could limit tenants for renting too many short-term leases, and maybe tenants would have to pay providers to be whitelisted again. There are many things to consider when it comes to strategies, and we believe that the Praetor App team can help, along with community providers.

We want help from the community including validators to figure out what is fair to charge. This is a proposal that passed charging $50/hr for frontend work -- we think this is fair for open-source work, even with our high taxes and normally higher hourly rates for contractors in our country. Or should we do this based off milestones, out of arbitrary rates to be voted on?

For reference on what the community pool has to work with, we calculated it to grow by $10-15k per day currently. The proposal, assuming our hourly estimates, would then request approximately $20k. Before the draft proposal, we will also research the infrastructure costs of signing X many providers every Y time. For the KYC auditing, we don't expect to get paid for infrastructure costs as this can include a business plan.

Links

video demo explaining the old planned infrastructure, to be replaced by Cloudmos

filtering demo based off dummy data

market research form still open

[instafinanzas]

Great initial post, hopefully more Akash network participants join this discussion

I'd like to see more benefits to the Akash network listed in the draft

Budget is ok as long as it sustains the project while delivering value to the Akash community

This project is coming from an Akash insiders and brings a great opportunity for the Akash core team and community to come together and find out the best way forward for this initiative

[piber-dev]

hello @instafinanzas, thanks for your feedback!

here are some benefits from each tool/contribution. please let us know if you want more details on anything, such as the benefits from each task in the indexer.

• the extended API & SDL filtering benefits tenants in terms of saving time and money in finding providers that they want to use. benchmarking and categorizing these makes it possible to find providers in your country, providers that have been active for a long time, or anything else that the filters add. for the providers, they save on fees/deposits/reserved resources as they don’t have to bid on leases they ”shouldn’t have” per the tenant’s requirements.
• the general auditing benefits both providers and tenants. the tenants benefit from finding reliable providers quickly without filtering on their own or researching providers. the providers benefit from being audited as this means that they will have bigger chances of getting leases from tenants wanting to find quality providers.
• the KYC auditing on both providers and tenants could benefit both providers and tenants. by introducing KYC there would be less attacks and malicious deployments to/on the network from anonymous addresses. providers could have legal benefits from knowing who is renting from them. tenants could benefit from knowing that the provider is tied to a company or a person that they trust.
• tracking RPC nodes benefits the network in that when something goes wrong due to RPC, a good RPC can quickly be found. documentation could be improved by having the RPC node list be automatically changed to nodes that are online. tools such as cloudmos could use this to automatically select good RPC nodes for their users to make it easier to use and to have fewer friction points.

[rodri-r]

Hey @piber-dev I think this is a great proposal too and I think having more than one auditor is of great benefit to the network.

I think this makes it more decentralized, as providers could pick between different auditors and benefit from the different perks/services that each auditor offers.

I like that you all want to cooperate with Cloudmos and Praetor, which also benefits the network.

I think that $50 per hour is fair for frontend work like requested in the proposal you mentioned and that your budget is also reasonable.

[piber-dev]

hi @rodri-r, thanks for your feedback! only thing is that i want to clarify that the frontend work was an example from a proposal that passed, we will mostly be doing backend work, so if you think this changes anything you should be aware.

[rodri-r]

Yes, I understand this will not be frontend. It does not change anything.

What I meant is that your comparison is good.

[cloud-j-luna]

Thank you for starting this discussion! I have a couple of questions:

1. I would like to see a (somewhat) detailed document of the auditing process and attributes that will differentiate you as an auditor and what is the added value proposition of your validator.

2. We create a set of criteria that we think are "good enough," and sign generally from this.

Why would users "care" about what you think is "good enough". What is "good enough"?

3. Regarding KYC auditing, I don't quite understand. The idea is to perform KYC on the providers themselves?

[piber-dev]

hello @cloud-j-luna and thanks for showing interest in the discussion. let us try to clear up any confusion:

1. we don't have a document for this at the moment, but this gif shows pretty well what our filtering will enable. for the auditing via SDL & API and the general auditing, we don't plan to have an auditing process besides having our automated tooling do this for us. for the KYC audits, we have yet to finalize any details, but we will try to automate this as much as possible with a sign-up page for the ones looking to KYC themselves and take payments in AKT if possible. if we cannot do this, we will try to let KYC service providers take payments directly. for the last part of the question, we are not currently planning on setting up a validator.
2. this is precisely what other auditors do, but we want to automate our signing to add/remove providers from the list. requirements are set to differentiate between providers that have shown to be serious, and those that have have not. if you quickly want a provider that shouldn't have any major issues, that isn't for example taking your money without spinning up a deployment, you'd use one that has been given a general audit. re: what "good enough" means is not yet defined, we will speak to tenants and providers to get a range of criteria that will automatically let providers enter signing.
3. the idea is to KYC both providers and tenants. to quote my reply in the another reply: "the KYC auditing on both providers and tenants could benefit both providers and tenants. by introducing KYC there would be less attacks and malicious deployments to/on the network from anonymous addresses. providers could have legal benefits from knowing who is renting from them. tenants could benefit from knowing that the provider is tied to a company or a person that they trust."

PS: our reply to @instafinanzas shows more of what benefits the different tools and contributions will have.

thanks again and let me know if you want more details on anything.

[baktun14]

Hey @piber-dev, this is a great proposal.

Like we talked during various meetings, we'd be happy to collaborate with you to bring automated auditing and benchmarking of providers. This is one of the projects we were planning on doing and I think would add a lot of value to Akash users and specifically on Cloudmos.

Could you ellaborate on the specifics of, like what will you track:

benchmarking, testing RPC nodes, tracking the lease uptime of providers, and other trackable metrics to include in their API

@88plug has done some work as well on this topic, so I think it would be a good opportunity to merge all of our efforts into one.

What I suggest for benchmarking:

• We use part of the code we wrote to automatically deploy on all the active providers
• Deploy the benchmarking image (you or maybe help from @88plug)
• Have that benchmarking data stored in the Cloudmos database
• Write API endpoints to have access to the data publicly
• Implement various UIs in Cloudmos to view/filter the data

Let me know what you think.

Cheers

[piber-dev]

hello @baktun14 & thank you for your detailed feedback.

Could you ellaborate on the specifics of, like what will you track:

for benchmarking providers, there are a lot of features that can be tested such as GPU/persistent storage/IP leases, which we are currently not planning to include, though if there is demand for this it would likely not be a big addition. we are thinking of verifying attributes related to e.g., CPU and testing different speeds for different resources and services. we also plan to verify that RPC nodes are working and that providers are online. if you have any suggestions on what else to track, let us know.

What I suggest for benchmarking:

We use part of the code we wrote to automatically deploy on all the active providers
Deploy the benchmarking image (you or maybe help from @88plug)
Have that benchmarking data stored in the Cloudmos database
Write API endpoints to have access to the data publicly
Implement various UIs in Cloudmos to view/filter the data

this makes sense. however, not sure how we would implement UI's to Cloudmos yet as neither of us has experience with the repository. if we do this, we would need help figuring out how many hours we should estimate for this and what we should implement. we can book a meeting to discuss this if you're interested. storing the data in the Cloudmos database is perfect as that means this proposal does not have to ask for infrastructure costs, unless the benchmarking will be paid by us.

[baktun14]

Ok thanks for the clarification.

For sure we will help you implement features on Cloudmos if this gets through.

[piber-dev]

This discussion is finished and the draft proposal is live at the link below. Thanks to everyone who participated in the conversation, giving feedback and asking questions, both here and in live calls 🙏

https://github.com/orgs/akash-network/discussions/406

// p-i-3