From 584f56ffad58e2bcdc8e841358fe3399657856b3 Mon Sep 17 00:00:00 2001 From: Sklujan <141430704+Sklujan@users.noreply.github.com> Date: Wed, 13 Dec 2023 21:10:41 +0200 Subject: [PATCH 1/3] Update news.md --- root-pages/news.md | 140 +++++++++++++++++++++------------------------ 1 file changed, 66 insertions(+), 74 deletions(-) diff --git a/root-pages/news.md b/root-pages/news.md index afe830c..18ad438 100644 --- a/root-pages/news.md +++ b/root-pages/news.md @@ -7,120 +7,112 @@ pagination_prev: null pagination_next: null --- -# Codex R&D Update for the week of 11 +# **Codex R&D Update for week 11** The Codex dev team is moving forward rapidly. The organisation’s development efforts mainly focus on launching the beta testnet by the end of the year. Following are some recent research and development highlights. -Development +# **Development** Development is currently broken into three distinct sub-teams: -Client, Testing and Infrastructure -Marketplace and -Research. -Below are summaries of each sub-team. +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research +Below are summaries of each sub-team. -Client -For the client, the team is focused on primary tasks: block merkelization, Poseidon2 NimNim-poseidon2 implementation, and wiring the proving system. +# **Client** -The client is the software that runs ^on the user’s local machine. It has all the components to handle the endpoints integrations, data, block management, erasure coding, and interactions with the marketplace. +For the client, the team is focused on these primary tasks: block merkelization, Poseidon2 Nim-poseidon2 implementation, and wiring the proving system. -See currently exposed endpoints for the Codex client (conform to OpenAPI specification) at: https://api.codex.storage +The client is the software that runs on the user’s local machine. It has all the components to handle the endpoints integration, data, block management, erasure coding, and interactions with the marketplace. +See currently exposed endpoints for the Codex client (conform to OpenAPI specification) at: -Active work is ongoing in order to integrate the above Codex client endpoints to be accessible via the Codex frontend written in React here: https://github.com/codex-storage/codex-frontend +Active work is ongoing to integrate the above Codex client endpoints to be accessible via the Codex frontend written in React here: -Epic: Block Merkelization +## **Epic: Block Merkelization** For block merkelization, the team is focused on reducing metadata overhead. They are optimising the structure for handling and indexing block data. This effort requires changes in block Merkelization. These are the most recent efforts, which can be publicly viewed on GitHub: +- Merkelization concrete PR (merged) +- +- Working on nim-datastore to support atomic updates (in review) +- +- Working on safe block deletion - using ref-counting +- Continuing work for #566 to integrate with Poseidon2 and the proving system -Merkelization concrete PR (merged) -https://github.com/codex-storage/nim-codex/pull/566 -Working on nim-datastore to support atomic updates (in review) -https://github.com/codex-storage/nim-datastore/pull/58 -Working on safe block deletion - using ref-counting -Continuing work for #566 to integrate with Poseidon2 and the proving system - +## **Epic: nim-poseidon2** -Epic: nim-poseidon2 +Poseidon2 is a faster version of the original Poseidon hash function used in zero-knowledge systems (zk). Nim-Poseidon2 is the Nim implementation the Codex team is developing to work efficiently and effectively with the Codex client. -Poseidon2 is a faster version of the original Poseidon hash function used in zero-knowledge systems (zk). Nim-Poseidon2 is the Nim implementation that the Codex team is developing to work efficiently and effectively with the Codex client. +- Merged outstanding PRs +- +- +- +- Added keyed compress +- https://github.com/codex-storage/nim-poseidon2/pull/10 +- Updated Merkle root with extra bits to mitigate possible attacks +- https://github.com/codex-storage/nim-poseidon2/pull/11 +- Fixed compilation problem with Constantine and secp256k1 +- https://github.com/codex-storage/nim-codex/pull/625 +## **Epic: Wiring the Proving System** -Merged outstanding PRs -![](https://github.com/codex-storage/nim-poseidon2/pull/7) -![](https://github.com/codex-storage/nim-poseidon2/pull/8) -https://github.com/codex-storage/nim-poseidon2/pull/9 -Added keyed compress -https://github.com/codex-storage/nim-poseidon2/pull/10 -Updated Merkle root with extra bits to mitigate possible attacks -https://github.com/codex-storage/nim-poseidon2/pull/11 -Fixed compilation problem with Constantine and secp256k1 -https://github.com/codex-storage/nim-codex/pull/625 +The proving system comprises a SNARK circuit that generates the proof. A Storage Provider must also submit the marketplace contract to prove it faithfully stores the associated slot data. Additional components, such as the trusted setup ceremony, hash functions, and the multiple manifests generated for different purposes, are included in the proving system. The team is working on ensuring the proving system is built out and functions to successfully remote audit the slot data by validators. These stories represent the most recent efforts: -Epic: Wiring the Proving System +- Nim reference implementation of the proof input generation (WIP) +- +- Ongoing implementation of Groth16 (WIP) +- -The proving system comprises a SNARK circuit that generates the proof a Storage Provider needs to submit to the marketplace contract in order to prove it is faithfully storing the associated slot data. Additional components such as the trusted setup ceremony, hash functions , and the multiple manifests generated for different purposes are included in the proving system. The team is working on ensuring the proving system is built out and functions to successfully remote audit the slot data by validators. These stories represent the most recent efforts: +# **Testing and Infrastructure** -Nim reference implementation of the proof input generation (WIP) -https://github.com/codex-storage/codex-storage-proofs-circuits/tree/master/reference/nim/proof_input -Ongoing implementation of Groth16 (WIP) -https://github.com/codex-storage/nim-groth16 -Testing and Infrastructure The testing and infrastructure group is preparing to deploy tens or hundreds of nodes and efficiently monitor the activity. This is crucial for a successful launch of the testnet. The following are the epics and stories of development for this process. The main areas of concern are ensuring the testnet is set up and configured properly. It will also include ensuring the client maintains a suitable level of stability. +## **Epic: 2023 Testnet setup** -Epic: 2023 Testnet setup - - -Configure TCP/UDP port forwarding for Testnet deployment -Configure Pods placement for D/C-Tests runners and tests workload -Check available options to build a service to get Codex Public IP for announcement -Deploy Codex Bootstrap nodes for Testnet -Deploy Geth Bootstrap nodes for Testnet +- Configure TCP/UDP port forwarding for Testnet deployment +- Configure Pods placement for D/C-Tests runners and tests workload +- Check available options to build a service to get Codex Public IP for announcement +- Deploy Codex Bootstrap nodes for Testnet +- Deploy Geth Bootstrap nodes for Testnet +## **Epic: Improve Client Stability** -Epic: Improve Client Stability +- Ongoing debugging of performance and stability issues in the testing environment +- Tooling developed +- Elastic search log retrieval +- +- DHTDht fixes related to performance and stability +- Fix: queue messages when there is no encryption key +- Fix: arrive at working keys in case of simultaneous cross-connect +- Fix timeout and delete -Ongoing debugging of performance and stability issues in the testing environment -Tooling developed -Elastic search log retrieval https://github.com/codex-storage/logtools -https://vimeo.com/884370956/e8a32a58c8?share=copy -DHTDht fixes related to performance and stability - -Fix: queue messages when there is no encryption key -Fix: arrive at working keys in case of simultaneous cross-connect -Fix timeout and delete - -Marketplace +# **Marketplace** The marketplace is a smart contract deployed on a blockchain. This smart contract defines all the logic required for users to ask for storage, store collaterals, assign storage nodes to user datasets, settle payments, deal with storage failures, storage provider proof issuance, and all the contract interactions. -Epic: End-to-end Testing - -Working on integration testing cleanup (WIP) -https://github.com/codex-storage/nim-codex/pull/607 -Debugging various issues for end-to-end testing +## **Epic: End-to-end Testing** +- Working on integration testing cleanup (WIP) +- +- Debugging various issues for end-to-end testing -Research +# **Research** Current research is mostly halted due to focus on implementing past research for the imminent launch of the testnet. Ongoing efforts include: +**Near-term:** -Near-term: - - -Ongoing “Groth16 prover” research and analysis. -Details for the functionality of the current proof system -Hash and Merkle tree conventions to make them safe -Continuing research for Codex’s use of erasure coding +- Ongoing “Groth16 prover” research and analysis. +- Details for the functionality of the current proof system +- Hash and Merkle tree conventions to make them safe +- Continuing research for Codex’s use of erasure coding -Long-term: +**Long-term:** -Figuring out how to aggregate proofs (proof compression) for proving system -Determining the correct proof system for Codex +- Figuring out how to aggregate proofs (proof compression) for proving system +- Determining the correct proof system for Codex From f0039b0f4b6130a76de1ca492d7549c6d87154f9 Mon Sep 17 00:00:00 2001 From: Sklujan <141430704+Sklujan@users.noreply.github.com> Date: Fri, 15 Dec 2023 23:06:33 +0200 Subject: [PATCH 2/3] Update news.md I made Jesse's corrections; please have another glance! --- root-pages/news.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/root-pages/news.md b/root-pages/news.md index 18ad438..13c9ac7 100644 --- a/root-pages/news.md +++ b/root-pages/news.md @@ -7,7 +7,7 @@ pagination_prev: null pagination_next: null --- -# **Codex R&D Update for week 11** +# **Codex R&D Update for Week of December 11, 2023** The Codex dev team is moving forward rapidly. The organisation’s development efforts mainly focus on launching the beta testnet by the end of the year. Following are some recent research and development highlights. @@ -47,11 +47,11 @@ For block merkelization, the team is focused on reducing metadata overhead. They Poseidon2 is a faster version of the original Poseidon hash function used in zero-knowledge systems (zk). Nim-Poseidon2 is the Nim implementation the Codex team is developing to work efficiently and effectively with the Codex client. - Merged outstanding PRs -- +- https://github.com/codex-storage/nim-poseidon2/pull/7 - - - Added keyed compress -- https://github.com/codex-storage/nim-poseidon2/pull/10 +- - Updated Merkle root with extra bits to mitigate possible attacks - https://github.com/codex-storage/nim-poseidon2/pull/11 - Fixed compilation problem with Constantine and secp256k1 @@ -84,8 +84,7 @@ The testing and infrastructure group is preparing to deploy tens or hundreds of - Tooling developed - Elastic search log retrieval - -- DHTDht fixes related to performance and stability - +- DHT fixes related to performance and stability - Fix: queue messages when there is no encryption key - Fix: arrive at working keys in case of simultaneous cross-connect - Fix timeout and delete From 71776d15230aa94c6b6da5b234c92b8bf12d396a Mon Sep 17 00:00:00 2001 From: Sklujan <141430704+Sklujan@users.noreply.github.com> Date: Tue, 19 Dec 2023 17:13:19 +0200 Subject: [PATCH 3/3] Update news.md --- root-pages/news.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/root-pages/news.md b/root-pages/news.md index 13c9ac7..b395863 100644 --- a/root-pages/news.md +++ b/root-pages/news.md @@ -47,15 +47,15 @@ For block merkelization, the team is focused on reducing metadata overhead. They Poseidon2 is a faster version of the original Poseidon hash function used in zero-knowledge systems (zk). Nim-Poseidon2 is the Nim implementation the Codex team is developing to work efficiently and effectively with the Codex client. - Merged outstanding PRs -- https://github.com/codex-storage/nim-poseidon2/pull/7 +- - - - Added keyed compress - - Updated Merkle root with extra bits to mitigate possible attacks -- https://github.com/codex-storage/nim-poseidon2/pull/11 +- - Fixed compilation problem with Constantine and secp256k1 -- https://github.com/codex-storage/nim-codex/pull/625 +- ## **Epic: Wiring the Proving System**