Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update news.md #33

Merged
merged 3 commits into from
Jan 2, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 66 additions & 74 deletions root-pages/news.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,120 +7,112 @@ pagination_prev: null
pagination_next: null
---

# Codex R&D Update for the week of 11
# **Codex R&D Update for week 11**

The Codex dev team is moving forward rapidly. The organisation’s development efforts mainly focus on launching the beta testnet by the end of the year. Following are some recent research and development highlights.

Development
# **Development**

Development is currently broken into three distinct sub-teams:
Client, Testing and Infrastructure
Marketplace and
Research.

Below are summaries of each sub-team.
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research

Below are summaries of each sub-team.

Client
For the client, the team is focused on primary tasks: block merkelization, Poseidon2 NimNim-poseidon2 implementation, and wiring the proving system.
# **Client**

The client is the software that runs ^on the user’s local machine. It has all the components to handle the endpoints integrations, data, block management, erasure coding, and interactions with the marketplace.
For the client, the team is focused on these primary tasks: block merkelization, Poseidon2 Nim-poseidon2 implementation, and wiring the proving system.

See currently exposed endpoints for the Codex client (conform to OpenAPI specification) at: https://api.codex.storage
The client is the software that runs on the user’s local machine. It has all the components to handle the endpoints integration, data, block management, erasure coding, and interactions with the marketplace.

See currently exposed endpoints for the Codex client (conform to OpenAPI specification) at: <https://api.codex.storage>

Active work is ongoing in order to integrate the above Codex client endpoints to be accessible via the Codex frontend written in React here: https://github.com/codex-storage/codex-frontend
Active work is ongoing to integrate the above Codex client endpoints to be accessible via the Codex frontend written in React here: <https://github.com/codex-storage/codex-frontend>

Epic: Block Merkelization
## **Epic: Block Merkelization**

For block merkelization, the team is focused on reducing metadata overhead. They are optimising the structure for handling and indexing block data. This effort requires changes in block Merkelization. These are the most recent efforts, which can be publicly viewed on GitHub:

- Merkelization concrete PR (merged)
- <https://github.com/codex-storage/nim-codex/pull/566>
- Working on nim-datastore to support atomic updates (in review)
- <https://github.com/codex-storage/nim-datastore/pull/58>
- Working on safe block deletion - using ref-counting
- Continuing work for #566 to integrate with Poseidon2 and the proving system

Merkelization concrete PR (merged)
https://github.com/codex-storage/nim-codex/pull/566
Working on nim-datastore to support atomic updates (in review)
https://github.com/codex-storage/nim-datastore/pull/58
Working on safe block deletion - using ref-counting
Continuing work for #566 to integrate with Poseidon2 and the proving system

## **Epic: nim-poseidon2**

Epic: nim-poseidon2
Poseidon2 is a faster version of the original Poseidon hash function used in zero-knowledge systems (zk). Nim-Poseidon2 is the Nim implementation the Codex team is developing to work efficiently and effectively with the Codex client.

Poseidon2 is a faster version of the original Poseidon hash function used in zero-knowledge systems (zk). Nim-Poseidon2 is the Nim implementation that the Codex team is developing to work efficiently and effectively with the Codex client.
- Merged outstanding PRs
- <https://github.com/codex-storage/nim-poseidon2/pull/7>
- <https://github.com/codex-storage/nim-poseidon2/pull/8>
- <https://github.com/codex-storage/nim-poseidon2/pull/9>
- Added keyed compress
- https://github.com/codex-storage/nim-poseidon2/pull/10
- Updated Merkle root with extra bits to mitigate possible attacks
- https://github.com/codex-storage/nim-poseidon2/pull/11
- Fixed compilation problem with Constantine and secp256k1
- https://github.com/codex-storage/nim-codex/pull/625

## **Epic: Wiring the Proving System**

Merged outstanding PRs
![](https://github.com/codex-storage/nim-poseidon2/pull/7)
![](https://github.com/codex-storage/nim-poseidon2/pull/8)
https://github.com/codex-storage/nim-poseidon2/pull/9
Added keyed compress
https://github.com/codex-storage/nim-poseidon2/pull/10
Updated Merkle root with extra bits to mitigate possible attacks
https://github.com/codex-storage/nim-poseidon2/pull/11
Fixed compilation problem with Constantine and secp256k1
https://github.com/codex-storage/nim-codex/pull/625
The proving system comprises a SNARK circuit that generates the proof. A Storage Provider must also submit the marketplace contract to prove it faithfully stores the associated slot data. Additional components, such as the trusted setup ceremony, hash functions, and the multiple manifests generated for different purposes, are included in the proving system. The team is working on ensuring the proving system is built out and functions to successfully remote audit the slot data by validators. These stories represent the most recent efforts:

Epic: Wiring the Proving System
- Nim reference implementation of the proof input generation (WIP)
- <https://github.com/codex-storage/codex-storage-proofs-circuits/tree/master/reference/nim/proof_input>
- Ongoing implementation of Groth16 (WIP)
- <https://github.com/codex-storage/nim-groth16>

The proving system comprises a SNARK circuit that generates the proof a Storage Provider needs to submit to the marketplace contract in order to prove it is faithfully storing the associated slot data. Additional components such as the trusted setup ceremony, hash functions , and the multiple manifests generated for different purposes are included in the proving system. The team is working on ensuring the proving system is built out and functions to successfully remote audit the slot data by validators. These stories represent the most recent efforts:
# **Testing and Infrastructure**

Nim reference implementation of the proof input generation (WIP)
https://github.com/codex-storage/codex-storage-proofs-circuits/tree/master/reference/nim/proof_input
Ongoing implementation of Groth16 (WIP)
https://github.com/codex-storage/nim-groth16
Testing and Infrastructure
The testing and infrastructure group is preparing to deploy tens or hundreds of nodes and efficiently monitor the activity. This is crucial for a successful launch of the testnet. The following are the epics and stories of development for this process. The main areas of concern are ensuring the testnet is set up and configured properly. It will also include ensuring the client maintains a suitable level of stability.

## **Epic: 2023 Testnet setup**

Epic: 2023 Testnet setup


Configure TCP/UDP port forwarding for Testnet deployment
Configure Pods placement for D/C-Tests runners and tests workload
Check available options to build a service to get Codex Public IP for announcement
Deploy Codex Bootstrap nodes for Testnet
Deploy Geth Bootstrap nodes for Testnet
- Configure TCP/UDP port forwarding for Testnet deployment
- Configure Pods placement for D/C-Tests runners and tests workload
- Check available options to build a service to get Codex Public IP for announcement
- Deploy Codex Bootstrap nodes for Testnet
- Deploy Geth Bootstrap nodes for Testnet

## **Epic: Improve Client Stability**

Epic: Improve Client Stability
- Ongoing debugging of performance and stability issues in the testing environment
- Tooling developed
- Elastic search log retrieval <https://github.com/codex-storage/logtools>
- <https://vimeo.com/884370956/e8a32a58c8?share=copy>
- DHTDht fixes related to performance and stability

- Fix: queue messages when there is no encryption key
- Fix: arrive at working keys in case of simultaneous cross-connect
- Fix timeout and delete

Ongoing debugging of performance and stability issues in the testing environment
Tooling developed
Elastic search log retrieval https://github.com/codex-storage/logtools
https://vimeo.com/884370956/e8a32a58c8?share=copy
DHTDht fixes related to performance and stability

Fix: queue messages when there is no encryption key
Fix: arrive at working keys in case of simultaneous cross-connect
Fix timeout and delete

Marketplace
# **Marketplace**

The marketplace is a smart contract deployed on a blockchain. This smart contract defines all the logic required for users to ask for storage, store collaterals, assign storage nodes to user datasets, settle payments, deal with storage failures, storage provider proof issuance, and all the contract interactions.

Epic: End-to-end Testing

Working on integration testing cleanup (WIP)
https://github.com/codex-storage/nim-codex/pull/607
Debugging various issues for end-to-end testing
## **Epic: End-to-end Testing**

- Working on integration testing cleanup (WIP)
- <https://github.com/codex-storage/nim-codex/pull/607>
- Debugging various issues for end-to-end testing

Research
# **Research**

Current research is mostly halted due to focus on implementing past research for the imminent launch of the testnet. Ongoing efforts include:

**Near-term:**

Near-term:


Ongoing “Groth16 prover” research and analysis.
Details for the functionality of the current proof system
Hash and Merkle tree conventions to make them safe
Continuing research for Codex’s use of erasure coding
- Ongoing “Groth16 prover” research and analysis.
- Details for the functionality of the current proof system
- Hash and Merkle tree conventions to make them safe
- Continuing research for Codex’s use of erasure coding

Long-term:
**Long-term:**

Figuring out how to aggregate proofs (proof compression) for proving system
Determining the correct proof system for Codex
- Figuring out how to aggregate proofs (proof compression) for proving system
- Determining the correct proof system for Codex

Loading