Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store preprocessed codewords on disk #17

Open
ValarDragon opened this issue Aug 14, 2020 · 0 comments
Open

Store preprocessed codewords on disk #17

ValarDragon opened this issue Aug 14, 2020 · 0 comments
Labels
Memory memory usage improvements

Comments

@ValarDragon
Copy link
Member

Fractal preprocesses the matrices, and the prover must store codewords output by the preprocessing. Currently the library keeps these in memory.

(Recall a codeword is the evaluation of a polynomial over a known domain, so in terms of serialization it is a vector of Field elements)

We should store the prover indexes codewords on disk. Additionally, if we store the raw polynomial the codeword represents, in addition to the codeword, then the prover should be able to get away with never loading these codewords into memory. Then the prover would only require random access to the codewords.

Then when constructing the proof, the codewords entries can be read directly from disk. (Maybe even via mmap if we want lower memory filecoin-project/rust-fil-proofs#986)

This will be crucial for large deployments of Fractal, due to the memory reduction. (Additionally, these codewords can be stored 'by a helper' for the helped setting)

@ValarDragon ValarDragon added the Memory memory usage improvements label Aug 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Memory memory usage improvements
Projects
None yet
Development

No branches or pull requests

1 participant