Synthetic PoRep support

[vmx]

Synthetic PoRep support is a bigger task. But I thought I start the conversation.

Deriving challenges

Deriving challenges for Synthetic PoRep is different from the current PoRep. You can find a spec in the link to the FIP provided above, if you prefer code, it can be found at https://github.com/filecoin-project/rust-fil-proofs/blob/8ce806eebba0dbba2928d34b69ed4d875397cf7c/storage-proofs-porep/src/stacked/vanilla/challenges.rs#L182-L374 (and is pretty self-contained).

In case you code something and want to see if it matches our implementation, you can have a look at the challenges-synth binary in the rust-fil-proofs proofs-bin branch. For 32GiB sectors a run would look like this:

echo '{"comm_r": "0x00112233445566778899aabbccddeeff00112233445566778899aabbccddee00", "num_draw_challenges": 262144, "num_generate_challenges": 1073741824, "replica_id": "0x00112233445566778899aabbccddeeff00112233445566778899aabbccddee00", "sector_size": 34359738368, "seed": "0x00112233445566778899aabbccddeeff00112233445566778899aabbccddee00"}'|RUST_LOG=trace cargo run --release --bin challenges-synth

Serialization

The serialization format is also different, but it's well documented.

Where to start

For me it would be great to have something I can call from Rust as soon as possible. I can use the challenge generation in Rust. So for me it would be great if I would get some API exposed to extract the proofs based on a challenge. Even if it's hacky and might be changed/removed in the future, it would help a lot. That would likely also include the serialization. So the ideal call for me would be something that takes the sector and a set of challenges and writes those to disk serialized in the correct format.

[vmx]

Based on the meeting yesterday, here's what I'd need to get things running:

An API I can call from Rust to extract the Merkle proofs. I would imagine it would take the challenges (the offsets) I'm interested in, it would then returns the corresponding proofs of all trees/labels.

As passing data from/to Rust might be a lot of work, I suggest that it just writes the results into a file in the same format as the Rust implementation does. You can find a description of the format at https://github.com/filecoin-project/rust-fil-proofs/blob/master/storage-proofs-porep/src/stacked/vanilla/params.rs#L431-L450.

[vmx]

As passing data from/to Rust might be a lot of work, I suggest that it just writes the results into a file in the same format as the Rust implementation does. You can find a description of the format at https://github.com/filecoin-project/rust-fil-proofs/blob/master/storage-proofs-porep/src/stacked/vanilla/params.rs#L431-L450.

I just found out that you are already able to write a commit-phase1-output file that I'm able to read on from Rust.. So as a start, it would work if you write a file in that format, just with more challenges (as SynthPoRep demands it). I can then take it from there.

[vmx]

To make this more concrete (sorry, I discover as I go that things are even simpler then I thought). I'd like to have an API where I pass in the challenges (the offset) and it writes out the merkle proofs at those offsets.

It would make sense to use the serialization linked above, as it's way smaller and omits some of the things that are not really needed for C2.