Implement Work Partitioner V2

[glyh]

This is part of the project Rework Snark Worker, intended to make job issuing from the Snark work coordinator issue finer-grained jobs, so that more parallelism is possible.

This, although is indeed a noop, will affect later PR where we switch the RPC logic completely. If this is buggy, CI will fail on later PR that actually does the switch.

Mechanism

Basics

The new layer is called a Work_partitioner. It's in function between a Work_selector and a Snark_worker. It's duty is:

• After receiving requests from Snark_worker, try to request the work_selector to get some work to do.
  • If the work is "too big", it will break it into smaller jobs. (Defined here: https://github.com/MinaProtocol/mina/blob/corvo/implement-work-partitioner/src/lib/snark_work_lib/partitioned.ml), it will store most jobs into its state, and only issue one small job to the snark worker;
  • On top of that, when snark worker is actually requesting a work, it should query its own state, and only query the underlying Work_selector if there's no "small jobs" to issue.
• After receiving responses from Snark_worker,
  • Try to combine them together if they're indeed small jobs. And upon completion on the combination. Submit the job. This job should have same shape as it originally requested from an underlying Work_selector.
  • If it's not a small job(some jobs distributed by the work_selector) are already small enough, just return it to the work_selector.

Small jobs

A "small job" is smaller than the original job provided by a Work_selector, in 2 possibilities:

• The Work_selector provide a "`Two" in a One_or_two, in this case, partitioner should distribute 2 specs separately
• The Work_selector distribute a Zkapp command, in this case, partitioner should distribute all base snark jobs, and merge jobs between them separately

Design Issue

• We don't want to not break other parts of the system, most notably GraphQL APIs. We bolt on a new layer atop the Work_selector. Ideally, when there's refactor, we should unite with the Work_selector.

• This layer, doesn't take care of the work reissue of a full job in the Work_selector's perspective, this is because they already have mechanism to do so there. So we only redistribute "small jobs".

• It's assumed that we don't care the order of zkapp command base snarks / merge snarks being performed.

RPCs

Use of assertion when merging 2 Single work to a One_or_two

In the function merge_to_one_result_exn here, we are using assertion. The assumptions should be true unless there's bug in code. Still, this worth some attention.

Several failwith when unwrapping errors in work partitioner

See here.

[glyh]

For now I haven't design priority here. Should I? We kinda have prioritization because we'll never query underlying selector if we ever have pending zkapp commands/single spec on hand. But that's not full priority because there's no priority with in between the zkapp commands.

EDIT:

I'm using a FIFO queue for zkapp command pools. So in theory completion of early arrival commands is prioritized instead of based on completion percentage. This should be fine as long as the expected completion time for each zkapp command is roughly the same.

If we instead prioritize based on completion rate, e.g. we sort by the number of jobs left to complete, there's likely some starving issues. Although we only starve across already pending zkapp commands rather than all works in the pool. Since we prioritize any pending zkapp commands to any unpartitioned work.

Here's the background thread. And I guess this need input from @georgeee.

[glyh]

Thanks @georgeee for drafting this graph during our session :)