Can Parity Blocks Be Read to Improve Performance in a Non-Degraded Array?

[WynnSmith]

This is a suggestion for the small business and on-site scenarios, with the possibility that a six disc array could be introduced and become as revered as the mirrored array. Considering CPU speeds allow parity blocks to perform at near data block speed, it seems possible to queue up parity block reads with data block reads to improve performance. The performance enhancement of smaller disk arrays in both streaming and iops could be dramatically improved while virtually eliminating the sacrifice of streaming performance for selecting additional levels of redundancy. For example, a small six disk array, implemented with RAIDz3, could read at nearly the speed of a pair of three disk vDevs. From this, a more generalized algorithm might later be developed for larger arrays.

[rincebrain]

The nicest CPU I have laying around, reconstructing raidz3 from 3 missing devices is about 580 MiB/s.

You could conceivably make an argument for that being worth it sometimes, but the scheduling required to ensure you're not bottlenecking by doing it would be very hard, and just optimistically doing it every time would be burning a looooooooot of CPU time.

[rincebrain]

You're asserting a performance gain with no evidence. My remark was that reconstruction from single parity on a single parity only RAID is not free, even if it's XOR, you're now paying it every read, and reconstructing from Q or R in a double or triple parity setup becomes increasingly more expensive because computing those is not purely XOR again, Galois Field math is fun, to say nothing of the fact that I believe the parity isn't read in normal reads unless the checksum fails, so for things that aren't long stretches of sequential reads where it might just read the parity because skipping 32k in the middle of a 30M+ sequential read doesn't make much sense, you'd potentially be adding a nontrivial number of reads to each disk with parity on it, further complicating deciding when this might be a good outcome.

So whether it's XOR or even more expensive things, it's not free to do every time, and if you're going to try to avoid doing it every time, I believe you would need a lot of understanding of how "much" CPU is available on the system over time and the performance of the individual disks to speculate when it might be a gain versus causing significant bottlenecks. If someone is using SHA256 without SHA-NI or the ARM SHA extensions, for example, or high zstd or gzip levels, to say nothing of if the system is doing anything but serving data from disks, how much tolerance you have for burning CPU time, as well as how much it costs, is going to vary significantly.

So I think this would be a performance loss in many cases, and trying to decide when it might be a gain would involve a lot of information from across the system. I don't think it's impossible to collect, necessarily, but I think it'd be a lot of work to do in order to prove this is viable, and I suspect you will find if you just modified the code to always do the parity reconstruction that it's a significant overhead to do without such logistics.

I could be wrong, of course, without a concrete implementation to inspect it's all just one person's opinions versus another's, but that's why I don't personally find the idea compelling enough to go work up a proof of concept.

[WynnSmith]

Ivan, Rince, thank you for the explanation. Your advanced knowledge of the subject naturally leads you to think about a generalized scheme in which you recognize many obstacles. I suspect you're also respectful of the Enterprise and Data Center applications where this scheme likely won't provide a benefit but possibly inject a penalty.

Do you remember the loud preaching against software RAID? In which years since then, has CPU design and performance failed to advance? I remember when my graphics product, at Intel, was cancelled because "when that 16 Mhz 386 hits the market, it'll be so fast there won't be any need for a graphics accelerator".

My focus is on Small Business, department, field office, and graphics work teams applications, and the possibility of an accelerate scheme which is NOT generalized but focused on a very specific disk array. Since many chassis for disk arrays are offered with multiples of six bays, the six disk array can attractively be seen as a special circumstance. Even this can't be generalized to RAIDz1, RAIDz2, and RAIDz3. Rather, RAIDz3 with six disk vDevs is the special circumstance for an accelerate scheme.

Why would anyone consider a six disk array using RAIDz3? Because we've frequently witnessed a chassis full of disks, of the same brand, purchased at the same time, fail multiple disks before a resilver could be accomplished. Sets of cherished Mirror vDevs CAN'T provide an equivalent protection level. (obligatory backup story goes here) A particular advantage of a six disk vDev is that such a chassis can conveniently accomodate multiple vDevs.
 
So, normally the disk array is not in a degraded state, otherwise, the accelerate scheme is turned off. Various sets of three disk jobs would be recognized as fast or slow submits. The simple logic here is that when we query the remaining queue size we can choose to limit requests to fast submits rather than next-available.

The six disk array can be elevated as a special circumstance just as the mirror set is a special circumstance. By focusing on it, a new "cherished" array can be introduced, studied, and just maybe or maybe not, be generalized in the future.

[nwf]

It could be helpful, perhaps, @WynnSmith, if you could give a mocked up trace of events that you wish could happen during normal and degraded operations. A very concrete narrative that began with something like "Suppose we are trying to read two stripes, S and T, out of a 6-disk raidz3; because S and T each have parity blocks that are not, under ordinary circumstances, read, then the read of S may trigger reads on disks A, B and C, while the read of T may trigger reads on disks D, E, and F. It could be that ..., and then ..., or we could ..." might have the most benefit.

[WynnSmith]

I've solved it. I just spent the past week doing more math than I've ever done in a month (or year?).

What I've proven is a way to record redundant data to a disk array in which the decoding is accomplished with primitive operations. I've proven two different schemes and believe larger schemes are possible.

In the 2+ scheme it's possible to start with two disks and then later add a third or fourth disk for redundancy and performance. What differentiates it from the striped mirror is that rather than a 67% chance of data integrity in a random two disk failure, my 2+2 scheme provides 100% data integrity if any two disks fail. Performance is equal to the striped mirror. In the 2+1 array, performance would be 150% of a three disk RAIDz1 for read operations and 200% performance of a four disk RAIDz2.

In the 3+ scheme it's possible to begin with three disks and then later add a fourth, fifth, or sixth disk for redundancy and performance. With six disks any three can fail while data integrity is preserved across the remaining three.

Because decoding is accomplished with primitive operations, the six disk array can be treated as two vDevs during read operations, doubling the performance of RAIDz3. In the 3+1 array performance would be 133% of RAIDz1 for read operations. In the 3+2 array, performance would be 167% of RAIDz2 for read operations.

The purpose of this message is to put a time stamp on my invention.

[WynnSmith]

Yesterday, I met with my patent attorney. Stay tuned...

[jumbi77]

In wider context this approach increase performance in raidz configs, at leat for small blocks like metdata. It never got landed though:

PR:
#5095

Ticket description:
#5096