Reduce risk of uninit ac in luma_ac and encode_coeffs

[kornelski]

luma_ac

In various pred_cfl_ac I've added checks that ac[..].len() must be exactly as long as needed, i.e. plane_bsize.width() * plane_bsize.height(). The multiplication got repetitive, so I've added plane_bsize.area().

Those prediction functions are in assembly. I haven't checked whether they initialize exactly plane_bsize.area() and nothing else, but I don't see a reason why they wouldn't, it would be a bug otherwise. Anyway, it's not a safety issue in pratice, since the allocated memory behind ac is the same size as before.

encode_coeffs

get_nz_map_contexts initializes context up to eob which is u16. Other code used usize for eob. To be type-safe sure it can't be longer than u16 anywhere (therefore reading past what get_nz_map_contexts could init), I've changed eob to be consistently u16 everywhere. It's never more than 1024 in reality.

get_nz_map_contexts used to write coeffs[scan[i]], but that had two issues:

1. It could not guarantee that all coeffs are initialized and contiguous to make an initialized slice.
2. It was unnecessary performance cost. Writing at scan's positions was a needless step, because they were also always read as coeffs[scan[i]]. Since all uses of the coeffs wanted a coefficient for scan[i], I've made them get it from coeffs[i] rather than coeffs[scan[i]]. scan acted only as an unnecessary shuffle. This is faster, guarantees a contiguous slice, and does not change the behavior.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files	Coverage Δ
src/asm/shared/predict.rs	99.35% <100.00%> (+0.02%)	⬆️
src/asm/shared/transform/inverse.rs	100.00% <100.00%> (ø)
src/asm/x86/predict.rs	98.89% <100.00%> (+<0.01%)	⬆️
src/asm/x86/quantize.rs	98.63% <100.00%> (+<0.01%)	⬆️
src/asm/x86/transform/inverse.rs	100.00% <100.00%> (ø)
src/context/block_unit.rs	93.73% <100.00%> (+0.02%)	⬆️
src/context/transform_unit.rs	89.90% <100.00%> (+0.11%)	⬆️
src/encoder.rs	86.93% <100.00%> (ø)
src/partition.rs	75.46% <100.00%> (+0.12%)	⬆️
src/predict.rs	94.89% <100.00%> (+0.04%)	⬆️
... and 4 more

📢 Thoughts on this report? Let us know!.

[lu-zero]

It looks very nice, but I'd like to have @barrbrain opinion :)

[kornelski]

Thanks for x86 fixes. As you can see, I'm on ARM, and conditionally-compiled x86 is a blind spot for me.

[vibhoothi]

maybe doing a quick AWCY run is a good idea too:)

[kornelski]

I don't expect this to change outputs at all. Hopefully tests already cover this, but you can verify by running compression and check file hashes before/after.

[barrbrain]

maybe doing a quick AWCY run is a good idea too:)

Unfortunately, we have build issues on AWCY presently.
I have confirmed unchanged output on test sequence at default settings:

ffmpeg -loglevel error -f lavfi -i testsrc=duration=10:size=320x240:rate=30 \
  -pix_fmt yuv420p -f yuv4mpegpipe - | rav1e -o >(sha256sum) -y -

[barrbrain]

The changes relating to scan order required close review. Everything else looked good to me at a cursory glance. Thank you for this effort.