Rearrange tests

[dkarrasch]

This brutally splits the triangular tests in two parts, and creates a dedicated test file for unitful tests. That should spread the load more evenly among several workers, when present.

To the best of my knowledge, this doesn't change any test overall. Mostly, this is copy-paste.

[dkarrasch]

The unitful tests take off some heavy load from the bidiag.jl test suite, and the triangular.jl tests include only the initial big (nested) loops. All other parts are moved to triangular2.jl.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.90%. Comparing base (9bc292d) to head (c4774f2).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1157      +/-   ##
==========================================
- Coverage   91.94%   91.90%   -0.05%     
==========================================
  Files          34       34              
  Lines       15352    15354       +2     
==========================================
- Hits        14116    14111       -5     
- Misses       1236     1243       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dkarrasch]

Hm, I missed to check upfront. So the triangular split doesn't yield much, the first block is the very dominant part, so we might as well keep the triangular tests as is. Though any reduction of its runtime is welcome, since this worker runs from start to end and determines the total runtime. I'll check the effect of removing the unitful tests from bidiag.jl tomorrow.

[dkarrasch]

This looks much much better now! Comments welcome.

[inkydragon]

Seems like a huge improvement, this PR reduces test time by ~30% (~6min) by splitting the test set and paying for ~1min of pre-compilation.

• precompile: 12m26s (746s) -> 13m16s (796s)
• test: 20m18.2s (1218s) -> 14m28.5s (868s)
• total: 34m0s -> 30m24s

It also seems that splitting unitful tests can further reduce testing time.
However, with only about 1 minute of optimization room left, it may not be worthwhile to try further.

Test time from windows-x86_64

baseline 959d985

2024-12-23 03:34:37 null	> julia --color=yes --project=.ci --code-coverage=@ -e 'using Pkg; Pkg.instantiate()'
2024-12-23 03:47:33 null	Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
precompile:   12m26s (746s)

Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
LinearAlgebra/triangular           (2) |  1214.98 |  31.50 |  2.6 |  112262.66 |  1906.79
LinearAlgebra/bidiag               (4) |  1037.09 |  18.35 |  1.8 |   61208.18 |  1399.99
LinearAlgebra/diagonal             (9) |   419.65 |   7.20 |  1.7 |   25293.00 |   918.26
LinearAlgebra/dense                (7) |   408.05 |   7.25 |  1.8 |   23990.13 |   864.70
LinearAlgebra/special              (5) |   410.54 |   8.13 |  2.0 |   32843.25 |   959.32

Test Summary: |   Pass  Broken   Total      Time
  Overall     | 134011      67  134078  20m18.2s (1218s)

https://buildkite.com/julialang/linearalgebra-dot-jl/builds/126#0193efdc-f7e2-46f5-8323-a28743eb99af/148-277

this pr

2024-12-24 03:16:05 null	> julia --color=yes --project=.ci --code-coverage=@ -e 'using Pkg; Pkg.instantiate()'
2024-12-24 03:29:21 null	Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
precompile:   13m16s (796s)

Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
LinearAlgebra/unitful              (2) |   736.88 |  12.43 |  1.7 |   40288.82 |   986.48
LinearAlgebra/triangular2          (3) |   629.16 |  15.99 |  2.5 |   62341.18 |  1240.81
LinearAlgebra/triangular3          (4) |   550.91 |  13.66 |  2.5 |   53912.77 |  1093.66
LinearAlgebra/bidiag               (5) |   490.26 |   7.84 |  1.6 |   28772.53 |   974.80
LinearAlgebra/special              (6) |   401.50 |   8.12 |  2.0 |   32676.01 |   960.22

Test Summary: |   Pass  Broken   Total      Time
  Overall     | 133493      67  133560  14m28.5s (868s)

https://buildkite.com/julialang/linearalgebra-dot-jl/builds/137#0193f4f0-a32e-4c28-81e3-61098a555cd8/152-299

[dkarrasch]

I'll take a look, but the unitful tests are again dominated by one block of tests (originally from bidiag.jl), like it was the case for triangular.

[dkarrasch]

I skipped the unitless combination in the loop and removed the Float32 case in the loop. After all, this relies on generic code anyway, so testing through BlasFloats doesn't make much sense. The Int case is interesting, because it requires type promotion underneath the Furlong coat.

EDIT: Reduces runtime of that testset by ~40%.

[dkarrasch]

Any objections against this approach?