Compare with Symbolics

[gdalle]

• Add a comparison to the test suite for the brusselator example
• Add a benchmarking file to play around, and the results are convincing!

julia> benchmark_brusselator(6, :tracer)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  128.471 μs …   5.457 ms  ┊ GC (min … max):  0.00% … 94.62%
 Time  (median):     143.145 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   170.058 μs ± 313.636 μs  ┊ GC (mean ± σ):  12.42% ±  6.53%

  ▃▅▇█▇▆▅▄▃▂▂▁                                                  ▂
  █████████████▇█▇▆▆▆▆▅▄▆▄▅▄▅▄▃▄▇▆▅▆▄▆▆▆▆▅▇▆▆▅▅▄▃▄▄▄▄▁▁▁▄▃▅▁▅▄▆ █
  128 μs        Histogram: log(frequency) by time        361 μs <

 Memory estimate: 261.39 KiB, allocs estimate: 2408.

julia> benchmark_brusselator(6, :symbolics)
BenchmarkTools.Trial: 1219 samples with 1 evaluation.
 Range (min … max):  3.390 ms … 17.434 ms  ┊ GC (min … max): 0.00% … 68.25%
 Time  (median):     3.662 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   4.097 ms ±  1.471 ms  ┊ GC (mean ± σ):  7.88% ± 13.68%

  ▃██▆▄▂▁▁                                                    
  ████████▇█▇▇▅▆▆▄▇▆▅▅▅▅▄▅▅▅▅▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▅▇▇▇▇█▆▄▄ █
  3.39 ms      Histogram: log(frequency) by time     9.84 ms <

 Memory estimate: 2.86 MiB, allocs estimate: 41727.

julia> benchmark_brusselator(24, :tracer)
BenchmarkTools.Trial: 1411 samples with 1 evaluation.
 Range (min … max):  2.379 ms … 14.906 ms  ┊ GC (min … max):  0.00% … 74.13%
 Time  (median):     2.952 ms              ┊ GC (median):     0.00%
 Time  (mean ± σ):   3.531 ms ±  1.703 ms  ┊ GC (mean ± σ):  12.31% ± 17.09%

   ▆▇█▇▅▄▃▂▂▁▂ ▁                                             ▁
  ▆███████████████▅▆▇▆▆▆▅▁▅▅▁▄▁▅▄▆▆▆▄▄▆▆█▆▇▅▆▆▆▅▆▄▆▅▄▄▅▄▄▁▆▅ █
  2.38 ms      Histogram: log(frequency) by time     10.8 ms <

 Memory estimate: 4.07 MiB, allocs estimate: 38060.

julia> benchmark_brusselator(24, :symbolics)
BenchmarkTools.Trial: 71 samples with 1 evaluation.
 Range (min … max):  62.836 ms … 85.328 ms  ┊ GC (min … max): 0.00% … 14.08%
 Time  (median):     70.235 ms              ┊ GC (median):    9.92%
 Time  (mean ± σ):   70.637 ms ±  3.876 ms  ┊ GC (mean ± σ):  8.66% ±  3.82%

                       █ ▁  ▂                                  
  ▃▃▃▃▃▁▃▁▃▃▁▃▁▁▁▃▅▃▃█▃█▆██▆█▅▆▆▁▅▃▃▃▃▁▁▁▁▁▃▁▃▁▃▁▁▁▃▁▁▁▃▁▁▁▁▃ ▁
  62.8 ms         Histogram: frequency by time        81.4 ms <

 Memory estimate: 45.80 MiB, allocs estimate: 672702.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.91%. Comparing base (35810e3) to head (dd84ab3).

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #4      +/-   ##
==========================================
+ Coverage   58.53%   60.91%   +2.38%     
==========================================
  Files           5        5              
  Lines          82       87       +5     
==========================================
+ Hits           48       53       +5     
  Misses         34       34              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[adrhill]

Looks like #7 gives another increase in performance:

julia> benchmark_brusselator(6, :tracer)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  61.042 μs …   3.182 ms  ┊ GC (min … max):  0.00% … 97.34%
 Time  (median):     63.959 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   76.298 μs ± 174.080 μs  ┊ GC (mean ± σ):  14.86% ±  6.34%

        ▂▇▆█▅▂▁                                                 
  ▁▁▁▂▃▆███████▆▅▄▄▃▄▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
  61 μs           Histogram: frequency by time         77.4 μs <

 Memory estimate: 261.19 KiB, allocs estimate: 2404.

julia> benchmark_brusselator(6, :symbolics)
BenchmarkTools.Trial: 2046 samples with 1 evaluation.
 Range (min … max):  2.194 ms …   5.503 ms  ┊ GC (min … max): 0.00% … 57.14%
 Time  (median):     2.307 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   2.445 ms ± 652.214 μs  ┊ GC (mean ± σ):  6.01% ± 12.20%

  ▇▇█▅                                                     ▁   
  ████▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅████ █
  2.19 ms      Histogram: log(frequency) by time      5.39 ms <

 Memory estimate: 2.86 MiB, allocs estimate: 41727.

julia> benchmark_brusselator(24, :tracer)
BenchmarkTools.Trial: 3799 samples with 1 evaluation.
 Range (min … max):  1.053 ms …   4.495 ms  ┊ GC (min … max):  0.00% … 70.75%
 Time  (median):     1.129 ms               ┊ GC (median):     0.00%
 Time  (mean ± σ):   1.315 ms ± 688.047 μs  ┊ GC (mean ± σ):  13.76% ± 17.47%

  ▅█▆▂                                                  ▁     ▁
  ████▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▃▆██████ █
  1.05 ms      Histogram: log(frequency) by time      4.03 ms <

 Memory estimate: 4.07 MiB, allocs estimate: 38056.

julia> benchmark_brusselator(24, :symbolics)
BenchmarkTools.Trial: 122 samples with 1 evaluation.
 Range (min … max):  37.626 ms … 44.862 ms  ┊ GC (min … max):  0.00% … 13.20%
 Time  (median):     41.842 ms              ┊ GC (median):    10.15%
 Time  (mean ± σ):   41.226 ms ±  2.034 ms  ┊ GC (mean ± σ):   8.21% ±  4.68%

  ▃█                                   ▅ ▂                     
  ██▇▁▁▇▁▁▁▁▁▁▁▁▅▁▁▃▁▁▁▁▁▁▅█▅▆▇▆▃▃▅▆▇▇██▆██▇▅▅▆▆▃▅▁▁▅▁▆▁▅▅▃▁▆ ▃
  37.6 ms         Histogram: frequency by time        44.6 ms <

 Memory estimate: 45.80 MiB, allocs estimate: 672702.

CC: @Vaibhavdixit02

[adrhill]

Update from #11:

[ Info: Benchmarking Brusselator of size 6 with tracer...
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  27.000 μs …  2.795 ms  ┊ GC (min … max):  0.00% … 98.14%
 Time  (median):     28.917 μs              ┊ GC (median):     0.00%
 Time  (mean ± σ):   33.634 μs ± 96.856 μs  ┊ GC (mean ± σ):  12.10% ±  4.15%

     ▂▆██▅▁    ▁▂▁                                             
  ▁▂▅███████▆▆▇███▇▆▅▃▃▂▂▂▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▃
  27 μs           Histogram: frequency by time        39.4 μs <

 Memory estimate: 108.19 KiB, allocs estimate: 1252.
 
[ Info: Benchmarking Brusselator of size 6 with symbolics...
BenchmarkTools.Trial: 2253 samples with 1 evaluation.
 Range (min … max):  2.048 ms …   4.927 ms  ┊ GC (min … max): 0.00% … 55.67%
 Time  (median):     2.085 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   2.218 ms ± 574.703 μs  ┊ GC (mean ± σ):  5.77% ± 11.90%

  █▇▁                                                       ▂  
  ███▃▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃██ █
  2.05 ms      Histogram: log(frequency) by time      4.77 ms <

 Memory estimate: 2.86 MiB, allocs estimate: 41727.

[ Info: Benchmarking Brusselator of size 24 with tracer...
BenchmarkTools.Trial: 7975 samples with 1 evaluation.
 Range (min … max):  507.959 μs …   3.436 ms  ┊ GC (min … max):  0.00% … 79.82%
 Time  (median):     545.792 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   626.101 μs ± 416.698 μs  ┊ GC (mean ± σ):  12.28% ± 14.41%

  ██▂                                                        ▁  ▁
  ███▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▃▅▆▇████ █
  508 μs        Histogram: log(frequency) by time       2.88 ms <

 Memory estimate: 2.15 MiB, allocs estimate: 23416.
 
[ Info: Benchmarking Brusselator of size 24 with symbolics...
BenchmarkTools.Trial: 131 samples with 1 evaluation.
 Range (min … max):  34.386 ms … 43.780 ms  ┊ GC (min … max):  0.00% … 12.79%
 Time  (median):     38.694 ms              ┊ GC (median):    10.20%
 Time  (mean ± σ):   38.394 ms ±  2.239 ms  ┊ GC (mean ± σ):   8.32% ±  4.78%

   ▇                       ▂ ▄ █   ▂                           
  ▅█▇▅▁▅▁▃▃▃▃▇▁▁▁▃▃▇█▆▆▇▅▃██▃█▅█▆█▆█▃▃▆▆▃▅▃▃▆▃▁▅▅▁▁▇▁▁▃▁▁▁▁▁▅ ▃
  34.4 ms         Histogram: frequency by time        43.7 ms <

 Memory estimate: 45.80 MiB, allocs estimate: 672702.

[adrhill]

Latest benchmarks including #38 show additional performance on large Brusselator examples (N=100):

julia> include("test/benchmark.jl")

[ Info: Benchmarking Brusselator of size 6 with tracer_bitset...
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  27.083 μs …  2.276 ms  ┊ GC (min … max): 0.00% … 97.82%
 Time  (median):     29.833 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   33.260 μs ± 66.983 μs  ┊ GC (mean ± σ):  7.89% ±  3.88%

         ▁▄▆▇██▇▆▅▅▄▃▃▂▂▂▂▂▂▁▁▂▂▂▁▁▁▁▁                        ▂
  ▃▂▅▅▆▅▆██████████████████████████████▇█████▇███████▇▇▇▆▆▆▆▅ █
  27.1 μs      Histogram: log(frequency) by time      39.6 μs <

 Memory estimate: 108.19 KiB, allocs estimate: 1252.

[ Info: Benchmarking Brusselator of size 6 with tracer_sortedvector...
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  30.458 μs …   3.309 ms  ┊ GC (min … max):  0.00% … 98.38%
 Time  (median):     34.583 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   41.049 μs ± 118.650 μs  ┊ GC (mean ± σ):  13.10% ±  4.48%

             ▅█▄▁                                               
  ▁▁▁▁▁▁▁▁▃▅█████▆▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
  30.5 μs         Histogram: frequency by time         48.5 μs <

 Memory estimate: 133.56 KiB, allocs estimate: 1181.

[ Info: Benchmarking Brusselator of size 6 with symbolics...
BenchmarkTools.Trial: 1577 samples with 1 evaluation.
 Range (min … max):  2.663 ms …   8.267 ms  ┊ GC (min … max): 0.00% … 57.91%
 Time  (median):     2.959 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   3.169 ms ± 867.071 μs  ┊ GC (mean ± σ):  5.94% ± 12.07%

   ▆██▇▆▅▄▂                                                   ▁
  ▄█████████▆▄▄▅▄▁▁▄▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▇▇▇▆▆▅▅▆▅█ █
  2.66 ms      Histogram: log(frequency) by time      7.24 ms <

 Memory estimate: 2.92 MiB, allocs estimate: 43251.

[ Info: Benchmarking Brusselator of size 24 with tracer_bitset...
BenchmarkTools.Trial: 7751 samples with 1 evaluation.
 Range (min … max):  489.292 μs …   3.712 ms  ┊ GC (min … max):  0.00% … 81.57%
 Time  (median):     566.792 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   644.186 μs ± 370.437 μs  ┊ GC (mean ± σ):  10.48% ± 13.87%

  ▂██▆▅▁                                                        ▂
  ██████▇▄▁▃▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▄▄▆▆▇▇▇▆▆▆▇▆▆▅▆▇▆▆ █
  489 μs        Histogram: log(frequency) by time       2.69 ms <

 Memory estimate: 2.15 MiB, allocs estimate: 23416.

[ Info: Benchmarking Brusselator of size 24 with tracer_sortedvector...
BenchmarkTools.Trial: 7497 samples with 1 evaluation.
 Range (min … max):  487.709 μs …   7.073 ms  ┊ GC (min … max):  0.00% … 87.42%
 Time  (median):     553.542 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   665.792 μs ± 569.811 μs  ┊ GC (mean ± σ):  15.45% ± 15.04%

  ██▆▃▁                                                         ▁
  ██████▆▄▅▅▃▁▁▃▃▃▁▁▃▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▄▆▆▆▆▆▇▇▇▇▆▆▆▆▇ █
  488 μs        Histogram: log(frequency) by time       3.85 ms <

 Memory estimate: 2.07 MiB, allocs estimate: 18473.

[ Info: Benchmarking Brusselator of size 24 with symbolics...
BenchmarkTools.Trial: 98 samples with 1 evaluation.
 Range (min … max):  46.705 ms … 56.221 ms  ┊ GC (min … max): 0.00% … 13.23%
 Time  (median):     51.905 ms              ┊ GC (median):    9.51%
 Time  (mean ± σ):   51.248 ms ±  2.495 ms  ┊ GC (mean ± σ):  7.65% ±  4.60%

  ▂ ▅ ▂▅▂                    █▅   ▂ ▅█▅▅ ▅▅   █ █  ▂   ▂       
  █▅█████▁▁▁▅▅▅▁█▁▁▁▁▁▁▁▁▅▅▁███▁▅▅█▅███████▁▅▅█▅█▁▅█▅▅▅█▅▅▁▁▅ ▁
  46.7 ms         Histogram: frequency by time        55.4 ms <

 Memory estimate: 46.69 MiB, allocs estimate: 697551.

[ Info: Benchmarking Brusselator of size 100 with tracer_bitset...
BenchmarkTools.Trial: 106 samples with 1 evaluation.
 Range (min … max):  30.303 ms … 146.411 ms  ┊ GC (min … max): 20.69% … 80.80%
 Time  (median):     37.091 ms               ┊ GC (median):    27.55%
 Time  (mean ± σ):   48.313 ms ±  29.633 ms  ┊ GC (mean ± σ):  42.71% ± 18.65%

   ▂▄█▃▁                                                        
  ▅█████▅▁▁▁▄▄▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▃▃▃▄▃▃ ▃
  30.3 ms         Histogram: frequency by time          134 ms <

 Memory estimate: 147.34 MiB, allocs estimate: 426016.

[ Info: Benchmarking Brusselator of size 100 with tracer_sortedvector...
BenchmarkTools.Trial: 430 samples with 1 evaluation.
 Range (min … max):   9.444 ms … 14.316 ms  ┊ GC (min … max):  0.00% … 26.54%
 Time  (median):     10.685 ms              ┊ GC (median):     0.00%
 Time  (mean ± σ):   11.648 ms ±  1.643 ms  ┊ GC (mean ± σ):  14.00% ± 13.23%

       ▁ ▁▃█▂                                                  
  ▂▃▃▆▄███████▄▄▃▅▃▃▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▃▃▃▄▅▅▄▅▅▅▆▆▅▅▅▆▇▄▅▆▃▅▃▅ ▃
  9.44 ms         Histogram: frequency by time        14.2 ms <

 Memory estimate: 35.40 MiB, allocs estimate: 320057.

[ Info: Benchmarking Brusselator of size 100 with symbolics...
BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max):  990.476 ms …   1.125 s  ┊ GC (min … max): 16.44% … 25.83%
 Time  (median):        1.036 s              ┊ GC (median):    18.79%
 Time  (mean ± σ):      1.051 s ± 62.179 ms  ┊ GC (mean ± σ):  21.00% ±  4.39%

  █ █                 █                               █      █  
  █▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁█ ▁
  990 ms          Histogram: frequency by time          1.12 s <

 Memory estimate: 811.00 MiB, allocs estimate: 12200790.
[ Info: Benchmarking NNlib.conv with tracer...
nothing

[adrhill]

Some performance improvements since the last update:

• ~15% faster with BitSet
• ~45% faster with SortedVector

N = 24:

julia> sct_bitset = TracerSparsityDetector();

julia> sct_sortedvector = TracerSparsityDetector(; gradient_tracer_type=GradientTracer{SortedVector{Int}});


julia> @benchmark jacobian_sparsity($f!, $y, $x, $sct_bitset)
BenchmarkTools.Trial: 8904 samples with 1 evaluation.
 Range (min … max):  482.000 μs …   1.846 ms  ┊ GC (min … max): 0.00% … 68.73%
 Time  (median):     525.667 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   560.802 μs ± 189.521 μs  ┊ GC (mean ± σ):  6.26% ± 11.87%

   ▆█▃▁                                                         ▁
  ▆████▇▃▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▅▅▇▆▇▇█▇▇▇▇ █
  482 μs        Histogram: log(frequency) by time       1.59 ms <

 Memory estimate: 2.05 MiB, allocs estimate: 22268.

julia> @benchmark jacobian_sparsity($f!, $y, $x, $sct_sortedvector)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  289.083 μs …   2.684 ms  ┊ GC (min … max):  0.00% … 87.37%
 Time  (median):     325.708 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   368.059 μs ± 284.448 μs  ┊ GC (mean ± σ):  11.22% ± 12.22%

  ▆█▂                                                           ▁
  ███▆▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▄▅▅▆▆▇▇▇ █
  289 μs        Histogram: log(frequency) by time       2.31 ms <

 Memory estimate: 1.24 MiB, allocs estimate: 9259.

N=100:

julia> @benchmark jacobian_sparsity($f!, $y, $x, $sct_bitset)
BenchmarkTools.Trial: 118 samples with 1 evaluation.
 Range (min … max):  26.623 ms … 114.751 ms  ┊ GC (min … max): 20.85% … 78.64%
 Time  (median):     31.809 ms               ┊ GC (median):    29.27%
 Time  (mean ± σ):   42.488 ms ±  25.661 ms  ┊ GC (mean ± σ):  44.87% ± 18.65%

    █▂▁                                                         
  ▄▅███▄▂▁▁▁▅▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▅▂ ▂
  26.6 ms         Histogram: frequency by time          108 ms <

 Memory estimate: 145.51 MiB, allocs estimate: 406018.

julia> @benchmark jacobian_sparsity($f!, $y, $x, $sct_sortedvector)
BenchmarkTools.Trial: 793 samples with 1 evaluation.
 Range (min … max):  5.385 ms … 9.506 ms  ┊ GC (min … max):  0.00% … 34.65%
 Time  (median):     5.626 ms             ┊ GC (median):     0.00%
 Time  (mean ± σ):   6.307 ms ± 1.114 ms  ┊ GC (mean ± σ):  11.78% ± 14.14%

   ▅█▇                                                       
  ▅███▇▇▆▃▃▂▃▂▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▂▄▄▄▄▃▄▄▄▃▃▃▃▃▂▂▂▂▃▂▂ ▃
  5.38 ms        Histogram: frequency by time       8.69 ms <

 Memory estimate: 21.05 MiB, allocs estimate: 160057.