Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add speed notebook #1635

Closed
wants to merge 2 commits into from
Closed

Add speed notebook #1635

wants to merge 2 commits into from

Conversation

blegat
Copy link
Member

@blegat blegat commented Nov 19, 2018

It helps to see where time is spent and allocations are done thanks to TimerOutputs to help resolve the regressions at #1403.
For P-median:
With Julia v1.0.2 and JuMP and MOI master:

────────────────────────────────────────────────────────────────────────────
                                     Time                   Allocations      
                             ──────────────────────   ───────────────────────
      Tot / % measured:           2.22s / 99.5%           1.01GiB / 100%     

 Section             ncalls     time   %tot     avg     alloc   %tot      avg
 ────────────────────────────────────────────────────────────────────────────
 P median                 1    2.21s   100%   2.21s   1.01GiB  100%   1.01GiB
   constraint             1    1.61s  73.0%   1.61s    898MiB  86.8%   898MiB
     customer loop        1    1.61s  72.9%   1.61s    897MiB  86.7%   897MiB
       facility         100    1.44s  65.4%  14.4ms    848MiB  82.0%  8.48MiB
       closest          100    166ms  7.53%  1.66ms   48.9MiB  4.73%   501KiB
     sum facility         1    658μs  0.03%   658μs    870KiB  0.08%   870KiB
   variable               1    402ms  18.2%   402ms    104MiB  10.1%   104MiB
     closest              1    401ms  18.2%   401ms    103MiB  9.93%   103MiB
     facility             1   1.12ms  0.05%  1.12ms   1.34MiB  0.13%  1.34MiB
   objective              1    194ms  8.81%   194ms   32.6MiB  3.15%  32.6MiB
   model                  1   25.8μs  0.00%  25.8μs   22.3KiB  0.00%  22.3KiB
   constants              1   20.8μs  0.00%  20.8μs   0.98KiB  0.00%  0.98KiB
 ────────────────────────────────────────────────────────────────────────────

With JuMP v0.18

 ────────────────────────────────────────────────────────────────────────────
                                     Time                   Allocations      
                             ──────────────────────   ───────────────────────
      Tot / % measured:           686ms / 73.7%            269MiB / 100%     

 Section             ncalls     time   %tot     avg     alloc   %tot      avg
 ────────────────────────────────────────────────────────────────────────────
 P median                 1    506ms   100%   506ms    269MiB  100%    269MiB
   constraint             1    432ms  85.4%   432ms    210MiB  77.9%   210MiB
     customer loop        1    432ms  85.4%   432ms    209MiB  77.9%   209MiB
       facility         100    328ms  64.9%  3.28ms    184MiB  68.5%  1.84MiB
       closest          100    104ms  20.5%  1.04ms   25.1MiB  9.35%   257KiB
     sum facility         1   40.6μs  0.01%  40.6μs    118KiB  0.04%   118KiB
   variable               1   60.4ms  11.9%  60.4ms   49.3MiB  18.4%  49.3MiB
     closest              1   60.0ms  11.9%  60.0ms   48.4MiB  18.0%  48.4MiB
     facility             1    344μs  0.07%   344μs   0.98MiB  0.37%  0.98MiB
   objective              1   13.5ms  2.66%  13.5ms   10.0MiB  3.72%  10.0MiB
   constants              1   20.4μs  0.00%  20.4μs   0.98KiB  0.00%  0.98KiB
   model                  1   7.79μs  0.00%  7.79μs   4.37KiB  0.00%  4.37KiB
 ────────────────────────────────────────────────────────────────────────────

For cont5:
With Julia v1.0.2 and JuMP and MOI master:

──────────────────────────────────────────────────────────────────────────────
                                       Time                   Allocations      
                               ──────────────────────   ───────────────────────
       Tot / % measured:            1.18s / 98.7%            452MiB / 100%     

 Section               ncalls     time   %tot     avg     alloc   %tot      avg
 ──────────────────────────────────────────────────────────────────────────────
 cont5                      1    1.17s   100%   1.17s    452MiB  100%    452MiB
   constraint               1    923ms  79.1%   923ms    360MiB  79.6%   360MiB
     PDE                    1    921ms  78.9%   921ms    357MiB  79.0%   357MiB
     Boundary condi...      1   1.88ms  0.16%  1.88ms   2.04MiB  0.45%  2.04MiB
     Initial condit...      1    459μs  0.04%   459μs    540KiB  0.12%   540KiB
   variable                 1    244ms  20.9%   244ms   92.3MiB  20.4%  92.3MiB
     y                      1    244ms  20.9%   244ms   92.3MiB  20.4%  92.3MiB
     u                      1    284μs  0.02%   284μs   74.4KiB  0.02%  74.4KiB
   constants                1   54.9μs  0.00%  54.9μs   35.5KiB  0.01%  35.5KiB
   model                    1   25.2μs  0.00%  25.2μs   22.3KiB  0.00%  22.3KiB
   objective                1   3.07μs  0.00%  3.07μs         -  0.00%        -
 ──────────────────────────────────────────────────────────────────────────────

With JuMP v0.18 (replacing lower_bound by lowerbound and upper_bound by upperbound)

──────────────────────────────────────────────────────────────────────────────
                                       Time                   Allocations      
                               ──────────────────────   ───────────────────────
       Tot / % measured:            648ms / 71.2%            180MiB / 100%     

 Section               ncalls     time   %tot     avg     alloc   %tot      avg
 ──────────────────────────────────────────────────────────────────────────────
 cont5                      1    461ms   100%   461ms    180MiB  100%    180MiB
   constraint               1    414ms  89.8%   414ms    152MiB  84.6%   152MiB
     PDE                    1    413ms  89.7%   413ms    151MiB  84.2%   151MiB
     Boundary condi...      1    598μs  0.13%   598μs    602KiB  0.33%   602KiB
     Initial condit...      1    174μs  0.04%   174μs    180KiB  0.10%   180KiB
   variable                 1   46.8ms  10.1%  46.8ms   27.6MiB  15.4%  27.6MiB
     y                      1   46.7ms  10.1%  46.7ms   27.6MiB  15.3%  27.6MiB
     u                      1   33.9μs  0.01%  33.9μs   23.9KiB  0.01%  23.9KiB
   constants                1   70.9μs  0.02%  70.9μs   35.5KiB  0.02%  35.5KiB
   model                    1   7.97μs  0.00%  7.97μs   4.37KiB  0.00%  4.37KiB
   objective                1   5.03μs  0.00%  5.03μs         -  0.00%        -
 ──────────────────────────────────────────────────────────────────────────────

[ci skip]

It helps to see where time is spent and allocations are done thanks to TimerOutputs
@codecov
Copy link

codecov bot commented Nov 19, 2018

Codecov Report

Merging #1635 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1635   +/-   ##
=======================================
  Coverage   90.86%   90.86%           
=======================================
  Files          28       28           
  Lines        3744     3744           
=======================================
  Hits         3402     3402           
  Misses        342      342

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d61020c...c4369d4. Read the comment docs.

@mlubin
Copy link
Member

mlubin commented Nov 19, 2018

Nice! This is very useful timing output. Is there a reason it needs to be a notebook, or can we get the same table from running a regular script?

@odow
Copy link
Member

odow commented Nov 20, 2018

It can be a regular script. It just uses this package: https://github.com/KristofferC/TimerOutputs.jl

@blegat
Copy link
Member Author

blegat commented Nov 20, 2018

I can be a regular script, I just find it more convenient to debug performance usage as a notebook. If it is a script, it can easily be copy pasted into a notebook so a script is fine.

@mlubin
Copy link
Member

mlubin commented Nov 20, 2018

Plain text is easier to review and edit, so I'd prefer to stick with that unless there's a compelling reason to check in the notebook.

@joaquimg
Copy link
Member

What about using this?
https://github.com/mpastell/Weave.jl
to autogenerate notebooks

@odow
Copy link
Member

odow commented Dec 3, 2020

Closing as out-dated. Benchmarks can be added as performance issues are addressed.

@odow odow closed this Dec 3, 2020
@odow odow deleted the bl/speed_notebook branch December 15, 2020 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants