added video

Author: mhjensen

doc/LectureNotes/_build/.doctrees/environment.pickle

@@ -1197,9 +1197,10 @@ <h2>Lecture Monday September 30, 2024<a class="headerlink" href="#lecture-monday
 <ol class="simple">
 <li><p>Stochastic Gradient descent with examples and automatic differentiation</p></li>
 <li><p>If we get time, we start with the basics of Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model</p></li>
+<li><p><a class="reference external" href="https://youtu.be/jdJoOrCIdII">Video of lecture</a></p></li>
+<li><p>Whiteboard notes at <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesSeptember30.pdf">https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesSeptember30.pdf</a></p></li>
 </ol>
-<!-- * [Video of lecture](https://youtu.be/75pr3hKY20U) -->
-<!-- * "Whiteboard notes at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesOct5.pdf> --></div>
+</div>
 <div class="section" id="suggested-readings-and-videos">
 <h2>Suggested readings and videos<a class="headerlink" href="#suggested-readings-and-videos" title="Permalink to this headline">¶</a></h2>
 <p><strong>Readings and Videos:</strong></p>
@@ -1288,17 +1289,17 @@ <h2>Simple implementation of GD for OLS, Ridge and Lasso<a class="headerlink" hr
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Parameters for OLS using gradient descent
-[[3.870337  ]
- [3.34871778]
- [4.83363025]]
+[[3.53732988]
+ [4.06277313]
+ [4.51728978]]
 Parameters for Ridge using gradient descent
-[[3.66951679]
- [3.76942272]
- [4.63546236]]
+[[3.62671667]
+ [3.79849321]
+ [4.63207408]]
 Parameters for Lasso using gradient descent
-[[3.65759901]
- [3.87732748]
- [4.58735893]]
+[[4.1547556 ]
+ [2.58013829]
+ [5.20288581]]
 </pre></div>
 </div>
 </div>
@@ -1350,11 +1351,11 @@ <h2>But none of these can compete with Newton’s method<a class="headerlink" hr
 [[4.]
  [3.]
  [5.]]
-0 [-23.78254537] [-30.51512235]
-1 [1.30917499e-14] [-1.01327148e-14]
-2 [9.76996262e-16] [1.28164443e-15]
-3 [-7.28306304e-16] [-8.17359583e-16]
-4 [-7.63833441e-16] [-1.32422234e-15]
+0 [-27.10587277] [-37.81035098]
+1 [1.16209264e-13] [2.08824304e-13]
+2 [8.8817842e-17] [2.62242138e-16]
+3 [-1.77635684e-17] [-1.50600514e-16]
+4 [-3.37507799e-16] [-2.280464e-16]
 beta from own Newton code
 [[4.]
  [3.]
@@ -1696,15 +1697,15 @@ <h2>Code with a Number of Minibatches which varies<a class="headerlink" href="#c
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.12220276]
- [2.95115528]]
-Eigenvalues of Hessian Matrix:[0.31601423 3.93074023]
+[[4.30637636]
+ [2.56947078]]
+Eigenvalues of Hessian Matrix:[0.30000613 4.12600095]
 theta from own gd
-[[4.12220276]
- [2.95115528]]
+[[4.30637636]
+ [2.56947078]]
 theta from own sdg
-[[4.12206013]
- [2.92370959]]
+[[4.34759233]
+ [2.63142731]]
 </pre></div>
 </div>
 <img alt="_images/week40_34_1.png" src="_images/week40_34_1.png" />
@@ -2418,12 +2419,12 @@ <h2>Using Autograd with OLS<a class="headerlink" href="#using-autograd-with-ols"
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.62286539]
- [3.36754671]]
-Eigenvalues of Hessian Matrix:[0.30134885 4.08702578]
+[[3.88064505]
+ [2.99072374]]
+Eigenvalues of Hessian Matrix:[0.30141906 4.73853838]
 theta from own gd
-[[3.62286539]
- [3.36754671]]
+[[3.88064505]
+ [2.99072374]]
 </pre></div>
 </div>
 <img alt="_images/week40_100_1.png" src="_images/week40_100_1.png" />
@@ -2494,73 +2495,73 @@ <h2>Same code but now with momentum gradient descent<a class="headerlink" href="
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
 [[4.]
  [3.]]
-Eigenvalues of Hessian Matrix:[0.27124938 4.56804783]
-0 [-8.64009263] [-10.94092702]
-1 [0.18925625] [-0.15527976]
-2 [0.17801827] [-0.14605929]
-3 [0.16744759] [-0.13738633]
-4 [0.1575046] [-0.12922837]
-5 [0.14815202] [-0.12155483]
-6 [0.1393548] [-0.11433693]
-7 [0.13107995] [-0.10754764]
-8 [0.12329646] [-0.10116149]
-9 [0.11597515] [-0.09515455]
-10 [0.10908858] [-0.0895043]
-11 [0.10261093] [-0.08418956]
-12 [0.09651792] [-0.07919041]
-13 [0.09078672] [-0.0744881]
-14 [0.08539583] [-0.07006502]
-15 [0.08032505] [-0.06590458]
-16 [0.07555537] [-0.06199119]
-17 [0.07106891] [-0.05831017]
-18 [0.06684886] [-0.05484773]
-19 [0.06287939] [-0.05159088]
-20 [0.05914563] [-0.04852743]
-21 [0.05563358] [-0.04564589]
-22 [0.05233008] [-0.04293545]
-23 [0.04922273] [-0.04038595]
-24 [0.0462999] [-0.03798785]
-25 [0.04355062] [-0.03573214]
-26 [0.0409646] [-0.03361037]
-27 [0.03853213] [-0.0316146]
-28 [0.03624411] [-0.02973733]
-29 [0.03409194] [-0.02797154]
+Eigenvalues of Hessian Matrix:[0.31702609 3.84351715]
+0 [-13.26083712] [-12.67834752]
+1 [-0.55020405] [0.5257011]
+2 [-0.50482139] [0.48233952]
+3 [-0.46318204] [0.44255455]
+4 [-0.42497724] [0.40605118]
+5 [-0.38992371] [0.37255873]
+6 [-0.3577615] [0.34182884]
+7 [-0.32825214] [0.31363366]
+8 [-0.30117681] [0.28776411]
+9 [-0.27633474] [0.26402837]
+10 [-0.25354173] [0.24225043]
+11 [-0.23262877] [0.22226881]
+12 [-0.21344077] [0.20393534]
+13 [-0.19583547] [0.18711407]
+14 [-0.17968231] [0.17168028]
+15 [-0.16486151] [0.15751952]
+16 [-0.15126318] [0.14452678]
+17 [-0.13878649] [0.13260573]
+18 [-0.12733892] [0.12166797]
+19 [-0.11683558] [0.11163239]
+20 [-0.10719859] [0.10242458]
+21 [-0.0983565] [0.09397626]
+22 [-0.09024373] [0.08622478]
+23 [-0.08280012] [0.07911268]
+24 [-0.07597049] [0.0725872]
+25 [-0.06970419] [0.06659997]
+26 [-0.06395476] [0.06110658]
+27 [-0.05867956] [0.0560663]
+28 [-0.05383947] [0.05144177]
+29 [-0.04939861] [0.04719868]
 theta from own gd
-[[4.11822173]
- [2.90300219]]
-0 [0.03206757] [-0.0263106]
-1 [0.03016341] [-0.02474828]
-2 [0.02780107] [-0.02281004]
-3 [0.02544154] [-0.02087411]
-4 [0.02322297] [-0.01905384]
-5 [0.02117843] [-0.01737634]
-6 [0.0193075] [-0.01584129]
-7 [0.01759974] [-0.01444013]
-8 [0.01604235] [-0.01316233]
-9 [0.01462254] [-0.01199741]
-10 [0.01332832] [-0.01093553]
-11 [0.01214862] [-0.00996762]
-12 [0.01107333] [-0.00908537]
-13 [0.01009321] [-0.00828121]
-14 [0.00919984] [-0.00754823]
-15 [0.00838555] [-0.00688012]
-16 [0.00764333] [-0.00627115]
-17 [0.0069668] [-0.00571608]
-18 [0.00635016] [-0.00521014]
-19 [0.00578809] [-0.00474898]
-20 [0.00527578] [-0.00432864]
-21 [0.00480881] [-0.0039455]
-22 [0.00438318] [-0.00359628]
-23 [0.00399521] [-0.00327797]
-24 [0.00364159] [-0.00298783]
-25 [0.00331927] [-0.00272337]
-26 [0.00302547] [-0.00248232]
-27 [0.00275768] [-0.0022626]
-28 [0.00251359] [-0.00206234]
-29 [0.00229111] [-0.0018798]
+[[3.85703369]
+ [3.13659941]]
+0 [-0.04532405] [0.04330558]
+1 [-0.04158557] [0.03973359]
+2 [-0.03703391] [0.03538463]
+3 [-0.03261373] [0.0311613]
+4 [-0.02859759] [0.02732402]
+5 [-0.02503392] [0.02391906]
+6 [-0.02189994] [0.02092464]
+7 [-0.01915337] [0.01830039]
+8 [-0.01674956] [0.01600363]
+9 [-0.01464686] [0.01399457]
+10 [-0.01280793] [0.01223754]
+11 [-0.01119981] [0.01070103]
+12 [-0.00979357] [0.00935742]
+13 [-0.0085639] [0.00818251]
+14 [-0.00748862] [0.00715512]
+15 [-0.00654835] [0.00625672]
+16 [-0.00572613] [0.00547113]
+17 [-0.00500716] [0.00478417]
+18 [-0.00437846] [0.00418347]
+19 [-0.0038287] [0.00365819]
+20 [-0.00334797] [0.00319887]
+21 [-0.0029276] [0.00279722]
+22 [-0.00256001] [0.002446]
+23 [-0.00223857] [0.00213888]
+24 [-0.0019575] [0.00187032]
+25 [-0.00171171] [0.00163548]
+26 [-0.00149679] [0.00143013]
+27 [-0.00130885] [0.00125057]
+28 [-0.00114451] [0.00109354]
+29 [-0.00100081] [0.00095624]
 theta from own gd wth momentum
-[[4.0076989 ]
- [2.99368326]]
+[[3.99723951]
+ [3.00263755]]
 </pre></div>
 </div>
 </div>
@@ -2649,18 +2650,18 @@ <h2>Including Stochastic Gradient Descent with Autograd<a class="headerlink" hre
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.4870934 ]
- [3.55042779]]
-Eigenvalues of Hessian Matrix:[0.28793787 4.56453869]
+[[3.95935439]
+ [3.09360113]]
+Eigenvalues of Hessian Matrix:[0.32606139 4.1033455 ]
 theta from own gd
-[[3.4870934 ]
- [3.55042779]]
+[[3.95935439]
+ [3.09360113]]
 </pre></div>
 </div>
 <img alt="_images/week40_104_1.png" src="_images/week40_104_1.png" />
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg
-[[3.48900413]
- [3.56763673]]
+[[3.96916206]
+ [3.10276112]]
 </pre></div>
 </div>
 </div>
@@ -2742,15 +2743,15 @@ <h2>Same code but now with momentum gradient descent<a class="headerlink" href="
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.11571507]
- [2.86067818]]
-Eigenvalues of Hessian Matrix:[0.25849623 4.84605507]
+[[4.16532977]
+ [2.86943859]]
+Eigenvalues of Hessian Matrix:[0.29633608 4.22128142]
 theta from own gd
-[[4.10888197]
- [2.86602331]]
+[[4.16445858]
+ [2.87020155]]
 theta from own sdg with momentum
-[[4.15361856]
- [2.92376655]]
+[[4.149433  ]
+ [2.89654756]]
 </pre></div>
 </div>
 </div>
@@ -2819,9 +2820,9 @@ <h2>Similar (second order function now) problem but now with AdaGrad<a class="he
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own AdaGrad
-[[1.99968451]
- [3.001795  ]
- [3.99848215]]
+[[1.99415296]
+ [3.03274273]
+ [3.96850191]]
 </pre></div>
 </div>
 </div>
@@ -2897,9 +2898,9 @@ <h2>RMSprop for adaptive learning rate with Stochastic Gradient Descent<a class=
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own RMSprop
-[[1.99887767]
- [3.01261048]
- [3.99294056]]
+[[2.00022969]
+ [2.99954278]
+ [4.00046631]]
 </pre></div>
 </div>
 </div>
@@ -2979,9 +2980,9 @@ <h2>And finally <a class="reference external" href="https://arxiv.org/pdf/1412.6
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own ADAM
-[[1.99996755]
- [3.00016535]
- [3.99977375]]
+[[1.99991245]
+ [3.00042721]
+ [3.99954434]]
 </pre></div>
 </div>
 </div>
@@ -3102,7 +3103,7 @@ <h3>A warm-up example<a class="headerlink" href="#a-warm-up-example" title="Perm
   return asarray(x, dtype=self.dtype)
 </pre></div>
 </div>
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x117c17be0&gt;]
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11975e550&gt;]
 </pre></div>
 </div>
 <img alt="_images/week40_120_2.png" src="_images/week40_120_2.png" />
@@ -3137,7 +3138,7 @@ <h3>A more advanced example<a class="headerlink" href="#a-more-advanced-example"
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x117893340&gt;
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x11b160a60&gt;
 </pre></div>
 </div>
 <img alt="_images/week40_122_1.png" src="_images/week40_122_1.png" />