From 845041239c94532f3c6e4fc275bd13b79a614376 Mon Sep 17 00:00:00 2001 From: Tobias-Brock Date: Thu, 28 Mar 2024 12:19:18 +0100 Subject: [PATCH] Add new chapters to regu and advrisk and add jupyter notebook solution for nested resampling exercise --- .../11_advriskmin/11-03-regression-l2-l1.md | 15 +++++++++++++++ .../chapters/11_advriskmin/11-03-regression-l2.md | 15 --------------- ...sion-l1.md => 11-04-regression-l1-deepdive.md} | 4 ++-- content/chapters/15_regularization/15-02-l1l2.md | 15 --------------- content/chapters/15_regularization/15-02-l2.md | 15 +++++++++++++++ content/chapters/15_regularization/15-03-l1.md | 15 +++++++++++++++ .../{15-03-l1vsl12.md => 15-04-l1vsl12.md} | 4 ++-- .../{15-04-enetlogreg.md => 15-05-enetlogreg.md} | 4 ++-- content/chapters/15_regularization/15-05-l0.md | 15 --------------- content/chapters/15_regularization/15-06-other.md | 15 +++++++++++++++ .../chapters/15_regularization/15-07-nonlin.md | 15 +++++++++++++++ .../{15-06-nonlin-bayes.md => 15-08-bayes.md} | 6 +++--- ...-geom-l2-wdecay.md => 15-09-geom-l2-wdecay.md} | 4 ++-- .../{15-08-geom-l1.md => 15-10-geom-l1.md} | 4 ++-- ...-early-stopping.md => 15-11-early-stopping.md} | 4 ++-- .../15_regularization/15-12-ridge-deep.md | 11 +++++++++++ .../{15-09-lasso-deep.md => 15-13-lasso-deep.md} | 4 ++-- .../15_regularization/15-14-bagging-deep.md | 11 +++++++++++ content/exercises/_index.md | 2 +- 19 files changed, 115 insertions(+), 63 deletions(-) create mode 100644 content/chapters/11_advriskmin/11-03-regression-l2-l1.md delete mode 100644 content/chapters/11_advriskmin/11-03-regression-l2.md rename content/chapters/11_advriskmin/{11-04-regression-l1.md => 11-04-regression-l1-deepdive.md} (77%) delete mode 100644 content/chapters/15_regularization/15-02-l1l2.md create mode 100644 content/chapters/15_regularization/15-02-l2.md create mode 100644 content/chapters/15_regularization/15-03-l1.md rename content/chapters/15_regularization/{15-03-l1vsl12.md => 15-04-l1vsl12.md} (81%) rename content/chapters/15_regularization/{15-04-enetlogreg.md => 15-05-enetlogreg.md} (81%) delete mode 100644 content/chapters/15_regularization/15-05-l0.md create mode 100644 content/chapters/15_regularization/15-06-other.md create mode 100644 content/chapters/15_regularization/15-07-nonlin.md rename content/chapters/15_regularization/{15-06-nonlin-bayes.md => 15-08-bayes.md} (68%) rename content/chapters/15_regularization/{15-07-geom-l2-wdecay.md => 15-09-geom-l2-wdecay.md} (85%) rename content/chapters/15_regularization/{15-08-geom-l1.md => 15-10-geom-l1.md} (81%) rename content/chapters/15_regularization/{15-10-early-stopping.md => 15-11-early-stopping.md} (84%) create mode 100644 content/chapters/15_regularization/15-12-ridge-deep.md rename content/chapters/15_regularization/{15-09-lasso-deep.md => 15-13-lasso-deep.md} (77%) create mode 100644 content/chapters/15_regularization/15-14-bagging-deep.md diff --git a/content/chapters/11_advriskmin/11-03-regression-l2-l1.md b/content/chapters/11_advriskmin/11-03-regression-l2-l1.md new file mode 100644 index 00000000..5033b80e --- /dev/null +++ b/content/chapters/11_advriskmin/11-03-regression-l2-l1.md @@ -0,0 +1,15 @@ +--- +title: "Chapter 11.03: L2- and L1 Loss" +weight: 11003 +--- +In this section, we revisit the \\(L2\\) and \\(L1\\) loss and present their risk minimizers -- the conditional mean -- and optimal constant model -- the empirical mean of observed target values is derived for the \\(L2\\) loss. The conditional median -- and optimal constant model -- the empirical median of observed target values is introduced for the \\(L1\\) loss. + + + +### Lecture video + +{{< video id="agQQzTI_6HI" >}} + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-advriskmin-regression-l2-l1.pdf" >}} diff --git a/content/chapters/11_advriskmin/11-03-regression-l2.md b/content/chapters/11_advriskmin/11-03-regression-l2.md deleted file mode 100644 index 2a02c321..00000000 --- a/content/chapters/11_advriskmin/11-03-regression-l2.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: "Chapter 11.03: L2 Loss" -weight: 11003 ---- -In this section, we revisit \\(L2\\) loss and derive its risk minimizer -- the conditional mean -- and optimal constant model -- the empirical mean of observed target values. - - - -### Lecture video - -{{< video id="agQQzTI_6HI" >}} - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-advriskmin-regression-l2.pdf" >}} diff --git a/content/chapters/11_advriskmin/11-04-regression-l1.md b/content/chapters/11_advriskmin/11-04-regression-l1-deepdive.md similarity index 77% rename from content/chapters/11_advriskmin/11-04-regression-l1.md rename to content/chapters/11_advriskmin/11-04-regression-l1-deepdive.md index 12810765..7d579cda 100644 --- a/content/chapters/11_advriskmin/11-04-regression-l1.md +++ b/content/chapters/11_advriskmin/11-04-regression-l1-deepdive.md @@ -1,5 +1,5 @@ --- -title: "Chapter 11.04: L1 Loss" +title: "Chapter 11.04: L1 Loss Deepdive" weight: 11004 --- In this section, we revisit \\(L1\\) loss and derive its risk minimizer -- the conditional median -- and optimal constant model -- the empirical median of observed target values. @@ -12,4 +12,4 @@ In this section, we revisit \\(L1\\) loss and derive its risk minimizer -- the c ### Lecture slides -{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-advriskmin-regression-l1.pdf" >}} +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-advriskmin-regression-l1-deepdive.pdf" >}} diff --git a/content/chapters/15_regularization/15-02-l1l2.md b/content/chapters/15_regularization/15-02-l1l2.md deleted file mode 100644 index 0787638c..00000000 --- a/content/chapters/15_regularization/15-02-l1l2.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: "Chapter 15.02: Lasso and Ridge Regression" -weight: 15002 ---- -We introduce Lasso and Ridge regression as the key approaches to regularizing linear models. - - - -### Lecture video - -{{< video id="yeN-xRfheYU" >}} - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-l1l2.pdf" >}} diff --git a/content/chapters/15_regularization/15-02-l2.md b/content/chapters/15_regularization/15-02-l2.md new file mode 100644 index 00000000..264e5abe --- /dev/null +++ b/content/chapters/15_regularization/15-02-l2.md @@ -0,0 +1,15 @@ +--- +title: "Chapter 15.02: Ridge Regression" +weight: 15002 +--- +We introduce Ridge regression as a key approach to regularizing linear models. + + + +### Lecture video + +{{< video id="yeN-xRfheYU" >}} + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-l2.pdf" >}} diff --git a/content/chapters/15_regularization/15-03-l1.md b/content/chapters/15_regularization/15-03-l1.md new file mode 100644 index 00000000..485e1379 --- /dev/null +++ b/content/chapters/15_regularization/15-03-l1.md @@ -0,0 +1,15 @@ +--- +title: "Chapter 15.03: Lasso Regression" +weight: 15003 +--- +We introduce Lasso regression as a key approach to regularizing linear models. + + + +### Lecture video + +{{< video id="yeN-xRfheYU" >}} + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-l1.pdf" >}} diff --git a/content/chapters/15_regularization/15-03-l1vsl12.md b/content/chapters/15_regularization/15-04-l1vsl12.md similarity index 81% rename from content/chapters/15_regularization/15-03-l1vsl12.md rename to content/chapters/15_regularization/15-04-l1vsl12.md index a05c23a6..e8603f38 100644 --- a/content/chapters/15_regularization/15-03-l1vsl12.md +++ b/content/chapters/15_regularization/15-04-l1vsl12.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.03: Lasso vs Ridge Regression" -weight: 15003 +title: "Chapter 15.04: Lasso vs Ridge Regression" +weight: 15004 --- This section provides a detailed comparison between Lasso and Ridge regression. diff --git a/content/chapters/15_regularization/15-04-enetlogreg.md b/content/chapters/15_regularization/15-05-enetlogreg.md similarity index 81% rename from content/chapters/15_regularization/15-04-enetlogreg.md rename to content/chapters/15_regularization/15-05-enetlogreg.md index e5d63f66..1bc7f280 100644 --- a/content/chapters/15_regularization/15-04-enetlogreg.md +++ b/content/chapters/15_regularization/15-05-enetlogreg.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.04: Elastic Net and Regularization for GLMs" -weight: 15004 +title: "Chapter 15.05: Elastic Net and Regularization for GLMs" +weight: 15005 --- In this section, we introduce the elastic net as a combination of Ridge and Lasso regression and discuss regularization for logistic regression. diff --git a/content/chapters/15_regularization/15-05-l0.md b/content/chapters/15_regularization/15-05-l0.md deleted file mode 100644 index acc665db..00000000 --- a/content/chapters/15_regularization/15-05-l0.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: "Chapter 15.05: L0 Regularization" -weight: 15005 ---- -In this section, we introduce \\(LQ\\) regularization and particularly discuss \\(L0\\) regularization as an important special case besides \\(L1\\) and \\(L2\\) that penalizes the number of non-zero parameters. - - - -### Lecture video - -{{< video id="gw6yLFoQzdQ" >}} - -### Lecture slides - -{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-l0.pdf" >}} diff --git a/content/chapters/15_regularization/15-06-other.md b/content/chapters/15_regularization/15-06-other.md new file mode 100644 index 00000000..d1a218c0 --- /dev/null +++ b/content/chapters/15_regularization/15-06-other.md @@ -0,0 +1,15 @@ +--- +title: "Chapter 15.06: Other Types of Regularization" +weight: 15006 +--- +In this section, we introduce other regularization approaches besides the important special cases \\(L1\\) and \\(L2\\). + + + +### Lecture video + +{{< video id="gw6yLFoQzdQ" >}} + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-others.pdf" >}} diff --git a/content/chapters/15_regularization/15-07-nonlin.md b/content/chapters/15_regularization/15-07-nonlin.md new file mode 100644 index 00000000..089f2da5 --- /dev/null +++ b/content/chapters/15_regularization/15-07-nonlin.md @@ -0,0 +1,15 @@ +--- +title: "Chapter 15.07: Regularization in NonLinear Models" +weight: 15007 +--- +In this section, we demonstrate regularization in non-linear models like neural networks. + + + +### Lecture video + +{{< video id="MdwK9e2wR_U" >}} + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-nonlin.pdf" >}} diff --git a/content/chapters/15_regularization/15-06-nonlin-bayes.md b/content/chapters/15_regularization/15-08-bayes.md similarity index 68% rename from content/chapters/15_regularization/15-06-nonlin-bayes.md rename to content/chapters/15_regularization/15-08-bayes.md index 45c2dc37..eac3ab2b 100644 --- a/content/chapters/15_regularization/15-06-nonlin-bayes.md +++ b/content/chapters/15_regularization/15-08-bayes.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.06: Regularization in NonLinear Models and Bayesian Priors" -weight: 15006 +title: "Chapter 15.08: Regularization and Bayesian Priors" +weight: 15008 --- In this section, we motivate regularization from a Bayesian perspective, showing how different penalty terms correspond to different Bayesian priors. @@ -12,4 +12,4 @@ In this section, we motivate regularization from a Bayesian perspective, showing ### Lecture slides -{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-nonlin-bayes.pdf" >}} +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-bayes.pdf" >}} diff --git a/content/chapters/15_regularization/15-07-geom-l2-wdecay.md b/content/chapters/15_regularization/15-09-geom-l2-wdecay.md similarity index 85% rename from content/chapters/15_regularization/15-07-geom-l2-wdecay.md rename to content/chapters/15_regularization/15-09-geom-l2-wdecay.md index 6b77f9e0..613d0738 100644 --- a/content/chapters/15_regularization/15-07-geom-l2-wdecay.md +++ b/content/chapters/15_regularization/15-09-geom-l2-wdecay.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.07: Geometric Analysis of L2 Regularization and Weight Decay" -weight: 15007 +title: "Chapter 15.09: Geometric Analysis of L2 Regularization and Weight Decay" +weight: 15009 --- In this section, we provide a geometric understanding of \\(L2\\) regularization, showing how parameters are shrunk according to the eigenvalues of the Hessian of empirical risk, and discuss its correspondence to weight decay. diff --git a/content/chapters/15_regularization/15-08-geom-l1.md b/content/chapters/15_regularization/15-10-geom-l1.md similarity index 81% rename from content/chapters/15_regularization/15-08-geom-l1.md rename to content/chapters/15_regularization/15-10-geom-l1.md index d5f58a1d..1ae0691e 100644 --- a/content/chapters/15_regularization/15-08-geom-l1.md +++ b/content/chapters/15_regularization/15-10-geom-l1.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.08: Geometric Analysis of L1 Regularization" -weight: 15008 +title: "Chapter 15.10: Geometric Analysis of L1 Regularization" +weight: 15010 --- In this section, we provide a geometric understanding of \\(L1\\) regularization and show that it encourages sparsity in the parameter vector. diff --git a/content/chapters/15_regularization/15-10-early-stopping.md b/content/chapters/15_regularization/15-11-early-stopping.md similarity index 84% rename from content/chapters/15_regularization/15-10-early-stopping.md rename to content/chapters/15_regularization/15-11-early-stopping.md index eab9c394..25dfc8b7 100644 --- a/content/chapters/15_regularization/15-10-early-stopping.md +++ b/content/chapters/15_regularization/15-11-early-stopping.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.10: Early Stopping" -weight: 15010 +title: "Chapter 15.11: Early Stopping" +weight: 15011 --- In this section, we introduce early stopping and show how it can act as a regularizer. diff --git a/content/chapters/15_regularization/15-12-ridge-deep.md b/content/chapters/15_regularization/15-12-ridge-deep.md new file mode 100644 index 00000000..7df9bf5d --- /dev/null +++ b/content/chapters/15_regularization/15-12-ridge-deep.md @@ -0,0 +1,11 @@ +--- +title: "Chapter 15.12: Details on Ridge Regression: Deep Dive" +weight: 15012 +--- +In this section, we consider Ridge regression as row-augmentation and as minimizing risk under feature noise. We also discuss the bias-variance tradeoff. + + + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-ridge-deepdive.pdf" >}} diff --git a/content/chapters/15_regularization/15-09-lasso-deep.md b/content/chapters/15_regularization/15-13-lasso-deep.md similarity index 77% rename from content/chapters/15_regularization/15-09-lasso-deep.md rename to content/chapters/15_regularization/15-13-lasso-deep.md index 6a7d8d91..33cf405e 100644 --- a/content/chapters/15_regularization/15-09-lasso-deep.md +++ b/content/chapters/15_regularization/15-13-lasso-deep.md @@ -1,6 +1,6 @@ --- -title: "Chapter 15.09: Soft-thresholding and L1 regularization: Deep Dive" -weight: 15009 +title: "Chapter 15.13: Soft-thresholding and L1 regularization: Deep Dive" +weight: 15013 --- In this section, we prove the previously stated proposition regarding soft-thresholding and L1 regularization. diff --git a/content/chapters/15_regularization/15-14-bagging-deep.md b/content/chapters/15_regularization/15-14-bagging-deep.md new file mode 100644 index 00000000..1a2661af --- /dev/null +++ b/content/chapters/15_regularization/15-14-bagging-deep.md @@ -0,0 +1,11 @@ +--- +title: "Chapter 15.14: Bagging as Regularization Method: Deep Dive" +weight: 15014 +--- +In this section, we consider bagging as a form of regularization. We also discuss which factors influence the effectiveness of bagging. + + + +### Lecture slides + +{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/raw/main/slides-pdf/slides-regu-bagging-deepdive.pdf" >}} diff --git a/content/exercises/_index.md b/content/exercises/_index.md index 46fc4cea..3b3b0187 100644 --- a/content/exercises/_index.md +++ b/content/exercises/_index.md @@ -17,7 +17,7 @@ __Exercises for Chapters 1-10 (LMU Lecture I2ML):__ | Exercise 9  | [Random forests](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/ex_forests.pdf) | [Random forests](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/sol_forests.pdf) | [Random forests](https://github.com/slds-lmu/lecture_i2ml/blob/master/exercises/forests/sol_forests_py.ipynb) | | Exercise 10  | [Neural networks](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/ex_nn.pdf) | [Neural networks](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/sol_nn.pdf) | | Exercise 11  | [Tuning](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/ex_tuning.pdf) | [Tuning](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/sol_tuning.pdf) | [Tuning](https://github.com/slds-lmu/lecture_i2ml/blob/master/exercises/tuning/sol_tuning_py.ipynb) | -| Exercise 12  | [Nested resampling](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/ex_nested_resampling.pdf)  | [Nested resampling](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/sol_nested_resampling.pdf)  | | +| Exercise 12  | [Nested resampling](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/ex_nested_resampling.pdf)  | [Nested resampling](https://github.com/slds-lmu/lecture_i2ml/raw/master/exercises-pdf/sol_nested_resampling.pdf)  | [Tuning](https://github.com/slds-lmu/lecture_i2ml/blob/master/exercises/nested-resampling/sol_nested_resampling_py.ipynb) |