Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add in missing chunk on WD and adjust chunk counter #321

Merged
merged 2 commits into from
Nov 26, 2024
Merged

add in missing chunk on WD and adjust chunk counter #321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a3ea5b8
add in missing chunk on WD and adjust chunk counter
chriskolb Nov 26, 2024
4c8f4cb
Update 15-09-wd.md
chriskolb Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions content/chapters/15_regularization/15-09-wd.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
title: "Chapter 15.09: Weight decay and L2"
weight: 15009
---
In this section, we show that L2 regularization with gradient descent is equivalent to weight decay and see how weight decay changes the optimization trajectory.

<!--more-->

### Lecture video

{{< video id="xASHDEAWP0U" >}}

### Lecture slides

{{< pdfjs file="https://github.com/slds-lmu/lecture_sl/blob/main/slides-pdf/slides-regu-wd-vs-l2.pdf" >}}
4 changes: 2 additions & 2 deletions ...apters/15_regularization/15-09-geom-l2.md → ...apters/15_regularization/15-10-geom-l2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Chapter 15.09: Geometry of L2 Regularization"
weight: 15009
title: "Chapter 15.10: Geometry of L2 Regularization"
weight: 15010
---
In this section, we provide a geometric understanding of \\(L2\\) regularization, showing how parameters are shrunk according to the eigenvalues of the Hessian of empirical risk, and discuss its correspondence to weight decay.

Expand Down
4 changes: 2 additions & 2 deletions ...apters/15_regularization/15-10-geom-l1.md → ...apters/15_regularization/15-11-geom-l1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Chapter 15.10: Geometry of L1 Regularization"
weight: 15010
title: "Chapter 15.11: Geometry of L1 Regularization"
weight: 15011
---
In this section, we provide a geometric understanding of \\(L1\\) regularization and show that it encourages sparsity in the parameter vector.

Expand Down
4 changes: 2 additions & 2 deletions ...15_regularization/15-11-early-stopping.md → ...15_regularization/15-12-early-stopping.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Chapter 15.11: Early Stopping"
weight: 15011
title: "Chapter 15.12: Early Stopping"
weight: 15012
---
In this section, we introduce early stopping and show how it can act as a regularizer.

Expand Down
4 changes: 2 additions & 2 deletions ...ers/15_regularization/15-12-ridge-deep.md → ...ers/15_regularization/15-13-ridge-deep.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Chapter 15.12: Details on Ridge Regression: Deep Dive"
weight: 15012
title: "Chapter 15.13: Details on Ridge Regression: Deep Dive"
weight: 15013
---
In this section, we consider Ridge regression as row-augmentation and as minimizing risk under feature noise. We also discuss the bias-variance tradeoff.

Expand Down
4 changes: 2 additions & 2 deletions ...ers/15_regularization/15-13-lasso-deep.md → ...ers/15_regularization/15-14-lasso-deep.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Chapter 15.13: Soft-thresholding and L1 regularization: Deep Dive"
weight: 15013
title: "Chapter 15.14: Soft-thresholding and L1 regularization: Deep Dive"
weight: 15014
---
In this section, we prove the previously stated proposition regarding soft-thresholding and L1 regularization.

Expand Down