Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R notebooks #1253

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

R notebooks #1253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8cd28f2
finished evaluation notebooks
manuelhelmerichs Aug 29, 2024
af3e2d6
finished forests notebooks
manuelhelmerichs Aug 29, 2024
e55ae7e
finished nested resampling notebooks
manuelhelmerichs Aug 29, 2024
3f42d9d
finished classification notebooks
manuelhelmerichs Aug 29, 2024
63458c0
finished regression notebooks
manuelhelmerichs Aug 29, 2024
a81dd62
finished trees notebooks
manuelhelmerichs Aug 29, 2024
48edd5d
finished tuning notebooks
manuelhelmerichs Aug 29, 2024
79d2448
Kernel spec
manuelhelmerichs Aug 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
776 changes: 402 additions & 374 deletions exercises/evaluation/ex_eval_1_R.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

304 changes: 158 additions & 146 deletions exercises/evaluation/ex_eval_2_R.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,148 +1,160 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "53329df4",
"metadata": {},
"source": [
"***\n",
"**Introduction to Machine Learning** <br>\n",
"__[https://slds-lmu.github.io/i2ml/](https://slds-lmu.github.io/i2ml/)__\n",
"***"
]
"cells": [
{
"cell_type": "markdown",
"id": "53329df4",
"metadata": {},
"source": [
"***\n",
"**Introduction to Machine Learning** <br>\n",
"__[https://slds-lmu.github.io/i2ml/](https://slds-lmu.github.io/i2ml/)__\n",
"***"
]
},
{
"cell_type": "markdown",
"id": "56e690c1",
"metadata": {},
"source": [
"# Exercise sheet 6: Evaluation 2"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "94cf8812",
"metadata": {
"vscode": {
"languageId": "r"
}
},
"outputs": [],
"source": [
"#| label: import\n",
"# Consider the following libraries for this exercise sheet:\n",
"\n",
"library(mlbench)\n",
"library(mlr3)\n",
"library(mlr3learners)"
]
},
{
"cell_type": "markdown",
"id": "bb7dd93d",
"metadata": {},
"source": [
"## Exercise 2: Resampling strategies"
]
},
{
"cell_type": "markdown",
"id": "5ca487b7",
"metadata": {},
"source": [
"> a) Why would we apply resampling rather than a single holdout split?\n",
"\n",
"> **\\# Enter your answer here:**"
]
},
{
"cell_type": "markdown",
"id": "086f9397",
"metadata": {},
"source": [
"> b) Classify the `german_credit` data into solvent and insolvent debtors using logistic regression. Compute the\n",
"training error w.r.t. MCE."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "582a28c4",
"metadata": {
"vscode": {
"languageId": "r"
}
},
"outputs": [],
"source": [
"# Enter your code here:"
]
},
{
"cell_type": "markdown",
"id": "d7767419",
"metadata": {},
"source": [
"> c) In order to evaluate your learner, compare test MCE using\n",
">> (i) three times ten-fold cross validation (3x10-CV) <br>\n",
">> (ii) 10x3-CV <br>\n",
">> (iii) 3x10-CV with stratification for the feature `foreign_worker` to ensure equal representation in all folds <br>\n",
">> (iv) a single holdout split with $90\\%$ training data"
]
},
{
"cell_type": "markdown",
"id": "0a9c9f8e",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
" <b>Hint:</b> you will need <code>rsmp</code>, <code>resample</code> and <code>aggregate</code>. <br>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2091102d",
"metadata": {
"vscode": {
"languageId": "r"
}
},
"outputs": [],
"source": [
"# Enter your code here:"
]
},
{
"cell_type": "markdown",
"id": "9f12230c",
"metadata": {},
"source": [
"> d) Discuss and compare your findings from c) and compare them to the training error from b).\n",
"\n",
"> **\\# Enter your answer here:**"
]
},
{
"cell_type": "markdown",
"id": "7bf9bb5e",
"metadata": {},
"source": [
"> e) Would you consider LOO-CV to be a good alternative?\n",
"\n",
"> **\\# Enter your answer here:**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "R (I2ML)",
"language": "R",
"name": "R-i2ml"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "R",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
{
"cell_type": "markdown",
"id": "56e690c1",
"metadata": {},
"source": [
"# Exercise sheet 6: Evaluation 2"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "94cf8812",
"metadata": {},
"outputs": [],
"source": [
"#| label: import\n",
"# Consider the following libraries for this exercise sheet:\n",
"\n",
"library(mlbench)\n",
"library(mlr3)\n",
"library(mlr3learners)"
]
},
{
"cell_type": "markdown",
"id": "bb7dd93d",
"metadata": {},
"source": [
"## Exercise 2: Resampling strategies"
]
},
{
"cell_type": "markdown",
"id": "5ca487b7",
"metadata": {},
"source": [
"> a) Why would we apply resampling rather than a single holdout split?\n",
"\n",
"> **\\# Enter your answer here:**"
]
},
{
"cell_type": "markdown",
"id": "086f9397",
"metadata": {},
"source": [
"> b) Classify the `german_credit` data into solvent and insolvent debtors using logistic regression. Compute the\n",
"training error w.r.t. MCE."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "582a28c4",
"metadata": {},
"outputs": [],
"source": [
"# Enter your code here:"
]
},
{
"cell_type": "markdown",
"id": "d7767419",
"metadata": {},
"source": [
"> c) In order to evaluate your learner, compare test MCE using\n",
">> (i) three times ten-fold cross validation (3x10-CV) <br>\n",
">> (ii) 10x3-CV <br>\n",
">> (iii) 3x10-CV with stratification for the feature `foreign_worker` to ensure equal representation in all folds <br>\n",
">> (iv) a single holdout split with $90\\%$ training data"
]
},
{
"cell_type": "markdown",
"id": "0a9c9f8e",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
" <b>Hint:</b> you will need <code>rsmp</code>, <code>resample</code> and <code>aggregate</code>. <br>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2091102d",
"metadata": {},
"outputs": [],
"source": [
"# Enter your code here:"
]
},
{
"cell_type": "markdown",
"id": "9f12230c",
"metadata": {},
"source": [
"> d) Discuss and compare your findings from c) and compare them to the training error from b).\n",
"\n",
"> **\\# Enter your answer here:**"
]
},
{
"cell_type": "markdown",
"id": "7bf9bb5e",
"metadata": {},
"source": [
"> e) Would you consider LOO-CV to be a good alternative?\n",
"\n",
"> **\\# Enter your answer here:**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "python-i2ml",
"language": "python",
"name": "python-i2ml"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
"nbformat": 4,
"nbformat_minor": 5
}
Loading