-
Notifications
You must be signed in to change notification settings - Fork 4
/
recap7.html
181 lines (114 loc) · 13.2 KB
/
recap7.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
---
layout: page
title: Lecture 7 Recap - Model Selection (Bayesian)
mathjax: true
weight: 0
---
<section class="main-container text">
<div class="main">
<h4>Date: February 16, 2021 (<a href="https://docs.google.com/forms/d/e/1FAIpQLSfOFrH5hFSp5mnV53qS4lTWFzm5pDAY_BLhMlIs-pl-GHAw-Q/viewform" target="_blank">Concept Check</a>, <a href="https://docs.google.com/forms/d/e/1FAIpQLSfOFrH5hFSp5mnV53qS4lTWFzm5pDAY_BLhMlIs-pl-GHAw-Q/viewanalytics" target="_blank">Class Responses</a>, <a href="{{ site.baseurl }}/ccsolutions" target="_blank">Solutions</a>)</h4>
<h4>Relevant Textbook Sections: 2.8, 2.9</h4>
<br>
<h4><a href="https://harvard.zoom.us/rec/play/3Rs5-CUfscUw4Spx0SouTEb2IVjbql0NdGyc_uelsI1ooPuGvtuPQEryOS0EdjGUvs5JzgJ6bRgbCwAB.0jRR41-cBQcNivT7">Lecture Video</a></h4>
<h4><a href="files/lecture7_ipad.pdf" target="_blank">iPad Notes</a></h4>
<h3>Lecture 7 Summary</h3>
<ul>
<li><a href="#recap7_1">Diversion: More Real World Context on Model Selection</a></li>
<li><a href="#recap7_2">Intro to Bayesian Model Selection</a></li>
<li><a href="#recap7_3">Setting Up a Beta-Bernoulli Model Example</a></li>
<li><a href="#recap7_4">Different Choices for Predictors</a></li>
<li><a href="#recap7_5">Model Selection With the Marginal Likelihood</a></li>
</ul>
<h2 id="recap7_1">Diversion: More Real World Context on Model Selection</h2>
We start the lecture with a few more real world examples to discuss. If inaccurate assumptions are made, models can be wrong even when all of the math is correct. For instance, we might assume that data features are independent of each other when the real-world features are actually correlated. <br><br>
Another example is a medical model assuming that antidepressants and patient features produce a recovery outcome, and that a correct drug could be selected for each patient to help them recover. Machine learning was used to predict antidepressant recommendations, but the model ended up overfitting. In the data, some patients’ illnesses were just less severe. These patients were more likely to get better no matter what medication they were prescribed. The model would predict medications and these patients would probably get well, but not because it was the best drug — this was just an easy patient to treat.
<h2 id="recap7_2">Intro to Bayesian Model Selection</h2>
When we talked about classification, we had discriminative and generative models. Each one makes sense in different scenarios.
<ul>
<li>Discriminative: Use $x$ to predict $y$ by maximizing $p(y | x)$. As an example, we might use data about a customer’s features ($x$) to predict what they will buy ($y$).</li>
<li>Generative: Consider how the data is being produced by modeling $p(x,y)$. As an example, we might have data about patient symptoms ($y$) and want to know what disease produced them ($x$).</li>
</ul>
Earlier in <a href="recap3.html">Lecture 3</a>, we discussed a generative setting where data was produced via the following story:
<ol>
<li>We have an input $x$</li>
<li>We have a noise sample $\epsilon \sim \mathcal{N}(0,\sigma^2)$</li>
<li>The data $y$ is produced as $y = w^Tx + \epsilon$</li>
</ol>
In the Lecture 3 story, $w$ was fixed. Today, we will add a step 0 where we consider $w$ to be a random variable as well. (The $w$’s are the different models we are considering. For example, they might represent all possible decision boundary lines.) Through the Bayesian view, we can look at three useful concepts: the posterior distribution, the posterior predictive, and the marginal likelihood.<br><br>
<h3>Posterior Distribution</h3>
After obtaining some data $X, y$, we can compute the posterior distribution $p(w | X,y)$. This tells us, “Given all the data you’ve seen in $X$ and $y$, these are the possibilities (and probabilities) for what $w$ will look like.” <br><br>
How can we find this distribution? First observe that in the data generation story we’re currently considering, we only need to know $w$, $x’$, and $\epsilon$ to get $y’$. So, we can make simplifications such as $p(y’ | w, x’, X, y) = p(y’ | w, x’)$, because the value of $y’$ does not depend on previously observed data. Also, seeing more $x$’s alone does not tell us more about what $w$ is, but seeing more labeled pairs of $x$’s and $y$’s together does tell us more about $w$’s distribution. Thus, $p(w|X) = p(w)$, but $p(w|X, y) \neq p(w)$. <br><br>
Using this along with the fact that $p(y | X)$ does not depend on $w$, we can apply Bayes’ Theorem and then do some simplifying:
\begin{align*}
p(w | X,y) &= \frac{p(y | X, w)p(w | X)}{p(y | X)}
\\ &= \frac{p(y | X, w)p(w)}{p(y | X)}
\\ &\propto p(y | X,w)p(w)
\end{align*}
Depending on the setting and prior, this could be an easy calculation (if there's conjugacy) or a hard calculation (if there's not).<br><br>
<u>Note:</u> Conjugacy is when the prior and posterior are from the same family of distributions, like if they're both Gaussian or both Beta. It can provide nice mathematical properties and cleaner forms that make calculations much easier.
<h3>Posterior Predictive</h3>
To make predictions, we use the posterior predictive $p(y’ | x’, X, y)$. For a new piece of test data $x’$, the posterior predictive $p(y’ | x’, X, y)$ tells us how likely the label for $x’$ is to be $y’$ given the data $X, y$ that we have already observed. To find the posterior predictive, we would integrate over the possible $w$’s in our posterior distribution for $w$:
$$p(y’ | x’, X, y) = \int_w p(y’ | w, x’)p(w | X, y)dw$$
<u>Note:</u> In the Bayesian framework, we avoid working directly with $w$. It’s a random variable we don’t know the value of, so instead of working with $w$, we integrate over the whole distribution for $w$ to consider the likelihoods of all the different possible values.
<h3>Marginal Likelihood</h3>
Finally, we can compute the marginal likelihood, the probability of the training set
$$p(x,y) = \int_w p(y | X,w)p(w)dw$$
The posterior predictive is what we use for inference (making predictions), but maximizing the marginal likelihood is what helps us select a model.
<h2 id="recap7_3">Setting Up a Beta-Bernoulli Model Example</h2>
<h3>Defining the Bernoulli Part</h3>
Let's consider a coin that comes up "1" with probability $\theta$. Let $x$ be the result of the toss. The probability of a flip landing a certain way comes from the Bernoulli PMF:
$$p(x | \theta) = \theta^x (1 - \theta)^{1 - x}$$
If we want to write out the probability of a dataset of multiple coin flips, then we have:
$$p(x_1 ... x_n | \theta) = \prod_n (\theta)^{x_n}(1-\theta)^{1 - x_n}$$
$$p(x_1 ... x_n | \theta) = \theta^{n_1}(1-\theta)^{n_0}$$
Where in the above, $n_1$ is the number of heads, and $n_0$ is the number of tails.<br><br>
<h3>Defining the Beta Part</h3>
Today we'll be considering a Bayesian perspective where $p(\theta)$ ~ $Beta(\alpha, \beta)$. <br><br>
Now in our data generation story, Step 0 is drawing $\theta$ from a Beta distribution. Step 1 is drawing $N$ coin flips $x_i$ from $Bern(\theta)$ after $\theta$ is determined. <br><br>
The Beta distribution is parameterized by $\alpha$ and $\beta$, which describe our prior beliefs about what the probability distribution for $w$ should look like. We can think of $\alpha$ and $\beta$ as values for how many heads and tails we've seen so far — if we set $\alpha > \beta$, it would be like saying we had seen more heads than tails so far, and our prior distribution would be higher on the right. When $\alpha = \beta = 1$, the distribution is uniform, which best represents a case where we are clueless about what $\theta$ might be. If we set $\alpha = \beta > 1$, it reflects a belief that we've seen an equal number of heads and tails, and that $\theta$ is likely to be around $0.5$.
<h3>Calculating the Posterior</h3>
Starting from the proportionality we found earlier and plugging in the Beta PDF, we have:
\begin{align*}
p(\theta | X) &\propto p(X|\theta)p(\theta)
\\ &= \theta^{n_1}(1-\theta)^{n_0}z_{\alpha, \beta}\theta^{\alpha - 1}(1-\theta)^{\beta - 1}
\\ &= z_{\alpha,\beta} \theta^{n_1 + \alpha - 1}(1 - \theta)^{n_0 + \beta - 1}
\end{align*}
(The $z$ is just a normalizing constant that will integrate to 1. We don't need to worry about it right now.)<br><br>
We just rewrote our posterior into the form of another Beta distribution, specifically $Beta(n_1 + \alpha, n_0 +\beta)$. Recall that our prior was $Beta(\alpha, \beta)$. Since they come from the same distribution family, we have conjugacy.
<h2 id="recap7_4">Different Choices for Predictors</h2>
Let's say we're interested in inferring what $\theta$ is. <br><br>
If we take the maximum likelihood frequentist approach, we would ignore the prior and get $\theta_{MLE} = argmax_{\theta}$ $\theta^{n_1}(1-\theta)^{n_0}$, which leads us to an intuitive answer, the fraction of flips that came up heads: $$\theta_{MLE} = \frac{n_1}{n_1 + n_0}$$
The Bayesian approach gives us additional methods for inferring $\theta$.
<h3>Finding the Maximum a Posteriori (MAP) </h3>
Imagine that we used $\alpha = \beta = 2$ for our prior and that we've done two coin flips, both of which were heads. Then we'd have $n_1 = 2, n_2 = 0$. Using these values of $\alpha, \beta$ for our prior is like believing that we've already "seen" 2 heads and 2 tails that don't count as part of our data, so we are beginning with a belief that the coin is likely to be equal.<br><br>
After seeing two heads, our posterior distribution would be shifted to be higher on the right side, towards 1. The point at the peak of the posterior distribution is the Maximum a posteriori predictor (MAP). The MAP predictor can be solved for as
$$\theta_{MAP} = \frac{n_1 + \alpha - 1}{n_0 + n_1 + \alpha + \beta - 2} = \frac34.$$
The form is the same as $\theta_{MLE}$'s form was, except the MAP includes the $\alpha$ and $\beta$ of the prior as well. The intuition here is that as the number of trials gets large, the data will overwhelm whatever prior you have. <br><br>
Using a frequentist MLE approach where we only consider the data we've seen, we would have gotten $\theta_{MLE} = 1$ as our estimator because we have seen only heads. $\theta_{MAP}$ incorporates our prior beliefs about how $\theta$ is distributed in addition to the data we've observed. <br><br>
<h3>Using the Posterior Predictive</h3>
The most Bayesian person would actually integrate over the posterior distribution instead of choosing a value for $\theta$ — after all, $\theta$ is supposed to be a random variable we don't know the value of. Let's consider the probability that our next coin flip will be heads. Conveniently, this is equal to the expected value of $\theta$. Even more conveniently, since we know $\theta$ follows a Beta distribution, we can just look up what the expected value of a Beta random variable will be.
\begin{align*}
p(x = 1|X) &= \int p(x=1|\theta) p(\theta | x_1 ... x_n )d\theta
\\ &= \mathbb{E}_{p(\theta | X)}(\theta)
\\ &= \frac{\alpha + n_1}{\alpha + \beta + n_1 + n_0} = \frac46 = \frac23
\end{align*}
<h3>Quick Summary</h3>
We've now seen three ways to infer $\theta$:
<ol>
<li>MLE: Let's predict using just the data we have seen.</li>
<li>MAP: Let's compute the posterior distribution and select the value for $\theta$ with highest probability.</li>
<li>Posterior Predictive: If we're feeling super Bayesian, we can avoid picking a single $\theta$ value. Instead, we'll integrate over the whole distribution to get an expected $\theta$. </li>
</ol>
<h2 id="recap7_5">Model Selection With the Marginal Likelihood</h2>
What if we didn't know whether to use $\alpha_1, \beta_1$ or $\alpha_2, \beta_2$ as our prior? We can make the decision by comparing $p(X| \alpha_1 \beta_1)$ vs. $p(X| \alpha_2, \beta_2)$ to see which option makes the data we observed more likely.
\begin{align*}
p(X) &= \int_{\theta}p(X | \theta)p(\theta)d\theta
\\ &= \int_{\theta} \theta^{n_1}(1-\theta)^{n_0}z_{\alpha, \beta}\theta^{\alpha - 1}(1-\theta)^{\beta - 1}d\theta
\\ &= z_{\alpha, \beta}\int_{\theta} \theta^{n_1 + \alpha - 1}(1 - \theta)^{n_0 + \beta - 1}\frac{z_{n_1 + \alpha, n_0 + \beta}}{z_{n_1 + \alpha, n_0 + \beta}}d\theta
\\ &= \frac{z_{\alpha, \beta}}{z_{n_1 + \alpha, n_0 + \beta}} \int_{\theta} \theta^{n_1 + \alpha - 1}(1 - \theta)^{n_0 + \beta - 1}z_{n_1 + \alpha, n_0 + \beta}d\theta
\\ &= \frac{z_{\alpha, \beta}}{z_{n_1 + \alpha, n_0 + \beta}} \cdot 1
\\ &= \frac{z_{\alpha, \beta}}{z_{n_1 + \alpha, n_0 + \beta}}
\end{align*}
The integrand at the end is equal to $1$ because we manipulated it into the form of a $Beta(n_1 + \alpha, n_0 + \beta)$ distribution, which will integrate to $1$.
</div>
</section>