update ic_classification_2

exercises/supervised-classification-tex/ex_rnw/ex_model-assumptions.Rnw

@@ -1,4 +1,16 @@
-Chapters 3.5 and 3.6 of the lecture introduce linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and Naive Bayes (NB), all of which constitute \textbf{generative approaches} to classification. With everything you have learned about QDA, LDA, and NB, how can model assumptions be reframed to fit all three approaches in the Venn diagram shown below?
+In Chapters 3.5 and 3.6 of the lecture, we discussed three key generative classification methods: Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), and Naive Bayes (NB). The Venn diagram below represents these methods as anonymized regions labeled A, B, and C, illustrating the relationships between these methods under a specific assumption.
+
+Your task:
+
+\begin{enumerate}
+\item Summarize the underlying assumptions of LDA, QDA, and NB.
+\item Assign each method (LDA, QDA, and NB) to the appropriate region (A, B, or C) in the Venn diagram based on these assumptions.\\
+\textbf{Hint:} Firstly, think about how B and C could be seen as special case of A, respectively. Then, consider the shared assumptions that might overlap between B and C to specify their intersection.
+\item As mentioned, the Venn diagram is true under a "specific assumption". What exactly is this assumption?
+\end{enumerate}
+
+%Your task is to analyze the underlying assumptions of these classification models and correctly place them within the diagram.
+
 \smallbreak
 <<echo=FALSE, message=FALSE, fig.align="left", fig.height = 1.7, fig.width = 2.6>>=
 library(ggplot2)
@@ -7,9 +19,7 @@ data_circles <- data.frame(x0 = c(0, -0.25, 0.32), y0 = c(0, 0.3, 0.2), r = c(1,
 
 data_labels <-  data.frame(x0 = c(-0.5, -0.55, 0.55), y0 = c(-0.65, 0.1, 0), name = c("A", "B", "C"))
 ggplot() +
-  geom_circle(data = data_circles, aes(x0 = x0, y0 = y0, r = r), size=0.8) +
+  geom_circle(data = data_circles, aes(x0 = x0, y0 = y0, r = r), linewidth = 0.8) +
   geom_text(data = data_labels, aes(x = x0, y = y0, label = name), size = 5) +
   theme_void()
-@
-\smallbreak
-Hint: Firstly, think about how B and C could be seen as special case of A, respectively. Then, think about how you could specify the intersection of B and C.
+@

exercises/supervised-classification-tex/ex_rnw/sol_model-assumptions.Rnw

@@ -1,13 +1,46 @@
 
-Note: This is only true assuming gaussian distributions. Otherwise, the ellipse of NB ends outside of that of QDA. But do not tell students this beforehand, they shall figure out by themselves that the question is kind of wrong -- on purpose.
-
-A is QDA, B and C are either LDA or Naive Bayes.
+\begin{enumerate}
+\item \textbf{Assumptions of Each Method:}
+    \begin{itemize}
+        \item \textbf{Quadratic Discriminant Analysis (QDA):}
+        \begin{itemize}
+            \item Assumes that each class has its own class-specific mean vector \( \boldsymbol{\mu}_k \) and covariance matrix \( \Sigma_k \).
+            \item Does not assume feature independence.
+            \item Decision boundaries are quadratic as covariance matrices can differ between classes.
+        \end{itemize}
+
+        \item \textbf{Linear Discriminant Analysis (LDA):}
+        \begin{itemize}
+            \item A special case of QDA where the covariance matrices are the same for all classes:
+            \[
+            \Sigma_k = \Sigma \quad \forall k
+            \]
+            \item Assumes shared covariance structure but different mean vectors for each class.
+            \item Results in linear decision boundaries due to the shared covariance matrix.
+        \end{itemize}
+
+        \item \textbf{Naive Bayes (NB):}
+        \begin{itemize}
+            \item Another special case of QDA, assuming conditional independence of features given the class label:
+            \[
+            p(\mathbf{x} | y = k) = \prod_{j=1}^p p(x_j | y = k)
+            \]
+            \item This results in diagonal covariance matrices where only the variances of the features are considered, and all covariances are zero.
+            \item Simplifies computation and parameter estimation.
+        \end{itemize}
+    \end{itemize}
+  \item \textbf{Assignments to Regions:} A is QDA (as it makes the most general assumptions with class-specific covariance matrices), B and C are either LDA or NB.
 \begin{enumerate}
 \item[1)] LDA can be seen as a special case of QDA if the covariance matrix is equal for all classes: $\Sigma_k = \Sigma \; \; \forall k$
-\item[2)] Naive Bayes can be seen as a special case of QDA if the features are conditionally independent given class $k$:
+\item[2)] Gaussian NB can be seen as a special case of QDA if the features are conditionally independent given class $k$:
 \begin{align}
 \pdfxyk = p((x_1, x_2, ..., x_p)|y = k)=\prodjp p(x_j|y = k),
 \end{align}
 which results in diagonal covariance matrices.
-\item[3)] Naive Bayes and LDA have an intersection if the covariance matrix is equal for all classes: $\Sigma_k = \Sigma \; \; \forall k$ \textbf{and} features are conditionally independent given class $k$, leaving each class with the same diagonal covariance matrix $\Sigma$.
+\item[3)] Gaussian NB and LDA have an intersection if the covariance matrix is equal for all classes: $\Sigma_k = \Sigma \; \; \forall k$ \textbf{and} features are conditionally independent given class $k$, leaving each class with the same diagonal covariance matrix $\Sigma$.
+\end{enumerate}
+
+\item \textbf{Specific Assumption:}
+The Venn diagram is valid under the assumption that the class-conditional distributions are Gaussian. This allows LDA, QDA, and NB to be represented as overlapping regions. In cases without Gaussian assumptions, the relationships may change, and NB may exceed the region of QDA (ellipse of NB ends outside of that of QDA to account for different distributions).
+
 \end{enumerate}