assignment2-chi.ipynb

{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "The patient IDs were removed from this version of the data, leaving 384 input features which were put in each of the ```“X_...”``` arrays. The corresponding CT scan slice location has been put in the ```“y_...”``` arrays. We shifted and scaled the ```“y_...”``` location values for the version of the data that you are using. The shift and scaling was chosen to make the training locations have zero mean and unit variance. The first 73 patients were put in the ```_train``` arrays, the next 12 in the ```_val``` arrays, and the final 12 in the ```_test``` arrays. Please use this training, validation, test split as given. **Do not shuffle the data further in this assignment.**",
   "id": "a27d8a4b7d41091a"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## Task 1: Get Started",
   "id": "36c4b6181f83ec3e"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:18.144395Z",
     "start_time": "2024-11-15T10:53:17.519320Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from support_code import *\n",
    "import numpy as np\n",
    "data = np.load('ct_data.npz')\n",
    "X_train = data['X_train']; X_val = data['X_val']; X_test = data['X_test']\n",
    "y_train = data['y_train']; y_val = data['y_val']; y_test = data['y_test']"
   ],
   "id": "5f3f2e01481b7657",
   "outputs": [],
   "execution_count": 1
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "Verify that (up to numerical rounding errors) the mean of the training positions in ```y_train``` is zero. The mean of the 5,785 positions in the ```y_val``` array is not zero. Report its mean with a “standard error”, temporarily assuming that each entry is independent. For comparison, also report the mean with a standard error of the first 5,785 entries in the ```y_train```. Explain how your results demonstrate that these standard error bars do not reliably indicate what the average of locations in future CT slice data will be. Why are standard error bars misleading here?",
   "id": "4a987b20fed8bec"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:18.160888Z",
     "start_time": "2024-11-15T10:53:18.156251Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Calculate mean and standard error for y_train\n",
    "y_train_mean = np.mean(y_train)\n",
    "y_train_std_error = np.std(y_train, ddof=1) / np.sqrt(len(y_train))\n",
    "\n",
    "# Calculate mean and standard error for the first 5,785 entries in y_train\n",
    "y_train_sample_mean = np.mean(y_train[:5785])\n",
    "y_train_sample_std_error = np.std(y_train[:5785], ddof=1) / np.sqrt(5785)\n",
    "\n",
    "# Calculate mean and standard error for y_val\n",
    "y_val_mean = np.mean(y_val)\n",
    "y_val_std_error = np.std(y_val, ddof=1) / np.sqrt(len(y_val))\n",
    "\n",
    "y_train_mean, y_train_std_error, y_val_mean, y_val_std_error, y_train_sample_mean, y_train_sample_std_error"
   ],
   "id": "1f38220e739fe0ed",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-9.13868774539957e-15,\n",
       " 0.0049535309340638205,\n",
       " -0.2160085093241599,\n",
       " 0.01290449880016868,\n",
       " -0.44247687859693674,\n",
       " 0.011927303389170828)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 2
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "Some of the input features are constants: they take on the same value for every training example. Identify these features, and remove them from the input matrices in the training, validation, and testing sets.\n",
    "\n",
    "Moreover, some of the input features are duplicates: some of the columns in the training set are identical. For each training set column, discard any later columns that are identical. Discard the same columns from the validation and testing sets.\n",
    "\n",
    "**Use these modified input arrays for the rest of the assignment.** Keep the names of the arrays the same (X_train, etc.), so we know what they’re called. You should not duplicate the code from this part in future questions. We will assume it has been run, and that the modified data are available.\n",
    "\n",
    "**Warning: As in the real world, mistakes at this stage would invalidate all of your results. We strongly recommend checking your code, for example on small test examples where you can see what it’s doing.**\n",
    "\n",
    "Report which columns of the X_... arrays you remove at each of the two stages. Report these as 0-based indexes. (For the second stage, you might report indexes in the original array, or after you did the first stage. It doesn’t matter, as long as your code is clear and correct.)"
   ],
   "id": "10d6003dd8d0df97"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:18.542856Z",
     "start_time": "2024-11-15T10:53:18.161603Z"
    }
   },
   "cell_type": "code",
   "source": [
    "constant_columns = [i for i in range(X_train.shape[1]) if np.all(X_train[:, i] == X_train[0, i])]\n",
    "X_train = np.delete(X_train, constant_columns, axis=1)\n",
    "X_val = np.delete(X_val, constant_columns, axis=1)\n",
    "X_test = np.delete(X_test, constant_columns, axis=1)\n",
    "\n",
    "_, unique_indices = np.unique(X_train, axis=1, return_index=True)\n",
    "duplicate_columns = [i for i in range(X_train.shape[1]) if i not in unique_indices]\n",
    "X_train = np.delete(X_train, duplicate_columns, axis=1)\n",
    "X_val = np.delete(X_val, duplicate_columns, axis=1)\n",
    "X_test = np.delete(X_test, duplicate_columns, axis=1)\n",
    "\n",
    "print(\"Constant columns removed:\", constant_columns)\n",
    "print(\"Duplicate columns removed:\", duplicate_columns)"
   ],
   "id": "fbd282beb87d1f52",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Constant columns removed: [59, 69, 179, 189, 351]\n",
      "Duplicate columns removed: [76, 77, 185, 195, 283, 354]\n"
     ]
    }
   ],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# Task 2: Linear Regression Baseline\n",
    "Using ```numpy.linalg.lstsq```, write a short function “fit_linreg(X, yy, alpha)” that fits the linear regression model\n",
    "$$f(\\b x;\\b w,b) = \\b w^\\top\\b x + b,$$\n",
    "by minimizing the cost function:\n",
    "$$E(\\b w, b) = \\alpha\\b w^\\top\\b w + \\sum_n (f(\\b x^{(n)};\\b w,b) - y^{(n)})^2,$$\n",
    "with regularization constant $\\alpha$. As discussed in the lecture materials, fitting a bias parameter $b$ and incorporating the regularization constant can both be achieved by augmenting the original data arrays. Use a data augmentation approach that maintains the numerical stability of the underlying ```lstsq``` solver, rather than a ‘normal equations’ approach. You should only regularize the weights $\\textbf{w}$ and not the bias $b$.\n",
    "\n",
    "(In the lecture materials we used $\\lambda$ for the regularization constant, matching Murphy and others. However, lambda is a reserved word in Python, so we swapped to ```alpha``` for our code.)\n",
    "\n",
    "Use your function to fit weights and a bias to ```X_train``` and ```y_train```. Use $\\alpha = 30$.\n",
    "\n",
    "We can fit the same model with a gradient-based optimizer. The support code has a function ```fit_linreg_gradopt```, which you should look at and try.\n",
    "\n",
    "Report the root-mean-square errors (RMSE) on the training and validation sets for the parameters fitted using both your ```fit_linreg``` and the provided ```fit_linreg_gradopt```. Do you get exactly the same results? Why or why not?"
   ],
   "id": "8fa161367e7a3106"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:18.546989Z",
     "start_time": "2024-11-15T10:53:18.544435Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_linreg(X, yy, alpha):\n",
    "    # Augment X with a column of ones for the bias term\n",
    "    X_aug = np.hstack((X, np.ones((X.shape[0], 1))))\n",
    "    \n",
    "    # Create an augmented matrix for regularization, leaving bias unregularized\n",
    "    reg_matrix = np.sqrt(alpha) * np.eye(X_aug.shape[1])\n",
    "    reg_matrix[-1, -1] = 0  # No regularization on the bias term\n",
    "    \n",
    "    # Augment yy to account for the regularization terms\n",
    "    X_aug_reg = np.vstack((X_aug, reg_matrix))\n",
    "    yy_reg = np.hstack((yy, np.zeros(X_aug.shape[1])))\n",
    "    \n",
    "    # Use lstsq to solve for weights and bias\n",
    "    w_aug, _, _, _ = np.linalg.lstsq(X_aug_reg, yy_reg, rcond=None)\n",
    "    return w_aug[:-1], w_aug[-1]  # Split weights and bias\n",
    "\n",
    "def rmse(y_true, y_pred):\n",
    "    return np.sqrt(np.mean((y_true - y_pred) ** 2))"
   ],
   "id": "df31349c9ea06e0a",
   "outputs": [],
   "execution_count": 4
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:21.388816Z",
     "start_time": "2024-11-15T10:53:18.547959Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# report fit_linreg\n",
    "alpha = 30\n",
    "w, b = fit_linreg(X_train, y_train, alpha)\n",
    "\n",
    "# report rmse\n",
    "y_train_pred = X_train @ w + b\n",
    "y_val_pred = X_val @ w + b\n",
    "\n",
    "# Calculate RMSE\n",
    "train_rmse_lstsq = rmse(y_train, y_train_pred)\n",
    "val_rmse_lstsq = rmse(y_val, y_val_pred)\n",
    "\n",
    "print(\"Training RMSE (lstsq):\", train_rmse_lstsq)\n",
    "print(\"Validation RMSE (lstsq):\", val_rmse_lstsq)\n",
    "\n",
    "w_gradopt, b_gradopt = fit_linreg_gradopt(X_train, y_train, alpha)\n",
    "\n",
    "# report rmse\n",
    "y_train_pred = X_train @ w_gradopt + b_gradopt\n",
    "y_val_pred = X_val @ w_gradopt + b_gradopt\n",
    "\n",
    "# Calculate RMSE\n",
    "train_rmse_lstsq = rmse(y_train, y_train_pred)\n",
    "val_rmse_lstsq = rmse(y_val, y_val_pred)\n",
    "\n",
    "print(\"Training RMSE (lstsq) in fit_linreg_gradopt:\", train_rmse_lstsq)\n",
    "print(\"Validation RMSE (lstsq) in fit_linreg_gradopt:\", val_rmse_lstsq)"
   ],
   "id": "2331cd1af1f9104c",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE (lstsq): 0.3567565397204054\n",
      "Validation RMSE (lstsq): 0.4230521968394695\n",
      "Training RMSE (lstsq) in fit_linreg_gradopt: 0.3567556103401202\n",
      "Validation RMSE (lstsq) in fit_linreg_gradopt: 0.42305510586203865\n"
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# Task 3: Invented classification tasks",
   "id": "e1a105ec82cb07b4"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "It is often easier to work with binary data than real-valued data: we don’t have to think so hard about how the values might be distributed, and how we might process them. We will invent some binary classification tasks, and fit these.\n",
    "\n",
    "We will pick 20 positions within the range of training positions, and use each of these to define a classification task:"
   ],
   "id": "f8232af76a321aa9"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "The logistic regression cost function and gradients are provided with the assignment in the function ```logreg_cost```. It is analogous to the ```linreg_cost``` function for least-squares regression, which is used by the ```fit_linreg_gradopt``` function that you used earlier.\n",
    "\n",
    "Fit logistic regression to each of the 20 classification tasks above with $\\alpha=30$\n",
    ".\n",
    "\n",
    "Given a feature vector, we can now obtain 20 different probabilities, the predictions of the 20 logistic regression models. Transform both the training and validation input matrices into new matrices with 20 columns, containing the probabilities from the 20 logistic regression models. You don’t need to loop over the rows of ```X_train``` or ```X_val```, you can use array-based operations to make the logistic regression predictions for every datapoint at once.\n",
    "\n",
    "Fit a regularized linear regression model ($\\alpha=30$) to your transformed 20-dimensional training set. Report the training and validation root mean square errors (RMSE) of this model."
   ],
   "id": "1eebd9bcb9b6937e"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:21.402867Z",
     "start_time": "2024-11-15T10:53:21.396445Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_logreg_gradopt(X, yy, alpha):\n",
    "    \"\"\"\n",
    "    Fit a regularized logistic regression model using gradient optimization.\n",
    "    \"\"\"\n",
    "    D = X.shape[1]\n",
    "    args = (X, yy, alpha)\n",
    "    init = (np.zeros(D), np.array(0))\n",
    "    ww, bb = minimize_list(logreg_cost, init, args)\n",
    "    return ww, bb"
   ],
   "id": "277c5d696b2893c6",
   "outputs": [],
   "execution_count": 6
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:21.411660Z",
     "start_time": "2024-11-15T10:53:21.404873Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def logreg_k(X, yy, K, alpha=30):\n",
    "    \"\"\"\n",
    "    Trains K binary logistic regression models on data using different threshold-based label splits.\n",
    "    \"\"\"\n",
    "    mx = np.max(yy); mn = np.min(yy); hh = (mx-mn)/(K+1)\n",
    "    thresholds = np.linspace(mn+hh, mx-hh, num=K, endpoint=True)\n",
    "    \n",
    "    # weights and biases for the K logistic regression models\n",
    "    w_logreg_k = np.zeros((K, X_train.shape[1]))\n",
    "    b_logreg_k = np.zeros((K))\n",
    "    \n",
    "    for kk in range(K):\n",
    "        # get binary training labels based on thresholds[kk]\n",
    "        labels = yy > thresholds[kk]\n",
    "        \n",
    "        w_logreg, b_logreg = fit_logreg_gradopt(X, labels, alpha)\n",
    "        \n",
    "        w_logreg_k[kk, :] = w_logreg\n",
    "        b_logreg_k[kk] = b_logreg\n",
    "    \n",
    "    return w_logreg_k, b_logreg_k"
   ],
   "id": "8be85869c2ec8847",
   "outputs": [],
   "execution_count": 7
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:33.482880Z",
     "start_time": "2024-11-15T10:53:21.413711Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def sigma(a):\n",
    "    return 1 / (1 + np.exp(-a))\n",
    "\n",
    "K = 20 # number of thresholded classification problems to fit\n",
    "\n",
    "# Transform both the training and validation input matrices into new matrices with K columns\n",
    "w_logreg_k, b_logreg_k = logreg_k(X_train, y_train, K)\n",
    "X_train_new = sigma(X_train @ w_logreg_k.T + b_logreg_k)\n",
    "X_val_new = sigma(X_val @ w_logreg_k.T + b_logreg_k)"
   ],
   "id": "f6b08c9de1985e1b",
   "outputs": [],
   "execution_count": 8
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:34.022476Z",
     "start_time": "2024-11-15T10:53:33.486135Z"
    }
   },
   "cell_type": "code",
   "source": [
    "w_linreg, b_linreg = fit_linreg_gradopt(X_train_new, y_train, alpha=30)\n",
    "\n",
    "pred_train = X_train_new @ w_linreg + b_linreg\n",
    "pred_val = X_val_new @ w_linreg + b_linreg\n",
    "\n",
    "rmse_train_gd = rmse(pred_train, y_train)\n",
    "rmse_val_gd = rmse(pred_val, y_val)\n",
    "rmse_train_gd, rmse_val_gd"
   ],
   "id": "d3d7b9762869655d",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.15441197301431134, 0.2542485967584797)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 9
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# Task 4: Small neural network",
   "id": "f56b3061c35835c"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "In Question 3 you fitted a small neural network. The logistic regression classifiers are sigmoidal hidden units, and a linear output unit predicts the outputs. However, you didn’t fit the parameters jointly to the obvious least squares cost function. A least squares cost function and gradients for this neural network are implemented in the ``nn_cost`` function provided.\n",
    "\n",
    "Try fitting the neural network model to the training set, with a) a sensible random initialization of the parameters; b) the parameters initialized using the fits made in Q3.\n",
    "\n",
    "Does one initialization strategy work better than the other? Does fitting the neural network jointly work better than the procedure in Q3? Your explanation should include any numbers that your answer is based on.\n"
   ],
   "id": "b5bf255c9a03e130"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "对比两种参数初始化策略对一个小型神经网络模型拟合效果的影响，并探讨联合拟合参数是否优于在第3题中使用的单独拟合方法。",
   "id": "51d8f83567a4a89c"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:53:34.039767Z",
     "start_time": "2024-11-15T10:53:34.035821Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_nn_gradopt(X, yy, alpha, init):\n",
    "    \"\"\"\n",
    "    Fit a regularized logistic regression model using gradient optimization.\n",
    "    \"\"\"\n",
    "    args = (X, yy, alpha)\n",
    "    # Initialization\n",
    "    ww, bb, V, bk = minimize_list(nn_cost, init, args)\n",
    "    return ww, bb, V, bk\n",
    "\n",
    "# Glorot Initialization\n",
    "def glorot_init(n_in, n_out):\n",
    "    limit = np.sqrt(6 / (n_in + n_out))\n",
    "    return np.random.uniform(-limit, limit, size=(n_out, n_in))\n",
    "\n",
    "def init_nn_parameter_glorot(D, K):\n",
    "    ww_init = glorot_init(K, 1).flatten()\n",
    "    bb_init = np.array(0)\n",
    "    V_init = glorot_init(D, K)\n",
    "    bk_init = np.zeros(K)\n",
    "    \n",
    "    init = (ww_init, bb_init, V_init, bk_init)\n",
    "    return init"
   ],
   "id": "654eeb97f728ad49",
   "outputs": [],
   "execution_count": 10
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:54:44.864059Z",
     "start_time": "2024-11-15T10:53:34.042148Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# params that fit in the Q3\n",
    "init = (w_linreg, b_linreg, w_logreg_k, b_logreg_k)\n",
    "params = fit_nn_gradopt(X_train, y_train, alpha, init)\n",
    "F_train = nn_cost(params, X_train)\n",
    "F_val = nn_cost(params, X_val)\n",
    "print(rmse(F_train, y_train), rmse(F_val, y_val))\n",
    "\n",
    "# a sensible random initialization of the parameters: glorot initialization\n",
    "init = init_nn_parameter_glorot(X_train.shape[1], K)\n",
    "params = fit_nn_gradopt(X_train, y_train, alpha, init)\n",
    "F_train = nn_cost(params, X_train)\n",
    "F_val = nn_cost(params, X_val)\n",
    "print(rmse(F_train, y_train), rmse(F_val, y_val))"
   ],
   "id": "c9a4035f3080b6e9",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1390997200990414 0.2690686604902916\n",
      "0.14264988206649667 0.2790158966289581\n"
     ]
    }
   ],
   "execution_count": 11
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# Task 5: Bayesian optimisation",
   "id": "8a17f4f096291255"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "A popular application area of Gaussian processes is Bayesian optimisation, where the uncertainty in the probabilistic model is used to guide the optimisation of a function. Here we will use Bayesian optimisation with Gaussian processes for choosing the regularisation parameter $\\alpha$. (We would normally use Bayesian optimisation when optimizing more than one parameter.)\n",
    "\n",
    "Gaussian processes are used to represent our belief about an unknown function. In this case, the function we are interested in is the neural network’s validation log root mean square error (log RMSE) as a function of the regularisation paramter $\\alpha$. In Bayesian optimisation, it is common to maximise the unknown function, so we will maximise the negative log RMSE.\n",
    "\n",
    "We start with a Gaussian process prior over this function. As we observe the actual log RMSEs for particular $\\alpha$’s we update our belief about the function by calculating the Gaussian process posterior.\n",
    "\n",
    "Besides the Gaussian process framework that you’re already familiar with, Bayesian optimisation involves a so-called acquisition function. Given our Gaussian process posterior model, we use this function to decide which parameter to query next. The acquisition function describes how useful we think it will be to try a given $\\alpha$, while considering the uncertainty that is represented in our posterior belief.\n",
    "\n",
    "There are many popular acquisition functions in Bayesian optimisation. One example is the probability of improvement. Suppose we have observed $y^{(1)}$ to $y^{(N)}$ (here negative log RMSE at locations $\\alpha^{(1)}$ to $\\alpha^{(N)}$). Then the function takes the following form: $$\n",
    "    \\mathit{PI}(\\alpha) = \\Phi\\left(\\frac{\\mu(\\alpha) - \\text{max}(y^{(1)},\\dots,y^{N})}{\\sigma(\\alpha)}\\right),$$ where $\\mu(\\alpha)$ is the Gaussian process posterior mean at location $\\alpha$, $\\sigma(\\alpha)$ is the posterior standard deviation at location $\\alpha$, and $\\Phi$ denotes the cumulative density function of the Gaussian with mean 0 and variance 1.\n",
    "    \n",
    "We pick the next regularization constant $\\alpha^{(N+1)}$ by maximizing the acquisition function: $$\n",
    "    \\alpha^{(N+1)} = {arg\\,max}_{\\alpha}\\mathit{PI}(\\alpha).$$\n",
    "We then evaluate our model for this regularization parameter and update our posterior about the unknown function that maps $\\alpha$ to negative log RMSE. We repeat the procedure multiple times and then pick the parameter that yielded the best performance $y$.\n",
    "\n",
    "Write a function train_nn_reg that trains the neural network from Q4 for a given $\\alpha$ parameter and returns the validation RMSE.\n",
    "\n",
    "Consider $\\alpha$ on the range ```np.arange(0, 50, 0.02)```. Pick three values from this set for $\\alpha$ as training locations and use ```train_nn_reg``` on these locations. Use the remaining locations as values that you will consider with the acquisition function.\n",
    "\n",
    "Take the performance of the neural network that you trained in Q4 a) as a baseline. Subtract your $\\alpha$-observed log RMSEs from the log of this baseline and take the resulting values as $y^{(1)}$ to $y^{(3)}$. Then calculate the Gaussian process posterior for these observations. To do so, use ``gp_post_par`` from the support code. Use the default parameters for ``sigma_y``, ``ell`` and ``sigma_f``.\n",
    "\n",
    "Implement the probability of improvement acquisition function. You can use scipy.stats.norm.cdf to calculate $\\Phi$. With this acquisition function, iteratively pick new $\\alpha$’s, re-train and evaluate each new model with ```train_nn_reg``` and update your posterior with ```gp_post_par```. Do five of these iterations.\n",
    "\n",
    "Report the maximum probability of improvement together with its $\\alpha$ for each of the five iterations. Also report the best $\\alpha$ that this procedure found, and its validation and test RMSEs. Have we improved the model?\n",
    "\n",
    "In this question, the function we are optimizing is the neural network’s validation log RMSE as a function of the regularisation parameter $\\alpha$. Where is the observation noise coming from?"
   ],
   "id": "905a8037892b33f6"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "写一个函数``train_nn_reg``，它为给定的$\\alpha$参数训练Q4中的神经网络，并返回验证RMSE。",
   "id": "b5fb945b6934ee6e"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:54:44.870252Z",
     "start_time": "2024-11-15T10:54:44.866327Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def train_nn_reg(alpha, X_train, y_train, X_val, y_val):\n",
    "    K = 20\n",
    "    D = X_train.shape[1]\n",
    "\n",
    "    # Initialize parameters\n",
    "    V_init_glorot = glorot_init(D, K)\n",
    "    ww_init_glorot = glorot_init(K, 1).flatten()\n",
    "    bk_init_glorot = np.zeros(K)\n",
    "    bb_init_glorot = 0.0\n",
    "\n",
    "    params_init_glorot = [ww_init_glorot, bb_init_glorot, V_init_glorot, bk_init_glorot]\n",
    "\n",
    "    # Set optimization arguments\n",
    "    args = (X_train, y_train, alpha)\n",
    "\n",
    "    # Train the neural network\n",
    "    params_opt_glorot = minimize_list(nn_cost, params_init_glorot, args)\n",
    "\n",
    "    y_val_pred = nn_cost(params_opt_glorot, X_val)\n",
    "\n",
    "    rmse_val_glorot = rmse(y_val, y_val_pred)\n",
    "\n",
    "    return rmse_val_glorot\n"
   ],
   "id": "dbe5125218d6ffc1",
   "outputs": [],
   "execution_count": 12
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:54:45.157327Z",
     "start_time": "2024-11-15T10:54:44.871450Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from scipy.stats import norm\n",
    "\n",
    "def acquisition_function_PI(mu, sigma, y_max):\n",
    "    with np.errstate(divide='warn'):\n",
    "        Z = (mu - y_max) / sigma\n",
    "        PI_values = norm.cdf(Z)\n",
    "    return PI_values"
   ],
   "id": "bb34a94e0a256712",
   "outputs": [],
   "execution_count": 13
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:56:35.798327Z",
     "start_time": "2024-11-15T10:54:45.157986Z"
    }
   },
   "cell_type": "code",
   "source": [
    "baseline_rmse = rmse(F_val, y_val) # Calculated in Q4\n",
    "baseline_log_rmse = np.log(baseline_rmse)\n",
    "\n",
    "# initial observation negative log RMSE\n",
    "np.random.seed(42)\n",
    "initial_alphas = np.random.choice(np.arange(0, 50, 0.2), size=3, replace=False)\n",
    "initial_log_rmses = [np.log(train_nn_reg(alpha, X_train, y_train, X_val, y_val)) for alpha in initial_alphas]\n",
    "initial_y = baseline_log_rmse - np.array(initial_log_rmses)\n",
    "\n",
    "print(\"Initial alphas and observed log RMSE differences (y values):\")\n",
    "for alpha, y in zip(initial_alphas, initial_y):\n",
    "    print(f\"alpha = {alpha}, y = {y}\")\n"
   ],
   "id": "447eec02db9c7725",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial alphas and observed log RMSE differences (y values):\n",
      "alpha = 28.400000000000002, y = 0.05944656396473902\n",
      "alpha = 1.2000000000000002, y = 0.13014428271547174\n",
      "alpha = 19.400000000000002, y = 0.04835829408831027\n"
     ]
    }
   ],
   "execution_count": 14
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T10:56:35.834146Z",
     "start_time": "2024-11-15T10:56:35.807190Z"
    }
   },
   "cell_type": "code",
   "source": [
    "remaining_alphas = np.setdiff1d(np.arange(0, 50, 0.2), initial_alphas)\n",
    "rest_mu, rest_cov = gp_post_par(remaining_alphas, initial_alphas, initial_y)"
   ],
   "id": "cbc74bfef2d85039",
   "outputs": [],
   "execution_count": 15
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-15T11:00:26.049514Z",
     "start_time": "2024-11-15T10:56:35.846287Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import numpy as np\n",
    "\n",
    "best_alpha = None\n",
    "best_rmse = float('inf')\n",
    "observed_alphas = list(initial_alphas)\n",
    "observed_ys = list(initial_y)\n",
    "\n",
    "for iteration in range(5):\n",
    "    # get std from posterior\n",
    "    rest_std = np.sqrt(np.diag(rest_cov))\n",
    "\n",
    "    y_max = max(observed_ys)\n",
    "\n",
    "    PI_values = acquisition_function_PI(rest_mu, rest_std, y_max)\n",
    "\n",
    "    # select alpha with max PI\n",
    "    max_PI_index = np.argmax(PI_values)\n",
    "    next_alpha = remaining_alphas[max_PI_index]\n",
    "    max_PI = PI_values[max_PI_index]\n",
    "    \n",
    "    # train_nn_reg on next alpha\n",
    "    next_rmse = train_nn_reg(next_alpha, X_train, y_train, X_val, y_val)\n",
    "    next_log_rmse = np.log(next_rmse)\n",
    "    next_y = baseline_log_rmse - next_log_rmse\n",
    "    \n",
    "    # update observed alphas and ys\n",
    "    observed_alphas.append(next_alpha)\n",
    "    observed_ys.append(next_y)\n",
    "    \n",
    "    # transform to numpy arrays\n",
    "    observed_alphas_np = np.array(observed_alphas)\n",
    "    observed_ys_np = np.array(observed_ys)\n",
    "    \n",
    "    # update GP posterior\n",
    "    rest_mu, rest_cov = gp_post_par(remaining_alphas, observed_alphas_np, observed_ys_np)\n",
    "    \n",
    "    print(f\"Iteration {iteration + 1}: max PI = {max_PI}, alpha = {next_alpha}, rmse = {next_rmse}\")\n",
    "    \n",
    "    # update best alpha\n",
    "    if next_rmse < best_rmse:\n",
    "        best_rmse = next_rmse\n",
    "        best_alpha = next_alpha\n",
    "\n",
    "print(\"\\nBest alpha found:\", best_alpha)\n",
    "print(\"Validation RMSE with best alpha:\", best_rmse)\n",
    "\n",
    "rmse_test = train_nn_reg(best_alpha, X_train, y_train, X_test, y_test)\n",
    "print(\"Test RMSE with best alpha:\", rmse_test)\n"
   ],
   "id": "c9facfdb247d582c",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1: max PI = 0.2804245624017896, alpha = 1.6, rmse = 0.23935159258761723\n",
      "Iteration 2: max PI = 0.23503502716452213, alpha = 3.4000000000000004, rmse = 0.2395033650014727\n",
      "Iteration 3: max PI = 0.2999777519657494, alpha = 3.4000000000000004, rmse = 0.26458917834253787\n",
      "Iteration 4: max PI = 0.21406486876376557, alpha = 0.0, rmse = 0.26697369255713543\n",
      "Iteration 5: max PI = 0.10804755433916247, alpha = 6.6000000000000005, rmse = 0.2432924573339652\n",
      "\n",
      "Best alpha found: 1.6\n",
      "Validation RMSE with best alpha: 0.23935159258761723\n",
      "Test RMSE with best alpha: 0.27282901141603133\n"
     ]
    }
   ],
   "execution_count": 16
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# What next?\n",
    "Try to improve regression performance beyond the methods we have tried so far. Explain what you tried, why you thought it might work better, how you evaluated your idea, and what you found.\n",
    "\n",
    "Do not write more than 300 words for this part, not including your code (which you do need to include). The bulk of the marks available for this part are for motivating and trying something sensible, which can be simple, and evaluating it sensibly.\n",
    "\n"
   ],
   "id": "35ea9d9ecec6b7a5"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-17T12:38:33.740453Z",
     "start_time": "2024-11-17T12:38:33.728809Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 2-layers neural network\n",
    "def nn2_cost(params, X, yy=None, alpha=None):\n",
    "    \"\"\"NN2_COST neural network with two hidden layers cost function and gradients, or predictions.\n",
    "\n",
    "       E, params_bar = nn2_cost([ww, bb, V1, bk1, V2, bk2], X, yy, alpha)\n",
    "                pred = nn2_cost([ww, bb, V1, bk1, V2, bk2], X)\n",
    "\n",
    "     Inputs:\n",
    "             params (ww, bb, V1, bk1, V2, bk2), where:\n",
    "                    ------------------------------------\n",
    "                        ww K2,  hidden-output weights\n",
    "                        bb      scalar output bias\n",
    "                        V1 K1,D first hidden-input weights\n",
    "                        bk1 K1, first hidden biases\n",
    "                        V2 K2,K1 second hidden-hidden weights\n",
    "                        bk2 K2, second hidden biases\n",
    "                    ------------------------------------\n",
    "                  X N,D input design matrix\n",
    "                 yy N,  regression targets\n",
    "              alpha     scalar regularization for weights\n",
    "\n",
    "     Outputs:\n",
    "                     E  sum of squares error\n",
    "            params_bar  gradients wrt params, same format as params\n",
    "     OR\n",
    "               pred N,  predictions if only params and X are given as inputs\n",
    "    \"\"\"\n",
    "    # Unpack parameters from list\n",
    "    ww, bb, V1, bk1, V2, bk2 = params\n",
    "\n",
    "    # Forward computation\n",
    "    # First hidden layer\n",
    "    A1 = np.dot(X, V1.T) + bk1[None, :]  # N x K1\n",
    "    P1 = 1 / (1 + np.exp(-A1))  # N x K1\n",
    "\n",
    "    # Second hidden layer\n",
    "    A2 = np.dot(P1, V2.T) + bk2[None, :]  # N x K2\n",
    "    P2 = 1 / (1 + np.exp(-A2))  # N x K2\n",
    "\n",
    "    # Output layer\n",
    "    F = np.dot(P2, ww) + bb  # N,\n",
    "\n",
    "    if yy is None:\n",
    "        return F\n",
    "\n",
    "    # Compute cost\n",
    "    res = F - yy  # N,\n",
    "    E = np.dot(res, res) + alpha * (np.sum(V1 * V1) + np.sum(V2 * V2) + np.dot(ww, ww))  # Scalar\n",
    "\n",
    "    # Reverse computation of gradients\n",
    "    F_bar = 2 * res  # N,\n",
    "\n",
    "    # Gradients for output layer\n",
    "    ww_bar = np.dot(P2.T, F_bar) + 2 * alpha * ww  # K2,\n",
    "    bb_bar = np.sum(F_bar)  # Scalar\n",
    "\n",
    "    # Gradients for second hidden layer\n",
    "    P2_bar = np.outer(F_bar, ww)  # N x K2\n",
    "    A2_bar = P2_bar * P2 * (1 - P2)  # N x K2\n",
    "\n",
    "    V2_bar = np.dot(A2_bar.T, P1) + 2 * alpha * V2  # K2 x K1\n",
    "    bk2_bar = np.sum(A2_bar, axis=0)  # K2\n",
    "\n",
    "    # Gradients for first hidden layer\n",
    "    P1_bar = np.dot(A2_bar, V2)  # N x K1\n",
    "    A1_bar = P1_bar * P1 * (1 - P1)  # N x K1\n",
    "\n",
    "    V1_bar = np.dot(A1_bar.T, X) + 2 * alpha * V1  # K1 x D\n",
    "    bk1_bar = np.sum(A1_bar, axis=0)  # K1\n",
    "\n",
    "    # Pack gradients into a tuple matching params\n",
    "    params_bar = (ww_bar, bb_bar, V1_bar, bk1_bar, V2_bar, bk2_bar)\n",
    "\n",
    "    return E, params_bar\n"
   ],
   "id": "42f4e2c47e53c547",
   "outputs": [],
   "execution_count": 18
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-17T12:45:32.493083Z",
     "start_time": "2024-11-17T12:44:49.048456Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_nn2_gradopt(X, yy, alpha, init):\n",
    "    args = (X, yy, alpha)\n",
    "    params = minimize_list(nn2_cost, init, args)\n",
    "    return params\n",
    "\n",
    "# Training the two-hidden-layer neural network\n",
    "H1 = 20  # Number of units in the first hidden layer\n",
    "H2 = 10  # Number of units in the second hidden layer\n",
    "alpha = best_alpha  # Using the best alpha found earlier\n",
    "\n",
    "# Initialize parameters\n",
    "def init_nn2_parameters(D, H1, H2):\n",
    "    V1 = glorot_init(D, H1)  # K1 x D\n",
    "    bk1 = np.zeros(H1)\n",
    "    V2 = glorot_init(H1, H2)  # K2 x K1\n",
    "    bk2 = np.zeros(H2)\n",
    "    ww = glorot_init(H2, 1).flatten()  # K2\n",
    "    bb = 0.0\n",
    "    return [ww, bb, V1, bk1, V2, bk2]\n",
    "\n",
    "init_params = init_nn2_parameters(X_train.shape[1], H1, H2)\n",
    "\n",
    "# Train the network\n",
    "params_opt = fit_nn2_gradopt(X_train, y_train, alpha, init_params)\n",
    "\n",
    "# Evaluate on validation set\n",
    "F_val = nn2_cost(params_opt, X_val)\n",
    "rmse_val = rmse(y_val, F_val)\n",
    "print(\"Validation RMSE with two hidden layers:\", rmse_val)\n",
    "\n",
    "# Evaluate on test set\n",
    "F_test = nn2_cost(params_opt, X_test)\n",
    "rmse_test = rmse(y_test, F_test)\n",
    "print(\"Test RMSE with two hidden layers:\", rmse_test)"
   ],
   "id": "31135538807b108",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation RMSE with two hidden layers: 0.22696155725707892\n",
      "Test RMSE with two hidden layers: 0.2506189969910134\n"
     ]
    }
   ],
   "execution_count": 21
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": "",
   "id": "86a0ef46c18c4661"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}