assignment2-sketch-ycy.ipynb

{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "The patient IDs were removed from this version of the data, leaving 384 input features which were put in each of the ```“X_...”``` arrays. The corresponding CT scan slice location has been put in the ```“y_...”``` arrays. We shifted and scaled the ```“y_...”``` location values for the version of the data that you are using. The shift and scaling was chosen to make the training locations have zero mean and unit variance. The first 73 patients were put in the ```_train``` arrays, the next 12 in the ```_val``` arrays, and the final 12 in the ```_test``` arrays. Please use this training, validation, test split as given. **Do not shuffle the data further in this assignment.**",
   "id": "80d738e442702dbb"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## Task 1: Get Started",
   "id": "6bcb47a17c3b9be7"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:08.088408Z",
     "start_time": "2024-11-13T23:31:07.636413Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import numpy as np\n",
    "data = np.load('ct_data.npz')\n",
    "X_train = data['X_train']; X_val = data['X_val']; X_test = data['X_test']\n",
    "y_train = data['y_train']; y_val = data['y_val']; y_test = data['y_test']"
   ],
   "id": "cc83d45f07a20002",
   "outputs": [],
   "execution_count": 1
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "Verify that (up to numerical rounding errors) the mean of the training positions in ```y_train``` is zero. The mean of the 5,785 positions in the ```y_val``` array is not zero. Report its mean with a “standard error”, temporarily assuming that each entry is independent. For comparison, also report the mean with a standard error of the first 5,785 entries in the ```y_train```. Explain how your results demonstrate that these standard error bars do not reliably indicate what the average of locations in future CT slice data will be. Why are standard error bars misleading here?",
   "id": "8b7423faf66959d7"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:08.374442Z",
     "start_time": "2024-11-13T23:31:08.368069Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Calculate mean and standard error for y_train\n",
    "y_train_mean = np.mean(y_train)\n",
    "y_train_std_error = np.std(y_train, ddof=1) / np.sqrt(len(y_train))\n",
    "\n",
    "# Calculate mean and standard error for the first 5,785 entries in y_train\n",
    "y_train_sample_mean = np.mean(y_train[:5785])\n",
    "y_train_sample_std_error = np.std(y_train[:5785], ddof=1) / np.sqrt(5785)\n",
    "\n",
    "# Calculate mean and standard error for y_val\n",
    "y_val_mean = np.mean(y_val)\n",
    "y_val_std_error = np.std(y_val, ddof=1) / np.sqrt(len(y_val))\n",
    "\n",
    "y_train_mean, y_train_std_error, y_train_sample_mean, y_train_sample_std_error, y_val_mean, y_val_std_error"
   ],
   "id": "57108587bc884c50",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-9.13868774539957e-15,\n",
       " 0.0049535309340638205,\n",
       " -0.44247687859693674,\n",
       " 0.011927303389170828,\n",
       " -0.2160085093241599,\n",
       " 0.01290449880016868)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 2
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "Some of the input features are constants: they take on the same value for every training example. Identify these features, and remove them from the input matrices in the training, validation, and testing sets.\n",
    "\n",
    "Moreover, some of the input features are duplicates: some of the columns in the training set are identical. For each training set column, discard any later columns that are identical. Discard the same columns from the validation and testing sets.\n",
    "\n",
    "**Use these modified input arrays for the rest of the assignment.** Keep the names of the arrays the same (X_train, etc.), so we know what they’re called. You should not duplicate the code from this part in future questions. We will assume it has been run, and that the modified data are available.\n",
    "\n",
    "**Warning: As in the real world, mistakes at this stage would invalidate all of your results. We strongly recommend checking your code, for example on small test examples where you can see what it’s doing.**\n",
    "\n",
    "Report which columns of the X_... arrays you remove at each of the two stages. Report these as 0-based indexes. (For the second stage, you might report indexes in the original array, or after you did the first stage. It doesn’t matter, as long as your code is clear and correct.)"
   ],
   "id": "4d17bcaf6c45c429"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:10.871902Z",
     "start_time": "2024-11-13T23:31:10.274041Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Step 1: Identify and remove constant columns\n",
    "constant_columns = [i for i in range(X_train.shape[1]) if np.all(X_train[:, i] == X_train[0, i])]\n",
    "X_train = np.delete(X_train, constant_columns, axis=1)\n",
    "X_val = np.delete(X_val, constant_columns, axis=1)\n",
    "X_test = np.delete(X_test, constant_columns, axis=1)\n",
    "\n",
    "# Step 2: Identify and remove duplicate columns\n",
    "_, unique_indices = np.unique(X_train, axis=1, return_index=True)\n",
    "duplicate_columns = [i for i in range(X_train.shape[1]) if i not in unique_indices]\n",
    "X_train = np.delete(X_train, duplicate_columns, axis=1)\n",
    "X_val = np.delete(X_val, duplicate_columns, axis=1)\n",
    "X_test = np.delete(X_test, duplicate_columns, axis=1)\n",
    "\n",
    "# Report columns removed in each stage\n",
    "print(\"Constant columns removed:\", constant_columns)\n",
    "print(\"Duplicate columns removed:\", duplicate_columns)"
   ],
   "id": "d70de25576b49fa7",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Constant columns removed: [59, 69, 179, 189, 351]\n",
      "Duplicate columns removed: [76, 77, 185, 195, 283, 354]\n"
     ]
    }
   ],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# Task 2: Linear Regression Baseline\n",
    "Using ```numpy.linalg.lstsq```, write a short function “fit_linreg(X, yy, alpha)” that fits the linear regression model\n",
    "$$f(\\b x;\\b w,b) = \\b w^\\top\\b x + b,$$\n",
    "by minimizing the cost function:\n",
    "$$E(\\b w, b) = \\alpha\\b w^\\top\\b w + \\sum_n (f(\\b x^{(n)};\\b w,b) - y^{(n)})^2,$$\n",
    "with regularization constant $\\alpha$. As discussed in the lecture materials, fitting a bias parameter $b$ and incorporating the regularization constant can both be achieved by augmenting the original data arrays. Use a data augmentation approach that maintains the numerical stability of the underlying ```lstsq``` solver, rather than a ‘normal equations’ approach. You should only regularize the weights $\\textbf{w}$ and not the bias $b$.\n",
    "\n",
    "(In the lecture materials we used $\\lambda$ for the regularization constant, matching Murphy and others. However, lambda is a reserved word in Python, so we swapped to ```alpha``` for our code.)\n",
    "\n",
    "Use your function to fit weights and a bias to ```X_train``` and ```y_train```. Use $\\alpha = 30$.\n",
    "\n",
    "We can fit the same model with a gradient-based optimizer. The support code has a function ```fit_linreg_gradopt```, which you should look at and try.\n",
    "\n",
    "Report the root-mean-square errors (RMSE) on the training and validation sets for the parameters fitted using both your ```fit_linreg``` and the provided ```fit_linreg_gradopt```. Do you get exactly the same results? Why or why not?"
   ],
   "id": "6d38b97189a563d7"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:13.135599Z",
     "start_time": "2024-11-13T23:31:13.130614Z"
    }
   },
   "cell_type": "code",
   "source": [
    "X_train_num, features_num = X_train.shape\n",
    "X_train_num, features_num"
   ],
   "id": "ecf9763f85f3ad88",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40754, 373)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 4
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:13.628126Z",
     "start_time": "2024-11-13T23:31:13.603356Z"
    }
   },
   "cell_type": "code",
   "source": [
    "phi = np.concatenate([X_train, np.zeros((X_train_num, 1))], axis=1)\n",
    "phi.shape, phi[:, -1]"
   ],
   "id": "6389da8485068e9",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((40754, 374), array([0., 0., 0., ..., 0., 0., 0.]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 5
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:14.046911Z",
     "start_time": "2024-11-13T23:31:14.016593Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Not use matrix operations--high computing cost\n",
    "\"\"\"\n",
    "for i in range(features_num + 1):\n",
    "    add_row = np.zeros(features_num + 1)\n",
    "    add_row[i] = 1\n",
    "    phi = np.concatenate([phi, add_row[np.newaxis, :]])\n",
    "phi[-1, -1] = 0\n",
    "\"\"\"\n",
    "\n",
    "# Use matrix operations -- get a eye matrix\n",
    "identity_matrix = np.eye(features_num + 1)\n",
    "identity_matrix[-1, -1] = 0\n",
    "\n",
    "phi = np.concatenate([phi, identity_matrix])\n",
    "\n",
    "identity_matrix, phi[-1, -1], phi.shape"
   ],
   "id": "ff68c9e975e4f7ea",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[1., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 1., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 1., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 1., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 1., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]]),\n",
       " 0.0,\n",
       " (41128, 374))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 6
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:15.024510Z",
     "start_time": "2024-11-13T23:31:15.020814Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# try\n",
    "y_train.shape, np.zeros((features_num + 1)).shape, np.zeros((features_num + 1, 1)).shape"
   ],
   "id": "53d0bf3e16277297",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((40754,), (374,), (374, 1))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 7
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:15.468884Z",
     "start_time": "2024-11-13T23:31:15.465961Z"
    }
   },
   "cell_type": "code",
   "source": [
    "Y = np.concatenate([y_train, np.zeros(features_num + 1)])\n",
    "Y.shape, Y[:, np.newaxis].shape, phi.shape"
   ],
   "id": "883b74f34f163aed",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((41128,), (41128, 1), (41128, 374))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T23:31:17.312282Z",
     "start_time": "2024-11-13T23:31:15.943493Z"
    }
   },
   "cell_type": "code",
   "source": [
    "w = np.linalg.lstsq(phi, Y[:, np.newaxis])\n",
    "w"
   ],
   "id": "c60ae2e7aa921364",
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/bw/kk6865h577s1fckglc0_xhz80000gn/T/ipykernel_81877/1503166684.py:1: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.\n",
      "To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.\n",
      "  w = np.linalg.lstsq(phi, Y[:, np.newaxis])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(array([[-5.14849854e-02],\n",
       "        [-1.08631266e-01],\n",
       "        [ 8.22456499e-02],\n",
       "        [ 2.83872801e-01],\n",
       "        [ 2.60394488e-01],\n",
       "        [ 1.19494049e-01],\n",
       "        [ 1.68744858e-02],\n",
       "        [ 2.35944393e-01],\n",
       "        [-3.20385336e-01],\n",
       "        [-1.73865238e-02],\n",
       "        [-5.03429199e-02],\n",
       "        [ 5.29468044e-02],\n",
       "        [-2.44884609e-02],\n",
       "        [-1.77291987e-03],\n",
       "        [-1.04068325e-02],\n",
       "        [ 4.06039770e-02],\n",
       "        [ 2.52409523e-02],\n",
       "        [ 2.23186640e-02],\n",
       "        [-1.22311809e-01],\n",
       "        [ 1.68718904e-01],\n",
       "        [ 7.94975956e-02],\n",
       "        [ 2.74324075e-02],\n",
       "        [-4.27183236e-02],\n",
       "        [-1.69503524e-03],\n",
       "        [-6.15091084e-02],\n",
       "        [-4.89240453e-02],\n",
       "        [-1.31386948e-02],\n",
       "        [-6.05110555e-01],\n",
       "        [ 4.90115638e-01],\n",
       "        [ 2.26251471e-01],\n",
       "        [-1.11229878e-02],\n",
       "        [ 3.12659183e-03],\n",
       "        [-5.45543385e-03],\n",
       "        [-8.01311822e-03],\n",
       "        [ 6.67414373e-02],\n",
       "        [-1.82856145e-01],\n",
       "        [-1.97597401e-01],\n",
       "        [ 2.75981839e-01],\n",
       "        [-1.85284617e-01],\n",
       "        [ 4.87995936e-02],\n",
       "        [-1.88197360e-01],\n",
       "        [ 2.59297096e-02],\n",
       "        [-1.83153676e-01],\n",
       "        [ 5.86212064e-02],\n",
       "        [-1.47153812e-01],\n",
       "        [-2.51944876e-02],\n",
       "        [-4.56600829e-03],\n",
       "        [ 9.77398508e-02],\n",
       "        [-1.42179107e+00],\n",
       "        [-2.60561447e-01],\n",
       "        [-9.27384744e-02],\n",
       "        [-4.55688389e-03],\n",
       "        [-7.94468416e-02],\n",
       "        [-6.17183678e-02],\n",
       "        [-3.10077086e-02],\n",
       "        [-8.35938334e-02],\n",
       "        [ 8.15694476e-02],\n",
       "        [-2.20078277e-01],\n",
       "        [-4.05408580e-02],\n",
       "        [-7.42666837e-02],\n",
       "        [ 6.52362590e-02],\n",
       "        [-1.35571581e-02],\n",
       "        [-1.27741412e-01],\n",
       "        [-1.11969987e-01],\n",
       "        [-1.03587308e-01],\n",
       "        [-1.25697634e-01],\n",
       "        [ 2.42102875e-01],\n",
       "        [ 1.51358823e-01],\n",
       "        [ 2.53985447e-02],\n",
       "        [ 8.39776001e-02],\n",
       "        [ 1.21492883e-01],\n",
       "        [ 2.57845777e-02],\n",
       "        [ 6.15017330e-02],\n",
       "        [ 6.64416079e-02],\n",
       "        [-1.73185759e-01],\n",
       "        [-3.49117144e-01],\n",
       "        [ 5.85871389e-02],\n",
       "        [-5.05140632e-02],\n",
       "        [-6.31037305e-02],\n",
       "        [ 2.50333463e-02],\n",
       "        [-2.37500655e-03],\n",
       "        [-9.48725737e-02],\n",
       "        [-1.68604641e-02],\n",
       "        [ 1.86342611e-01],\n",
       "        [ 9.00122476e-02],\n",
       "        [-2.24155290e-01],\n",
       "        [ 6.03145950e-02],\n",
       "        [-1.33117099e-01],\n",
       "        [ 6.02486770e-02],\n",
       "        [-4.27772877e-02],\n",
       "        [ 9.84523572e-03],\n",
       "        [ 3.64429476e-02],\n",
       "        [-1.21951975e-01],\n",
       "        [ 3.66876397e-01],\n",
       "        [ 6.78034525e-01],\n",
       "        [ 4.58759431e-01],\n",
       "        [-8.56129787e-02],\n",
       "        [-5.04049286e-03],\n",
       "        [ 4.06836881e-02],\n",
       "        [ 1.25673540e-02],\n",
       "        [ 3.47732483e-02],\n",
       "        [ 4.06893558e-02],\n",
       "        [ 2.26211607e-01],\n",
       "        [ 3.71560258e-02],\n",
       "        [-1.24155249e-01],\n",
       "        [-8.92104017e-02],\n",
       "        [-1.88950777e-01],\n",
       "        [ 3.61963964e-04],\n",
       "        [-3.38670170e-02],\n",
       "        [ 6.14330630e-02],\n",
       "        [ 2.40646469e-01],\n",
       "        [ 1.53840276e-01],\n",
       "        [ 1.14406189e-01],\n",
       "        [ 4.24117263e-04],\n",
       "        [-2.71991566e-01],\n",
       "        [-2.77112132e-01],\n",
       "        [-2.20398568e-01],\n",
       "        [ 1.03961017e-01],\n",
       "        [ 4.59659249e-02],\n",
       "        [-5.59494751e-02],\n",
       "        [-9.08479546e-03],\n",
       "        [-1.20504967e-02],\n",
       "        [-3.49898567e-02],\n",
       "        [-1.66064127e-02],\n",
       "        [-1.08640587e-01],\n",
       "        [ 1.48503648e-01],\n",
       "        [-8.62284901e-02],\n",
       "        [ 6.12540979e-02],\n",
       "        [ 2.24097829e-01],\n",
       "        [-1.15545323e-01],\n",
       "        [ 3.18206933e-02],\n",
       "        [-1.93150848e-01],\n",
       "        [-8.97204704e-02],\n",
       "        [-1.40577741e-01],\n",
       "        [-9.38665186e-02],\n",
       "        [ 9.59754236e-02],\n",
       "        [-2.22356879e-02],\n",
       "        [-6.64879009e-02],\n",
       "        [ 6.53826334e-02],\n",
       "        [ 1.31341310e-01],\n",
       "        [ 2.25431883e-02],\n",
       "        [-7.14132712e-02],\n",
       "        [-7.15045276e-02],\n",
       "        [-2.12496399e-02],\n",
       "        [ 1.38904257e-01],\n",
       "        [-5.76746230e-02],\n",
       "        [-7.48757493e-02],\n",
       "        [-8.75064137e-02],\n",
       "        [-9.54383966e-02],\n",
       "        [ 5.88233105e-02],\n",
       "        [-1.90755560e-02],\n",
       "        [ 1.48466094e-02],\n",
       "        [-8.73823342e-02],\n",
       "        [-3.79278208e-02],\n",
       "        [ 2.95932286e-02],\n",
       "        [ 1.50051690e-01],\n",
       "        [-1.33304009e-02],\n",
       "        [ 3.51263252e-02],\n",
       "        [ 1.95268310e-04],\n",
       "        [-7.07503416e-02],\n",
       "        [ 1.67746457e-02],\n",
       "        [-5.44364143e-02],\n",
       "        [ 1.21469699e-01],\n",
       "        [-1.35485455e-02],\n",
       "        [-9.19807942e-02],\n",
       "        [ 4.51164940e-01],\n",
       "        [-1.19356660e-01],\n",
       "        [-6.80323062e-03],\n",
       "        [-1.96655183e-02],\n",
       "        [ 8.82704466e-04],\n",
       "        [-1.11369203e-01],\n",
       "        [ 1.98542218e-02],\n",
       "        [-2.36388811e-01],\n",
       "        [ 4.99910536e-03],\n",
       "        [ 5.13967025e-01],\n",
       "        [-1.12763377e-01],\n",
       "        [-3.12455388e-02],\n",
       "        [ 1.02589750e-01],\n",
       "        [ 5.19725011e-02],\n",
       "        [-5.81524167e-03],\n",
       "        [ 9.99523490e-04],\n",
       "        [-6.15443922e-03],\n",
       "        [-2.37107368e-01],\n",
       "        [ 2.52868330e-03],\n",
       "        [ 1.36616259e-01],\n",
       "        [ 1.61547854e-02],\n",
       "        [-4.07902021e-02],\n",
       "        [ 2.50145346e-02],\n",
       "        [-1.06838398e-01],\n",
       "        [-7.72529684e-02],\n",
       "        [ 1.93836166e-02],\n",
       "        [-6.97632312e-02],\n",
       "        [-5.86514928e-02],\n",
       "        [-9.99444461e-03],\n",
       "        [-4.61907275e-02],\n",
       "        [-1.43956750e-02],\n",
       "        [-1.60077170e-02],\n",
       "        [ 2.23312030e-02],\n",
       "        [-5.63842159e-02],\n",
       "        [ 7.32767869e-02],\n",
       "        [-3.98167434e-03],\n",
       "        [-2.69515742e-02],\n",
       "        [-4.11016287e-02],\n",
       "        [-1.05499344e-01],\n",
       "        [ 9.62437643e-02],\n",
       "        [ 7.41896117e-03],\n",
       "        [ 6.11887243e-02],\n",
       "        [-1.07192337e-01],\n",
       "        [ 3.24673449e-04],\n",
       "        [-2.16471650e-01],\n",
       "        [-1.30198223e-01],\n",
       "        [ 1.12693788e-02],\n",
       "        [-7.65105220e-02],\n",
       "        [ 3.07304618e-02],\n",
       "        [ 1.62680171e-01],\n",
       "        [-2.54911395e-02],\n",
       "        [-7.06451682e-02],\n",
       "        [-1.18780838e-01],\n",
       "        [-1.30497568e-01],\n",
       "        [-1.89362064e-01],\n",
       "        [-1.27511686e-01],\n",
       "        [ 1.75103540e-01],\n",
       "        [-1.39495670e-01],\n",
       "        [ 2.83352272e-03],\n",
       "        [-1.84211504e-01],\n",
       "        [ 4.79942344e-02],\n",
       "        [-3.19315501e-02],\n",
       "        [ 4.15573391e-02],\n",
       "        [-1.04908219e-01],\n",
       "        [-1.14035489e-01],\n",
       "        [ 1.44397995e-01],\n",
       "        [-1.09380031e-01],\n",
       "        [ 8.35521308e-04],\n",
       "        [ 1.49423976e-02],\n",
       "        [-1.67427180e-01],\n",
       "        [ 2.29539590e-02],\n",
       "        [-4.18203614e-02],\n",
       "        [ 5.37042904e-02],\n",
       "        [ 9.40019448e-01],\n",
       "        [-5.22981973e-01],\n",
       "        [-2.72368687e-02],\n",
       "        [ 4.78936226e-02],\n",
       "        [-1.99070858e-02],\n",
       "        [ 8.14839168e-02],\n",
       "        [ 2.83017298e-02],\n",
       "        [-6.86677163e-03],\n",
       "        [-6.75072213e-01],\n",
       "        [ 3.83346814e-01],\n",
       "        [ 2.86987372e-02],\n",
       "        [-2.65632336e-02],\n",
       "        [-1.17634511e-03],\n",
       "        [ 3.35020516e-02],\n",
       "        [-8.92727437e-03],\n",
       "        [-3.52284225e-02],\n",
       "        [ 3.06211645e-01],\n",
       "        [ 1.05456308e-01],\n",
       "        [-9.40479634e-02],\n",
       "        [-1.51085377e-02],\n",
       "        [ 6.31306320e-02],\n",
       "        [-2.88709862e-02],\n",
       "        [ 7.66198003e-02],\n",
       "        [ 9.83081917e-02],\n",
       "        [-6.77959902e-02],\n",
       "        [ 1.73979301e-01],\n",
       "        [-4.25406875e-02],\n",
       "        [-2.43112805e-01],\n",
       "        [-7.61213635e-02],\n",
       "        [-1.66980294e-02],\n",
       "        [-2.31686802e-02],\n",
       "        [-1.45442554e-01],\n",
       "        [ 1.28750182e-01],\n",
       "        [-6.16547405e-01],\n",
       "        [-5.20125243e-02],\n",
       "        [-4.57657471e-02],\n",
       "        [ 2.97929481e-02],\n",
       "        [ 4.36534480e-02],\n",
       "        [-7.32887754e-03],\n",
       "        [-3.46468790e-03],\n",
       "        [ 2.72771897e-02],\n",
       "        [ 6.42596744e-02],\n",
       "        [ 4.10469252e-02],\n",
       "        [ 1.13688339e-02],\n",
       "        [ 3.45206002e-02],\n",
       "        [ 3.00580198e-02],\n",
       "        [ 4.12379659e-02],\n",
       "        [ 1.65331958e-01],\n",
       "        [ 1.02929120e-01],\n",
       "        [ 5.08506180e-02],\n",
       "        [-9.14613307e-02],\n",
       "        [ 5.67305144e-03],\n",
       "        [ 5.68922464e-03],\n",
       "        [ 6.42427571e-02],\n",
       "        [ 5.53394947e-02],\n",
       "        [-2.24786594e-01],\n",
       "        [-2.05485036e-01],\n",
       "        [-1.96824161e-03],\n",
       "        [-6.28500445e-02],\n",
       "        [ 8.87022822e-02],\n",
       "        [ 8.30701551e-02],\n",
       "        [-2.58803547e-02],\n",
       "        [-3.07044874e-02],\n",
       "        [ 3.00351154e-01],\n",
       "        [ 6.00454948e-01],\n",
       "        [-2.10985677e-02],\n",
       "        [-4.70607270e-02],\n",
       "        [-7.92702276e-02],\n",
       "        [ 4.98648876e-02],\n",
       "        [ 8.45846044e-02],\n",
       "        [ 3.69054703e-02],\n",
       "        [ 3.13026975e-02],\n",
       "        [ 3.94163605e-02],\n",
       "        [-2.37447959e-02],\n",
       "        [ 4.00648024e-02],\n",
       "        [-2.93819967e-02],\n",
       "        [-1.41437361e-01],\n",
       "        [ 3.13726054e-02],\n",
       "        [ 8.94819038e-03],\n",
       "        [ 3.31287511e-03],\n",
       "        [-2.01572527e-01],\n",
       "        [-7.02186409e-02],\n",
       "        [-4.67515549e-02],\n",
       "        [-2.90580362e-02],\n",
       "        [-3.09585465e-02],\n",
       "        [ 2.89295439e-02],\n",
       "        [ 1.53376617e-02],\n",
       "        [ 2.19077199e-02],\n",
       "        [-9.57259929e-02],\n",
       "        [-2.40200088e-02],\n",
       "        [-3.17389109e-02],\n",
       "        [-3.83123315e-02],\n",
       "        [-7.84089367e-02],\n",
       "        [ 8.96994142e-02],\n",
       "        [ 1.68955385e-01],\n",
       "        [-2.00529969e-01],\n",
       "        [ 1.40318292e-01],\n",
       "        [ 3.20231878e-02],\n",
       "        [ 2.43841769e-02],\n",
       "        [ 2.30908978e-02],\n",
       "        [ 5.28377745e-02],\n",
       "        [ 9.95484586e-02],\n",
       "        [ 2.18729063e-01],\n",
       "        [ 1.21000262e-02],\n",
       "        [ 1.95041603e-03],\n",
       "        [-3.03979821e-02],\n",
       "        [ 2.72776412e-04],\n",
       "        [ 1.13538013e-04],\n",
       "        [ 6.07784104e-02],\n",
       "        [ 7.15806089e-02],\n",
       "        [-2.03594990e-01],\n",
       "        [ 7.89646725e-03],\n",
       "        [ 4.22075998e-02],\n",
       "        [-1.49055835e-01],\n",
       "        [-5.54680900e-02],\n",
       "        [ 7.24058914e-02],\n",
       "        [ 1.65941024e-02],\n",
       "        [-4.98053709e-02],\n",
       "        [-1.70032534e-01],\n",
       "        [-1.65017079e-02],\n",
       "        [-1.44074961e-02],\n",
       "        [-2.55116914e-02],\n",
       "        [-4.91401032e-02],\n",
       "        [ 1.29190143e-02],\n",
       "        [-5.40185022e-02],\n",
       "        [-2.49239406e-01],\n",
       "        [ 1.04027611e-01],\n",
       "        [ 1.04241458e-02],\n",
       "        [-8.98915594e-03],\n",
       "        [-1.87638960e-02],\n",
       "        [ 6.13135014e-02],\n",
       "        [ 5.58511671e-02],\n",
       "        [ 1.25401074e-01],\n",
       "        [ 1.82493579e-01],\n",
       "        [ 3.38224335e-01],\n",
       "        [ 0.00000000e+00]]),\n",
       " array([], dtype=float64),\n",
       " 373,\n",
       " array([8.76447309e+02, 4.27198788e+02, 3.08222814e+02, 2.94171231e+02,\n",
       "        2.33452264e+02, 2.15714179e+02, 1.91496690e+02, 1.71415952e+02,\n",
       "        1.63374168e+02, 1.38010489e+02, 1.32766395e+02, 1.26759350e+02,\n",
       "        1.23707653e+02, 1.19205303e+02, 1.12044518e+02, 1.08436093e+02,\n",
       "        1.06348856e+02, 1.03601705e+02, 1.00492161e+02, 9.61587865e+01,\n",
       "        9.52403845e+01, 9.36842733e+01, 9.07389946e+01, 8.72557309e+01,\n",
       "        8.67630235e+01, 8.44734067e+01, 8.38435573e+01, 8.06275196e+01,\n",
       "        7.90508895e+01, 7.82530705e+01, 7.75060652e+01, 7.56825538e+01,\n",
       "        7.49720314e+01, 7.35997269e+01, 7.33765451e+01, 7.27071570e+01,\n",
       "        7.16753396e+01, 7.04415388e+01, 6.93458096e+01, 6.82031688e+01,\n",
       "        6.72467338e+01, 6.70319487e+01, 6.63874569e+01, 6.59250575e+01,\n",
       "        6.48146161e+01, 6.40763110e+01, 6.33330796e+01, 6.26996422e+01,\n",
       "        6.19778832e+01, 6.11640303e+01, 6.08957413e+01, 6.02708943e+01,\n",
       "        5.99312005e+01, 5.94430573e+01, 5.80401097e+01, 5.77880495e+01,\n",
       "        5.75344023e+01, 5.70475307e+01, 5.66625105e+01, 5.65812071e+01,\n",
       "        5.61467677e+01, 5.58561175e+01, 5.53452329e+01, 5.47111497e+01,\n",
       "        5.40729527e+01, 5.38036793e+01, 5.31373597e+01, 5.28628627e+01,\n",
       "        5.27150147e+01, 5.25020358e+01, 5.22613376e+01, 5.21230072e+01,\n",
       "        5.15469458e+01, 5.10895690e+01, 5.06737651e+01, 5.05558109e+01,\n",
       "        5.00067393e+01, 4.96560724e+01, 4.92852888e+01, 4.91890484e+01,\n",
       "        4.86478632e+01, 4.85167142e+01, 4.81825176e+01, 4.81203877e+01,\n",
       "        4.75085195e+01, 4.69400149e+01, 4.65602746e+01, 4.64004555e+01,\n",
       "        4.60592781e+01, 4.58826073e+01, 4.56211085e+01, 4.54492111e+01,\n",
       "        4.51069203e+01, 4.47856193e+01, 4.46623398e+01, 4.44773271e+01,\n",
       "        4.40808383e+01, 4.40229841e+01, 4.36983101e+01, 4.34893911e+01,\n",
       "        4.31889001e+01, 4.29418788e+01, 4.28415396e+01, 4.26007486e+01,\n",
       "        4.23582469e+01, 4.20766087e+01, 4.19280566e+01, 4.13981048e+01,\n",
       "        4.10446277e+01, 4.09187383e+01, 4.08636937e+01, 4.06468291e+01,\n",
       "        4.04986189e+01, 4.03797087e+01, 4.01368094e+01, 4.00286063e+01,\n",
       "        3.97186119e+01, 3.96110420e+01, 3.94484262e+01, 3.92879670e+01,\n",
       "        3.89847227e+01, 3.89032505e+01, 3.86258633e+01, 3.84589808e+01,\n",
       "        3.82507357e+01, 3.79541742e+01, 3.78353164e+01, 3.76425414e+01,\n",
       "        3.75774010e+01, 3.75245162e+01, 3.72265028e+01, 3.69771073e+01,\n",
       "        3.69582884e+01, 3.65110851e+01, 3.63585716e+01, 3.62502356e+01,\n",
       "        3.59693177e+01, 3.58206688e+01, 3.56423192e+01, 3.54768615e+01,\n",
       "        3.53364000e+01, 3.53058497e+01, 3.51301741e+01, 3.49814497e+01,\n",
       "        3.47355688e+01, 3.46135837e+01, 3.43588425e+01, 3.41972972e+01,\n",
       "        3.40952305e+01, 3.39672954e+01, 3.37913494e+01, 3.35736892e+01,\n",
       "        3.33884387e+01, 3.33251336e+01, 3.32455336e+01, 3.30509965e+01,\n",
       "        3.28995424e+01, 3.27504668e+01, 3.26283071e+01, 3.22908975e+01,\n",
       "        3.21789764e+01, 3.19873773e+01, 3.18643678e+01, 3.17525851e+01,\n",
       "        3.17142643e+01, 3.15712153e+01, 3.15203372e+01, 3.13903610e+01,\n",
       "        3.11257736e+01, 3.10043835e+01, 3.07684831e+01, 3.06861405e+01,\n",
       "        3.05860866e+01, 3.04786417e+01, 3.02138949e+01, 3.00561941e+01,\n",
       "        3.00332740e+01, 2.98861393e+01, 2.96774148e+01, 2.96377773e+01,\n",
       "        2.94574735e+01, 2.93565386e+01, 2.91063784e+01, 2.90246266e+01,\n",
       "        2.87117580e+01, 2.86738025e+01, 2.84982856e+01, 2.84156800e+01,\n",
       "        2.83610394e+01, 2.82219329e+01, 2.80276866e+01, 2.80041722e+01,\n",
       "        2.79130540e+01, 2.76402775e+01, 2.76071395e+01, 2.74174194e+01,\n",
       "        2.73666136e+01, 2.72770511e+01, 2.71386765e+01, 2.68446960e+01,\n",
       "        2.67655280e+01, 2.66979573e+01, 2.64964823e+01, 2.64091700e+01,\n",
       "        2.62924398e+01, 2.61667104e+01, 2.61086840e+01, 2.59038100e+01,\n",
       "        2.58373239e+01, 2.57444944e+01, 2.55529802e+01, 2.55087180e+01,\n",
       "        2.52903592e+01, 2.51809591e+01, 2.49855925e+01, 2.47931178e+01,\n",
       "        2.47375797e+01, 2.46523416e+01, 2.45169483e+01, 2.43684427e+01,\n",
       "        2.42962578e+01, 2.41673000e+01, 2.40416546e+01, 2.39743860e+01,\n",
       "        2.37408432e+01, 2.36698548e+01, 2.36297954e+01, 2.34225533e+01,\n",
       "        2.33252822e+01, 2.32748826e+01, 2.31895769e+01, 2.30896819e+01,\n",
       "        2.30587393e+01, 2.29426370e+01, 2.29087305e+01, 2.27114037e+01,\n",
       "        2.26149143e+01, 2.25695875e+01, 2.24796220e+01, 2.23661017e+01,\n",
       "        2.22202427e+01, 2.20785965e+01, 2.19548295e+01, 2.19323378e+01,\n",
       "        2.17350443e+01, 2.16873182e+01, 2.15471548e+01, 2.13876872e+01,\n",
       "        2.11837918e+01, 2.11097551e+01, 2.09446824e+01, 2.08365064e+01,\n",
       "        2.07467874e+01, 2.06309693e+01, 2.05697112e+01, 2.04591527e+01,\n",
       "        2.03364486e+01, 2.00958996e+01, 1.99845995e+01, 1.98792991e+01,\n",
       "        1.98318499e+01, 1.97366614e+01, 1.94963144e+01, 1.93327841e+01,\n",
       "        1.91368100e+01, 1.89437061e+01, 1.88069269e+01, 1.85648853e+01,\n",
       "        1.83468731e+01, 1.82386084e+01, 1.81869114e+01, 1.80266432e+01,\n",
       "        1.79700275e+01, 1.79046325e+01, 1.76808032e+01, 1.75387480e+01,\n",
       "        1.73622378e+01, 1.72921616e+01, 1.71806212e+01, 1.68198098e+01,\n",
       "        1.66540629e+01, 1.65767604e+01, 1.63558658e+01, 1.62355397e+01,\n",
       "        1.60801906e+01, 1.60295194e+01, 1.58184092e+01, 1.54839946e+01,\n",
       "        1.54417591e+01, 1.52032130e+01, 1.51124983e+01, 1.47183387e+01,\n",
       "        1.46243959e+01, 1.44367993e+01, 1.41081934e+01, 1.39923798e+01,\n",
       "        1.37832406e+01, 1.36305233e+01, 1.34574512e+01, 1.32957675e+01,\n",
       "        1.32349978e+01, 1.30622949e+01, 1.25631307e+01, 1.23203496e+01,\n",
       "        1.21935684e+01, 1.21255772e+01, 1.20322044e+01, 1.17813686e+01,\n",
       "        1.17143513e+01, 1.15531156e+01, 1.14658898e+01, 1.12659173e+01,\n",
       "        1.12264089e+01, 1.09002632e+01, 1.08152349e+01, 1.05114440e+01,\n",
       "        1.01624249e+01, 9.97741893e+00, 9.81778922e+00, 9.38488278e+00,\n",
       "        9.33007061e+00, 9.12455501e+00, 9.00462085e+00, 8.79513077e+00,\n",
       "        8.59831621e+00, 8.45770110e+00, 8.31044019e+00, 8.03212565e+00,\n",
       "        7.75312468e+00, 7.48104442e+00, 7.34172523e+00, 7.19172170e+00,\n",
       "        6.99395611e+00, 6.84161904e+00, 6.61051929e+00, 6.54122181e+00,\n",
       "        6.23331016e+00, 6.15212296e+00, 6.01779919e+00, 5.80681388e+00,\n",
       "        5.75021482e+00, 5.71105692e+00, 5.44881206e+00, 5.25477696e+00,\n",
       "        5.14328947e+00, 5.02721472e+00, 4.87939635e+00, 4.82905858e+00,\n",
       "        4.65684114e+00, 4.48323380e+00, 4.27899240e+00, 4.24180656e+00,\n",
       "        4.15733201e+00, 3.76987932e+00, 3.60683253e+00, 3.32377574e+00,\n",
       "        3.30525288e+00, 2.84256948e+00, 2.74171361e+00, 2.61202915e+00,\n",
       "        2.59890632e+00, 2.21483938e+00, 2.17772389e+00, 2.11298285e+00,\n",
       "        2.00819452e+00, 1.90034726e+00, 1.85643372e+00, 1.65601895e+00,\n",
       "        1.52923951e+00, 1.48780012e+00, 1.33224040e+00, 1.20444406e+00,\n",
       "        1.06311382e+00, 6.92834386e-14]))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 9
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:45:14.626506Z",
     "start_time": "2024-11-14T00:45:14.623159Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_linreg(X, yy, alpha):\n",
    "    X_train_num, features_num = X.shape\n",
    "    \n",
    "    # construct phi\n",
    "    phi = np.concatenate([X_train, np.ones((X_train_num, 1))], axis=1) \n",
    "    identity_matrix = np.eye(features_num + 1)\n",
    "    identity_matrix[-1, -1] = 0\n",
    "    phi = np.concatenate([phi, np.sqrt(alpha) * identity_matrix])\n",
    "    \n",
    "    # construct Y\n",
    "    Y = np.concatenate([yy, np.zeros(features_num + 1)])\n",
    "    \n",
    "    w = np.linalg.lstsq(phi, Y[:, np.newaxis], rcond=None)[0]\n",
    "    \n",
    "    return w[:-1, 0], w[-1, 0]\n",
    "\n",
    "def rmse(pred, yy):\n",
    "    return np.sqrt(np.mean((pred - yy) ** 2))"
   ],
   "id": "9ae1bbee023c1f4",
   "outputs": [],
   "execution_count": 41
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:45:16.327251Z",
     "start_time": "2024-11-14T00:45:14.828156Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# use lstsq\n",
    "w, b = fit_linreg(X_train, y_train, alpha=30)\n",
    "\n",
    "pred_train = X_train @ w + b\n",
    "pred_val = X_val @ w + b\n",
    "\n",
    "rmse_train_lst = rmse(pred_train, y_train)\n",
    "rmse_val_lst = rmse(pred_val, y_val)\n",
    "rmse_train_lst, rmse_val_lst"
   ],
   "id": "519cf929387bcbb6",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.3567565397204054, 0.4230521968394695)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 42
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:45:18.625172Z",
     "start_time": "2024-11-14T00:45:16.417470Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# use grad\n",
    "from support_code import *\n",
    "ww, bb = fit_linreg_gradopt(X_train, y_train, alpha=30)\n",
    "\n",
    "pred_train_grad = X_train @ ww + b\n",
    "pred_val_grad = X_val @ ww + b\n",
    "\n",
    "rmse_train_gd = rmse(pred_train_grad, y_train)\n",
    "rmse_val_gd = rmse(pred_val_grad, y_val)\n",
    "rmse_train_gd, rmse_val_gd"
   ],
   "id": "520508c3a4df72dd",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.3567561016332041, 0.4230114312756827)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 43
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# Task 3: Invented classification tasks",
   "id": "64214c672e41c6b9"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "It is often easier to work with binary data than real-valued data: we don’t have to think so hard about how the values might be distributed, and how we might process them. We will invent some binary classification tasks, and fit these.\n",
    "\n",
    "We will pick 20 positions within the range of training positions, and use each of these to define a classification task:"
   ],
   "id": "ad2aff00b4ecd398"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "The logistic regression cost function and gradients are provided with the assignment in the function ```logreg_cost```. It is analogous to the ```linreg_cost``` function for least-squares regression, which is used by the ```fit_linreg_gradopt``` function that you used earlier.\n",
    "\n",
    "Fit logistic regression to each of the 20 classification tasks above with $\\alpha=30$\n",
    ".\n",
    "\n",
    "Given a feature vector, we can now obtain 20 different probabilities, the predictions of the 20 logistic regression models. Transform both the training and validation input matrices into new matrices with 20 columns, containing the probabilities from the 20 logistic regression models. You don’t need to loop over the rows of ```X_train``` or ```X_val```, you can use array-based operations to make the logistic regression predictions for every datapoint at once."
   ],
   "id": "e5bc627a33b9e37"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:45:59.098873Z",
     "start_time": "2024-11-14T00:45:59.092845Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# test something to be used\n",
    "pred = np.array([[1, 2, 3, 4, 5, 6, 7]])\n",
    "\n",
    "pred = np.concatenate([pred, pred])\n",
    "print(pred)\n",
    "pred[pred > 4] = 100\n",
    "pred[pred <= 4] = 0\n",
    "print(pred)"
   ],
   "id": "44e0a2666b2aacf",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3 4 5 6 7]\n",
      " [1 2 3 4 5 6 7]]\n",
      "[[  0   0   0   0 100 100 100]\n",
      " [  0   0   0   0 100 100 100]]\n"
     ]
    }
   ],
   "execution_count": 44
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:45:59.245121Z",
     "start_time": "2024-11-14T00:45:59.242634Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_logreg_gradopt(X, yy, alpha):\n",
    "    D = X.shape[1]\n",
    "    args = (X, yy, alpha)\n",
    "    init = (np.zeros(D), np.array(0))\n",
    "    ww, bb = minimize_list(logreg_cost, init, args)\n",
    "    return ww, bb"
   ],
   "id": "37b87caccea36b4c",
   "outputs": [],
   "execution_count": 45
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:45:59.422547Z",
     "start_time": "2024-11-14T00:45:59.419061Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def logreg_k(X, yy, K, alpha=30):\n",
    "    mx = np.max(yy); mn = np.min(yy); hh = (mx-mn)/(K+1)\n",
    "    thresholds = np.linspace(mn+hh, mx-hh, num=K, endpoint=True)\n",
    "    \n",
    "    # concate方法 频繁concate并不好\n",
    "    # X_train_new = np.array([])\n",
    "    # 预先存在数组 效率更高\n",
    "    X_train_new = np.zeros((X.shape[0], K))\n",
    "    \n",
    "    models = []\n",
    "    \n",
    "    for kk in range(K):\n",
    "        # get binary training labels based on thresholds[kk]\n",
    "        labels = yy > thresholds[kk]\n",
    "        \n",
    "        # fit logistic regression to these labels\n",
    "        ww, bb = fit_logreg_gradopt(X, labels, alpha)\n",
    "        models.append((ww, bb))\n",
    "        \n",
    "        pred_term = X @ ww + bb\n",
    "        pred = 1 / (1 + np.exp(-pred_term))\n",
    "        \n",
    "        # transform to binary\n",
    "        # pred[pred >= 0.5] = 1\n",
    "        # pred[pred < 0.5] = 0\n",
    "        # 更好的方法：\n",
    "        # pred = np.where(pred >= 0.5, 1, 0)\n",
    "        \n",
    "        X_train_new[:, kk] = pred\n",
    "        \n",
    "        # # concate方法： concatenate logreg outputs together\n",
    "        # pred = pred.reshape(-1, 1)\n",
    "        # if X_train_new.shape[0] == 0:\n",
    "        #     X_train_new = pred\n",
    "        # else:\n",
    "        #     X_train_new = np.concatenate([X_train_new, pred], axis=1)\n",
    "            \n",
    "    return X_train_new, models"
   ],
   "id": "5eb2f07609ad41c8",
   "outputs": [],
   "execution_count": 46
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:46:08.733181Z",
     "start_time": "2024-11-14T00:45:59.649498Z"
    }
   },
   "cell_type": "code",
   "source": [
    "K = 20 # number of thresholded classification problems to fit\n",
    "\n",
    "# Transform both the training and validation input matrices into new matrices with 20 columns\n",
    "X_train_new, models = logreg_k(X_train, y_train, K)\n",
    "X_val_new = np.zeros((X_val.shape[0], K))\n",
    "k = 0\n",
    "for ww, bb in models:\n",
    "    pred_term = X_val @ ww + bb\n",
    "    pred = 1 / (1 + np.exp(-pred_term))\n",
    "    X_val_new[:, k] = pred\n",
    "    k = k + 1\n",
    "\n",
    "np.sum(X_train_new, axis=0), np.sum(X_val_new, axis=0), X_train_new, X_val_new"
   ],
   "id": "40bde25fa4418917",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([40047.99959223, 38424.00087396, 36789.03361221, 35155.00098873,\n",
       "        32969.95573317, 29150.98262565, 25498.04349378, 22587.02437339,\n",
       "        20035.04484011, 17485.02854007, 15020.99369934, 12809.0833263 ,\n",
       "        11068.03136158,  9460.95241151,  7861.98690246,  6262.99878295,\n",
       "         4662.0208563 ,  3064.99916823,  1461.99531663,   270.99939589]),\n",
       " array([5596.84695888, 5267.53315041, 5002.37270799, 4691.26128097,\n",
       "        4248.87686741, 3592.11305488, 3097.94446818, 2642.39321908,\n",
       "        2260.93868824, 1868.48505716, 1574.11959566, 1375.25140825,\n",
       "        1202.7425488 ,  993.36142465,  747.87863552,  658.83042942,\n",
       "         516.83226281,  303.1882229 ,  152.15405267,   19.56834674]),\n",
       " array([[9.99919980e-01, 9.96026867e-01, 9.67205666e-01, ...,\n",
       "         1.21649260e-05, 5.10796251e-05, 4.90089315e-04],\n",
       "        [9.99917091e-01, 9.95984594e-01, 9.64214471e-01, ...,\n",
       "         1.53870327e-05, 6.76737294e-05, 6.43990447e-04],\n",
       "        [9.99905325e-01, 9.96471771e-01, 9.70044306e-01, ...,\n",
       "         1.38034737e-05, 6.58848097e-05, 6.76305172e-04],\n",
       "        ...,\n",
       "        [9.99939642e-01, 9.99869875e-01, 9.99902151e-01, ...,\n",
       "         4.56353308e-04, 1.27949556e-04, 2.62021829e-04],\n",
       "        [9.99794369e-01, 9.99680623e-01, 9.99882384e-01, ...,\n",
       "         7.01577998e-03, 1.27213610e-03, 7.53799902e-04],\n",
       "        [9.99841970e-01, 9.99653001e-01, 9.99883200e-01, ...,\n",
       "         3.86843017e-03, 6.88788566e-04, 5.51860332e-04]]),\n",
       " array([[9.99245277e-01, 9.99159543e-01, 9.98884802e-01, ...,\n",
       "         2.32697760e-02, 4.58020682e-02, 2.25479608e-02],\n",
       "        [9.99249022e-01, 9.98928490e-01, 9.98246857e-01, ...,\n",
       "         1.34899467e-02, 2.67375707e-02, 1.37355926e-02],\n",
       "        [9.99587266e-01, 9.99422210e-01, 9.99341119e-01, ...,\n",
       "         1.24147467e-03, 1.26318305e-03, 1.89211543e-03],\n",
       "        ...,\n",
       "        [9.99573766e-01, 9.99806924e-01, 9.94303872e-01, ...,\n",
       "         1.34750577e-02, 1.07857207e-03, 1.01216790e-03],\n",
       "        [9.99492926e-01, 9.99762778e-01, 9.98929358e-01, ...,\n",
       "         4.04137316e-02, 7.06098932e-03, 7.74882048e-04],\n",
       "        [9.99761063e-01, 9.99832223e-01, 9.99860244e-01, ...,\n",
       "         4.74959894e-05, 2.58307012e-05, 4.23605059e-05]]))"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 47
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "Fit a regularized linear regression model ($\\alpha=30$) to your transformed 20-dimensional training set. Report the training and validation root mean square errors (RMSE) of this model.",
   "id": "b7077722bf0b3e67"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T00:46:09.101630Z",
     "start_time": "2024-11-14T00:46:08.810588Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from support_code import *\n",
    "ww, bb = fit_linreg_gradopt(X_train_new, y_train, alpha=30)\n",
    "\n",
    "pred_train = X_train_new @ ww + bb\n",
    "pred_val = X_val_new @ ww + bb\n",
    "\n",
    "rmse_train_gd = rmse(pred_train, y_train)\n",
    "rmse_val_gd = rmse(pred_val, y_val)\n",
    "rmse_train_gd, rmse_val_gd"
   ],
   "id": "e7e40e895c853a6b",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.15441185585977435, 0.25424947251332386)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 48
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-06T17:00:00.374545Z",
     "start_time": "2024-11-06T17:00:00.372446Z"
    }
   },
   "cell_type": "code",
   "source": "",
   "id": "ef76a338d07b08b7",
   "outputs": [],
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# Task 4: Small neural network",
   "id": "4ab85307b697a027"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "In Question 3 you fitted a small neural network. The logistic regression classifiers are sigmoidal hidden units, and a linear output unit predicts the outputs. However, you didn’t fit the parameters jointly to the obvious least squares cost function. A least squares cost function and gradients for this neural network are implemented in the nn_cost function provided.\n",
    "\n",
    "Try fitting the neural network model to the training set, with a) a sensible random initialization of the parameters; b) the parameters initialized using the fits made in Q3.\n",
    "\n",
    "Does one initialization strategy work better than the other? Does fitting the neural network jointly work better than the procedure in Q3? Your explanation should include any numbers that your answer is based on.\n"
   ],
   "id": "e978d0e1c3e0c2ce"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:06:36.084699Z",
     "start_time": "2024-11-14T01:06:36.079233Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def init_nn_parameter_q3(D, K):\n",
    "    ww_init = np.zeros(K)\n",
    "    V_init = np.zeros((K, D))\n",
    "    bk_init = np.zeros(K)\n",
    "    bb_init = 0.0\n",
    "\n",
    "    init = (ww_init, bb_init, V_init, bk_init)\n",
    "    return init"
   ],
   "id": "d76d2c06cdfcc99b",
   "outputs": [],
   "execution_count": 64
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:06:36.301778Z",
     "start_time": "2024-11-14T01:06:36.297122Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def glorot_init_uniform(n_in, n_out):\n",
    "    limit = np.sqrt(6 / (n_in + n_out))\n",
    "    return (np.random.rand(n_out, n_in) * 2 - 1) * limit \n",
    "\n",
    "def glorot_init_norm(n_in, n_out):\n",
    "    limit = np.sqrt(2 / (n_in + n_out))\n",
    "    return 0.1 * np.random.randn(n_out, n_in) * limit\n",
    "\n",
    "def init_nn_parameter_glorot_uniform(D, K):\n",
    "    \n",
    "    ww_init = glorot_init_uniform(K, 1).flatten()\n",
    "    bb_init = np.array(0)\n",
    "    V_init = glorot_init_uniform(D, K)\n",
    "    bk_init = np.zeros(K)\n",
    "    \n",
    "    init = (ww_init, bb_init, V_init, bk_init)\n",
    "    return init\n",
    "\n",
    "def init_nn_parameter_glorot_norm(D, K):\n",
    "    \n",
    "    ww_init = glorot_init_norm(K, 1).flatten()\n",
    "    bb_init = np.array(0)\n",
    "    V_init = glorot_init_norm(D, K)\n",
    "    bk_init = np.zeros(K)\n",
    "    \n",
    "    init = (ww_init, bb_init, V_init, bk_init)\n",
    "    return init"
   ],
   "id": "4f1e6193ae0220c8",
   "outputs": [],
   "execution_count": 65
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:06:36.474917Z",
     "start_time": "2024-11-14T01:06:36.471107Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def fit_nn_gradopt(X, yy, alpha, init_func=init_nn_parameter_glorot_uniform):\n",
    "    \"\"\"\n",
    "    Fit a regularized logistic regression model using gradient optimization.\n",
    "    \"\"\"\n",
    "    D = X.shape[1]\n",
    "    K = 20\n",
    "    args = (X, yy, alpha)\n",
    "    # Initialization\n",
    "    init = init_func(D, K)\n",
    "    ww, bb, V, bk = minimize_list(nn_cost, init, args)\n",
    "    return ww, bb, V, bk\n",
    "\n",
    "def pred_nn(X, params):\n",
    "    return nn_cost(params, X, yy=None, alpha=None)"
   ],
   "id": "ef5dcc3c78c2e6",
   "outputs": [],
   "execution_count": 66
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:06:36.853812Z",
     "start_time": "2024-11-14T01:06:36.650097Z"
    }
   },
   "cell_type": "code",
   "source": [
    "print('Use Q3 init.')\n",
    "params = fit_nn_gradopt(X_train, y_train, alpha=30, init_func=init_nn_parameter_q3)\n",
    "F_train = pred_nn(X_train, params)\n",
    "F_val = pred_nn(X_val, params)\n",
    "rmse(F_train, y_train), rmse(F_val, y_val)"
   ],
   "id": "6a963003b0f19bbd",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use Q3 init.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.9999877311903766, 1.0049112281771595)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 67
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:07:27.643673Z",
     "start_time": "2024-11-14T01:06:36.933089Z"
    }
   },
   "cell_type": "code",
   "source": [
    "print('Use glorot init uniform.')\n",
    "\n",
    "params = fit_nn_gradopt(X_train, y_train, alpha=30, init_func=init_nn_parameter_glorot_uniform)\n",
    "F_train = pred_nn(X_train, params)\n",
    "F_val = pred_nn(X_val, params)\n",
    "rmse(F_train, y_train), rmse(F_val, y_val)"
   ],
   "id": "76b7310942d366b3",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use glorot init uniform.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.14000849718782019, 0.27109724580734235)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 68
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:08:12.077325Z",
     "start_time": "2024-11-14T01:07:27.746492Z"
    }
   },
   "cell_type": "code",
   "source": [
    "print('Use glorot init norm.')\n",
    "params = fit_nn_gradopt(X_train, y_train, alpha=30, init_func=init_nn_parameter_glorot_norm)\n",
    "F_train = pred_nn(X_train, params)\n",
    "F_val = pred_nn(X_val, params)\n",
    "rmse(F_train, y_train), rmse(F_val, y_val)"
   ],
   "id": "c4639f57ca0b94da",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use glorot init norm.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.14193876455052917, 0.26503619412190044)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 69
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# Task 5: Bayesian optimisation",
   "id": "89e065c05e9bef3b"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "A popular application area of Gaussian processes is Bayesian optimisation, where the uncertainty in the probabilistic model is used to guide the optimisation of a function. Here we will use Bayesian optimisation with Gaussian processes for choosing the regularisation parameter $\\alpha$. (We would normally use Bayesian optimisation when optimizing more than one parameter.)\n",
    "\n",
    "Gaussian processes are used to represent our belief about an unknown function. In this case, the function we are interested in is the neural network’s validation log root mean square error (log RMSE) as a function of the regularisation paramter $\\alpha$. In Bayesian optimisation, it is common to maximise the unknown function, so we will maximise the negative log RMSE.\n",
    "\n",
    "We start with a Gaussian process prior over this function. As we observe the actual log RMSEs for particular $\\alpha$’s we update our belief about the function by calculating the Gaussian process posterior.\n",
    "\n",
    "Besides the Gaussian process framework that you’re already familiar with, Bayesian optimisation involves a so-called acquisition function. Given our Gaussian process posterior model, we use this function to decide which parameter to query next. The acquisition function describes how useful we think it will be to try a given $\\alpha$, while considering the uncertainty that is represented in our posterior belief.\n",
    "\n",
    "There are many popular acquisition functions in Bayesian optimisation. One example is the probability of improvement. Suppose we have observed $y^{(1)}$ to $y^{(N)}$ (here negative log RMSE at locations $\\alpha^{(1)}$ to $\\alpha^{(N)}$). Then the function takes the following form: $$\n",
    "    \\mathit{PI}(\\alpha) = \\Phi\\left(\\frac{\\mu(\\alpha) - \\text{max}(y^{(1)},\\dots,y^{N})}{\\sigma(\\alpha)}\\right),$$ where $\\mu(\\alpha)$"
   ],
   "id": "b845ec47f6bbdc3b"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": "",
   "id": "cde9ca1d98181be3"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}