Tutorial Five - Principle Component Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14b6a2e1",
   "metadata": {},
   "source": [
    "# Tutorial Five: Principle Component Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "193de056",
   "metadata": {},
   "source": [
    "## 实验前的准备\n",
    "\n",
    "本次实验我们载入一些Python的安装包，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c27bb779",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_diabetes\n",
    "\n",
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import seaborn as sns\n",
    "import scipy.stats as stats\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "from sklearn import preprocessing # Data Preprocessing\n",
    "import statsmodels.api as sm # LSE, Ridge Regression\n",
    "from statsmodels.stats.outliers_influence import variance_inflation_factor # VIF \n",
    "from statsmodels.multivariate.pca import PCA # PCR"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d5c727f",
   "metadata": {},
   "source": [
    "设置数据目录，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "7682d917",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"/Users/lyuni/ECNU_DaSE/Courses/Stat_ML/Experiment/Data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c4017d",
   "metadata": {},
   "source": [
    "在本次实验中，所使用的数据集是"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d9acf7f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   AtBat  Hits  HmRun  Runs  RBI  Walks  Years  CAtBat  CHits  CHmRun  CRuns  \\\n",
      "0    315    81      7    24   38     39     14    3449    835      69    321   \n",
      "1    479   130     18    66   72     76      3    1624    457      63    224   \n",
      "2    496   141     20    65   78     37     11    5628   1575     225    828   \n",
      "3    321    87     10    39   42     30      2     396    101      12     48   \n",
      "4    594   169      4    74   51     35     11    4408   1133      19    501   \n",
      "\n",
      "   CRBI  CWalks  PutOuts  Assists  Errors  Salary  \n",
      "0   414     375      632       43      10   475.0  \n",
      "1   266     263      880       82      14   480.0  \n",
      "2   838     354      200       11       3   500.0  \n",
      "3    46      33      805       40       4    91.5  \n",
      "4   336     194      282      421      25   750.0  \n"
     ]
    }
   ],
   "source": [
    "Data = pd.read_csv(\"Data_4.csv\")\n",
    "print(Data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57b35eac",
   "metadata": {},
   "source": [
    "我们可以了解这个数据集的样本量、特征维度等特征，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a397ac0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of features is 16\n",
      "The sample size is 263\n"
     ]
    }
   ],
   "source": [
    "p = Data.shape[1]-1\n",
    "n = Data.shape[0]\n",
    "\n",
    "print(\"The number of features is\",p)\n",
    "print(\"The sample size is\",n)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "714db853",
   "metadata": {},
   "source": [
    "## 背景\n",
    "\n",
    "本次数据来自于1986年至1987年间美国职业棒球大联盟。这个数据记录了263名大联盟的选手所采集的收入数据及其历史棒球运动的表现。\n",
    "\n",
    "## 数据\n",
    "在本次数据中，共有17个变量（1个响应变量与16个自变量），具体如下表所示：\n",
    "\n",
    "<table width = c(100,800,700), center = True>\n",
    "    <tr>\n",
    "        <td> 变量名                  </td>\n",
    "        <td> 英文变量含义                 </td>\n",
    "        <td> 中文变量含义                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> AtBat                          </td>\n",
    "        <td> Number of times at bat in 1986 </td>\n",
    "        <td> 1986年间轮到击球的次数            </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> Hits                         </td>\n",
    "        <td> Number of hits in 1986       </td>\n",
    "        <td> 1986年间击中球的次数           </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> HmRun                         </td>\n",
    "        <td> Number of home runs in 1986   </td>\n",
    "        <td> 1986年间全垒打的次数             </td>\n",
    "    </tr>\n",
    "     <tr>\n",
    "        <td> Runs                              </td>\n",
    "        <td> Number of runs in 1986            </td>\n",
    "        <td> 1986年间得分的次数                   </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> RBI                               </td>\n",
    "        <td> Number of runs batted in in 1986  </td>\n",
    "        <td> 1986年间打者打点的次数                   </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> Walks                               </td>\n",
    "        <td> Number of walks in 1986  </td>\n",
    "        <td> 1986年间打击手被保送的次数                   </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> Years                               </td>\n",
    "        <td> Number of years in the major leagues  </td>\n",
    "        <td> 在大联盟的年限                  </td>\n",
    "    </tr>\n",
    "     <tr>\n",
    "        <td> CAtBat                               </td>\n",
    "        <td> Number of times at bat during his career  </td>\n",
    "        <td> 在其棒球职业生涯中轮到击球的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> CHits                              </td>\n",
    "        <td> Number of hits during his career  </td>\n",
    "        <td> 在其棒球职业生涯中击中球的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> CHmRun                              </td>\n",
    "        <td> Number of home runs during his career  </td>\n",
    "        <td> 在其棒球职业生涯中全垒打的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> CRuns                              </td>\n",
    "        <td> Number of runs during his career  </td>\n",
    "        <td> 在其棒球职业生涯中得分的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> CRBI                              </td>\n",
    "        <td> Number of runs batted in during his career  </td>\n",
    "        <td> 在其棒球职业生涯中打者打点的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> CWalks                              </td>\n",
    "        <td> Number of walks during his career  </td>\n",
    "        <td> 在其棒球职业生涯中打击手被保送的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> PutOuts                              </td>\n",
    "        <td> Number of put outs in 1986  </td>\n",
    "        <td> 在1986年间接杀的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> Assists                             </td>\n",
    "        <td> Number of assists in 1986  </td>\n",
    "        <td> 在1986年间助杀的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> Errors                            </td>\n",
    "        <td> Number of errors in 1986          </td>\n",
    "        <td> 在1986年间失误的次数                 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> Salary                            </td>\n",
    "        <td> 1987 annual salary on opening day in thousands of dollars          </td>\n",
    "        <td> 在1987年开业时年薪（单位：千美元）         </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "533d68c1",
   "metadata": {},
   "source": [
    "## 任务\n",
    "\n",
    "本次实验中，我们需要解决以下问题：\n",
    "\n",
    "1. 如何利用主成分回归改进模型？\n",
    "2. 主成分回归估计是否比最小二乘估计更优？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb02475",
   "metadata": {},
   "source": [
    "## 解决方案\n",
    "\n",
    "在正式分析这个问题之前，我们介绍一下数据分析的基础步骤。\n",
    "第一步：数据采集（问卷、爬虫、系统采集等方式）与数据清洗；\n",
    "第二步：数据预处理（缺失、异常等，划分训练集和测试（验证）集）；\n",
    "第三步：数据的探索性分析（计算简单的统计量、绘制基础图表）；\n",
    "第四步：构建模型（针对所提出的问题，利用模型来解决）；\n",
    "第五步：模型诊断（考虑所提出是否符合数据）；\n",
    "第六步：得出结论（根据数据分析结果给出合理且科学的决策）。\n",
    "\n",
    "在本次实验中，我们简化了前三步骤。在数据预处理中，需要划分训练集和测试集，常见的方法有：留出法（Hold-out）和$K$折交叉验证法（K-fold Cross Validation）。我们这里介绍留出法，具体如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898033d0",
   "metadata": {},
   "source": [
    "在留出法中，需要划分两个数据集——训练集和测试集。\n",
    "\n",
    "第一，需要确定两个数据集的样本量。通常按一定比例，如8:2。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "49a574f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_train = 210 # 80% instances used for training\n",
    "n_test = n-n_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344b2d6c",
   "metadata": {},
   "source": [
    "第二，在所有样本中随机选取。这里我们固定随机种子，优势是：这样的划分是相对确定性的，有利于在编程过程中反复验证结果是否正确。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "228a9442",
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 1\n",
    "random.seed(seed)\n",
    "index = range(0,n)\n",
    "index_selected = random.sample(index,n_train)\n",
    "index_selected.sort()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b059f9d7",
   "metadata": {},
   "source": [
    "第三，我们根据所确定的索引将数据划分为两个数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3eea6df7",
   "metadata": {},
   "outputs": [],
   "source": [
    "Data_train = Data.loc[index_selected]\n",
    "Data_test = Data.drop(index = index_selected)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ccd6463",
   "metadata": {},
   "source": [
    "在数据预处理中，除了划分训练集和测试集之外，我们还需要根据模型重新构建数据类型。比如在回归模型中，将响应变量和特征拆分为两个部分，即"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d823b408",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Pre-processing\n",
    "X_train = Data_train.drop(columns = ['Salary'],axis = 1)\n",
    "Y_train = Data_train.Salary\n",
    "X_test = Data_test.drop(columns = [\"Salary\"],axis=1)\n",
    "Y_test = Data_test.Salary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a56045",
   "metadata": {},
   "source": [
    "在分析多重共线性时，我们需要将数据进行**标准化**。\n",
    "\n",
    "在数据$\\{(y_i,\\mathbf{x}_i):i=1,2,\\cdots,n\\}$，其中，$\\mathbf{x}_i = (x_{i1},x_{i2},\\cdots,x_{ip})'$，传统的标准化指的是\n",
    "$$\n",
    "x_{ij}^{\\ast} = \\frac{x_{ij}-\\bar{x}_j}{\\sqrt{\\frac{1}{n-1} \\sum_{i=1}^n (x_{ij} - \\bar{x}_j)^2}}\n",
    "$$\n",
    "注：这里分母中的方差通常用样本方差（即前面的系数为$(n-1)^{-1}$），也有使用总体方差（即前面的系数为$(n)^{-1}$）。在$n$很大时，分母上方差无明显差异。传统的标准化指的是：使得均值为0、方差为1。\n",
    "\n",
    "而在本实验中，我们所定义的**标准化**为\n",
    "$$\n",
    "x_{ij}^{\\ast} = \\frac{x_{ij}-\\bar{x}_j}{\\sqrt{\\sum_{i=1}^n (x_{ij} - \\bar{x}_j)^2}}\n",
    "$$\n",
    "这里与传统的标准化相差一个常数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b03499d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_standardized = preprocessing.scale(X_train, with_mean = True, with_std=True)/np.sqrt(n_train)\n",
    "Y_train_centered = preprocessing.scale(Y_train, with_mean = True, with_std=False)\n",
    "Y_train_mean = np.average(Y_train)\n",
    "X_test_standardized = preprocessing.scale(X_test, with_mean = True, with_std=True)/np.sqrt(n_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff73f33e",
   "metadata": {},
   "source": [
    "这里有一个问题值得注意：关于测试数据的标准化，对于测试数据而言，其标准化的方式仅仅只能利用可以观测到的数据，不能使用需要**未知**信息。在回归问题中，这样进行标准化是可行的，因为在测试集中特征是可观测到的，才能通过这些特征来进行预测，所以这种方法是可行的。如果是在时间序列等场景，一定需要注意，不要利用未来的信息做数据预处理，这样是不正确的。\n",
    "\n",
    "### Task 1: 构建主成分回归\n",
    "\n",
    "第一步，对数据进行主成分分析。\n",
    "\n",
    "问题：什么是主成分呢？\n",
    "\n",
    "以下我们先介绍主成分分析的基本想法。对于$p$维随机向量$\\mathbf{x} = (x_1,x_2,\\cdots,x_p)'$，假定\n",
    "- 其均值向量为$E(\\mathbf{x}) = \\mathbf{\\mu}$;\n",
    "- 其方差-协方差矩阵$Var(\\mathbf{x}) = \\Sigma$.\n",
    "我们所提取的主成分实际上是$p$维随机向量$\\mathbf{x}$的$p$个线性组合，即\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "z_1 = \\mathbf{a}_1' \\mathbf{x} = a_{11}x_1 + a_{21}x_2 + \\cdots+a_{p1}x_p\\\\\n",
    "z_2 = \\mathbf{a}_2' \\mathbf{x} = a_{12}x_1 + a_{22}x_2 + \\cdots+a_{p2}x_p\\\\\n",
    "\\vdots\\\\\n",
    "z_p = \\mathbf{a}_p' \\mathbf{x} = a_{1p}x_1 + a_{2p}x_2 + \\cdots+a_{pp}x_p\\\\\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "根据概率论的知识可知，$Var(z_i) = \\mathbf{a}_i' \\Sigma \\mathbf{a}_i$ 以及 $Cov(z_i,z_j) = \\mathbf{a}_i' \\Sigma \\math{a}_j$。\n",
    "\n",
    "如何确定$\\mathbf{a}_i,i=1,2,\\cdots,p$？它们需要满足\n",
    "- $\\mathbf{a}_i'\\mathbf{a}_i = 1$；\n",
    "- 当$i>1$时，$\\mathbf{a}_i'\\Sigma\\mathbf{a}_j = 0$；\n",
    "-$\\mathbf{a_i} = \\arg\\max_{\\mathbf{a}'\\mathbf{a} = 1, \\mathbf{a}'\\Sigma\\mathbf{a}_j,j=1,2,\\cdots,i-1} Var(\\mathbf{a}'\\mathbf{x})$；\n",
    "\n",
    "如何求解？对于$\\mathbf{a}_1 = (a_{11},a_{21},\\cdots,a_{p1})'$满足\n",
    "$$\n",
    "\\mathbf{a}_1 = \\arg\\max_{\\mathbf{a}} \\mathbf{a}'\\mathbf{x} \\quad \\text{s.t.} \\quad \\mathbf{a}_1'\\mathbf{a}_1 = 1.\n",
    "$$\n",
    "我们采用拉格朗日乘子法，令\n",
    "$$\n",
    "l(\\mathbf{a}_1) = Var(\\mathbf{a}_1'\\mathbf{x}) - \\lambda(\\mathbf{a}_1'\\mathbf{a}_1 - 1) = \\mathbf{a}_i'\\Sigma\\mathbf{a}_i - \\lambda (\\mathbf{a}_1'\\mathbf{a}_1-1)\n",
    "$$\n",
    "关于$l(\\mathbf{a}_1) $对$\\mathbf{a}_1$求导，即\n",
    "$$\n",
    "\\begin{cases}\n",
    "\\frac{\\partial l}{\\partial \\mathbf{a}_1} = 2(\\Sigma - \\lambda I) \\mathbf{a}_1 = 0\\\\\n",
    "\\frac{\\partial l}{\\partial \\lambda} = \\mathbf{a}_1'\\mathbf{a}_1 - 1 = 0\n",
    "\\end{cases}\n",
    "$$\n",
    "于是，$\\Sigma \\mathbf{a}_1 = \\lambda \\mathbf{a}_1$，由此可知，$\\lambda$是$\\Sigma$的特征值，而$\\mathbf{a}_1$是该特征值对应的特征向量。\n",
    "\n",
    "因此，我们对$Var(\\mathbf{x}) = \\Sigma$求特征值和特征向量，即\n",
    "- $\\Sigma$的特征值$\\lambda_1 \\geq \\lambda_2 \\geq \\cdots \\geq \\lambda_p \\geq 0$.\n",
    "- $\\mathbf{a}_1,\\mathbf{a}_2,\\cdots,\\mathbf{a}_p$为相应的单位正交特征向量。所以，$\\mathbf{x}$的第$i$个主成分为\n",
    "$$\n",
    "z_{i} = \\mathbf{a}_i'\\mathbf{x}, i=1,2,\\cdots,p.\n",
    "$$\n",
    "\n",
    "在实际数据中，如果某一个随机变量的方差过大，会导致主成分的权重会向方差大的随机变量倾斜，那么所构造的主成分是不合理的。于是，采用随机变量$\\mathbf{x}$的相关阵$Corr(\\mathbf{x})$来代替方差-协方差矩阵$\\Sigma$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "b01a32bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The percentages of total variance are [7.2985e+00 3.9851e+00 1.7352e+00 9.1210e-01 6.8070e-01 5.5120e-01\n",
      " 2.6490e-01 1.8050e-01 1.2800e-01 1.0050e-01 6.0000e-02 5.3300e-02\n",
      " 2.8200e-02 1.6100e-02 4.4000e-03 1.3000e-03]\n"
     ]
    }
   ],
   "source": [
    "model_pca = PCA(X_train_standardized,standardize = False, demean = True)\n",
    "model_pca_cr = model_pca.eigenvals # contribution rate of each component\n",
    "print(\"The percentages of total variance are\", np.around(model_pca_cr,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd740f12",
   "metadata": {},
   "source": [
    "以上计算的就是$\\mathbf{X}'\\mathbf{X}$的特征值，也被称为每个主成分的贡献率。通过以下代码可以进行验证。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ded19679",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The eigen values of X'X are  [7.2985e+00 3.9851e+00 1.7352e+00 9.1210e-01 6.8070e-01 5.5120e-01\n",
      " 2.6490e-01 1.8050e-01 1.2800e-01 1.0050e-01 6.0000e-02 5.3300e-02\n",
      " 2.8200e-02 1.6100e-02 4.4000e-03 1.3000e-03]\n"
     ]
    }
   ],
   "source": [
    "Corr_Mat = X_train_standardized.T @ X_train_standardized\n",
    "Lambda, V = np.linalg.eig(Corr_Mat)\n",
    "Lambda = sorted(Lambda,reverse = True)\n",
    "print(\"The eigen values of X'X are \", np.around(Lambda,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d371ea11",
   "metadata": {},
   "source": [
    "在主成分分析中，主成分个数也是一个超参数。如何选择主成分的个数也是一个重要的问题。\n",
    "\n",
    "在主成分分析中，我们可以绘制scree plot来作出主成分个数的主观判断。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fd2d3c83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaV0lEQVR4nO3de5RlZX3m8e/TIJLCCyolXqC7kUQIGkHsON5ivCQGL5Fxlo6aMgkzxo5r1BGjMWrPmsSZkGRpVGKc6KrRCBMqRCU4MUaNxHjBRI0FgiB4JX2BoJQmCKEzKvCbP/YuqC6ruk5ddp1dp76ftc46Z79nn71/u7qrn373fs+7U1VIktQ3W4ZdgCRJCzGgJEm9ZEBJknrJgJIk9ZIBJUnqJQNKktRLBpS0wSU5I8mnh12HtNYMKGmeJI9L8vdJvpvkn5P8XZKfHHJNv5XkB0n+NcmNbX2PXsF2PpHkV7qoUVprBpQ0R5J7AB8E/hC4N/BA4PXA95a5nUPXvjreU1V3A8aBTwMXJkkH+5F6wYCSDvRggKo6v6puq6p/q6qPVtUXZ1dI8qIkVye5OclVSU5t23cn+Y0kXwRuSXJokke1vZ0bk1ye5AlztnPPJO9Kcn2S65L8dpJDliqwqn4AnAvcD7jP/PeTPCbJ59se4OeTPKZtPwv4KeBtbU/sbav5QUldM6CkA30VuC3JuUmemuRec99M8hzgt4BfAu4BPBP4zpxVng88HTgSOBr4K+C3aXpjrwL+PMl4u+45wK3AjwIPB54CLHn6LcldgTOAfVX17Xnv3bvd51tpwuvNwF8luU9V7QIuBl5aVXerqpcu/eOQhseAkuaoqpuAxwEF/G9gJskHkhzdrvIrwBuq6vPV+HpV7ZmzibdW1b6q+jfgBcCHqupDVXV7VV0ETANPa7f3NODMqrqlqm4A3gI87yDl/cckNwL7gEcAz1pgnacDX6uqP6mqW6vqfODLwM+v7CciDU8X58mlDa2qrqbpoZDkROA84Gya3tGxwDcO8vF9c15vA56TZG443AX4ePveXYDr51xG2jLv8/O9t6pesET5DwD2zGvbQ3MtTdpQDCjpIKrqy0nOAX61bdoHHH+wj8x5vQ/4k6p60fyVktyfZuDFUVV16xqVC/BPNOE311bgIwvUJ/Wap/ikOZKcmOSVSY5pl4+l6Tl9tl3lncCrkjwijR9NMj8QZp0H/HySn0tySJLDkzwhyTFVdT3wUeBNSe6RZEuS45P89CoP4UPAg5P8QjtI47nASTQjEwG+BTxolfuQ1oUBJR3oZuDfAZ9LcgtNMF0JvBKgqt4HnAX8abvu/6UZAPFDqmofcDrwOmCGpkf169z5e/dLwGHAVcC/ABcA919N8VX1HeAZbb3fAV4NPGPOYIo/AJ6d5F+SvHU1+5K6Fm9YKEnqI3tQkqReMqAkSb1kQEmSesmAkiT1Uq++B3XUUUfV9u3bh12GJGkdXXLJJd+uqvH57b0KqO3btzM9PT3sMiRJ6yjJ/NlPgA1wim9qCrZvhy1bmuepqWFXJElaD73qQc03NQU7d8L+/c3ynj3NMsDExPDqkiR1r9c9qF277gynWfv3N+2SpNHW64Dau3d57ZKk0dHrgNq6dXntkqTR0euAOussGBs7sG1srGmXJI22XgfUxARMTsK2bZA0z5OTDpCQpM2g16P4oAkjA0mSNp/OelBJTkhy2ZzHTUnO7Gp/kqTR0lkPqqq+ApwCkOQQ4Drg/V3tT5I0WtbrGtSTgW9U1YLTWUiSNN96BdTzgPMXeiPJziTTSaZnZmbWqRxJUt91HlBJDgOeCbxvoferarKqdlTVjvHxH5rMVpK0Sa1HD+qpwKVV9a112JckaUSsR0A9n0VO70mStJhOAyrJEcDPAhd2uR9J0ujp9Iu6VXULcJ8u9yFJGk29nupIkrR5GVCSpF4yoCRJvWRASZJ6yYCSJPWSASVJ6iUDSpLUSwaUJKmXDChJUi8ZUJKkXjKgJEm9ZEBJknrJgJIk9ZIBJUnqJQNKktRLBpQkqZcMKElSLxlQkqReMqAkSb1kQEmSeqnTgEpyZJILknw5ydVJHt3l/iRJo+PQjrf/B8BHqurZSQ4DxjrenyRpRHQWUEnuCTweOAOgqr4PfL+r/UmSRkuXp/iOA2aAdyf5QpJ3Jjli/kpJdiaZTjI9MzPTYTmSpI2ky4A6FDgVeHtVPRy4BXjN/JWqarKqdlTVjvHx8Q7LkSRtJF0G1LXAtVX1uXb5AprAkiRpSZ0FVFV9E9iX5IS26cnAVV3tT5I0WroexfcyYKodwXcN8J863p8kaUR0GlBVdRmwo8t9SJJGkzNJSJJ6yYCSJPWSASVJ6iUDSpLUSwaUJKmXDChJUi8ZUJKkXjKgJEm9ZEBJknrJgJIk9ZIBJUnqJQNKktRLBpQkqZcMKElSLxlQkqReMqAkSb1kQEmSesmAkiT1kgElSeolA0qS1EuHdrnxJLuBm4HbgFurakeX+5MkjY5OA6r1xKr69jrsR5I0QjzFJ0nqpa4DqoCPJrkkyc6FVkiyM8l0kumZmZmOy5EkbRRdB9TjqupU4KnAS5I8fv4KVTVZVTuqasf4+HjH5UiSNopOA6qqrmufbwDeDzyyy/1JkkZHZwGV5Igkd599DTwFuLKr/UmSRkuXo/iOBt6fZHY/f1pVH+lwf5KkEdJZQFXVNcDJXW1fkjTaHGYuSeolA0qS1EsGlCSplwwoSVIvGVCSpF4yoCRJvWRASZJ6yYCSJPWSASVJ6qWBAirJ0UneleTD7fJJSV7YbWmSpM1s0B7UOcBfAw9ol78KnNlBPZIkAYMH1FFV9V7gdoCquhW4rbOqJEmb3qABdUuS+9DcIZckjwK+21lVkqRNb9DZzH8N+ABwfJK/A8aBZ3dWlSRp0xsooKrq0iQ/DZwABPhKVf2g08okSZvaQAGV5JfmNZ2ahKr6Px3UJEnSwKf4fnLO68OBJwOXAgaUJKkTg57ie9nc5SRHAn/WRUGSJMHKZ5K4BThuLQuRJGmuQa9B/SXtEHOaUDsJeG9XRUmSNGgP6veBN7WP3wUeX1WvGeSDSQ5J8oUkH1xhjZ2amoLt22HLluZ5amrYFUmSYPBrUJ9cxT5eDlwN3GMV2+jE1BTs3An79zfLe/Y0ywATE8OrS5K0RA8qyc1JblrgcXOSm5baeJJjgKcD71yrgtfSrl13htOs/fubdknScB20B1VVd1/l9s8GXg0sup0kO4GdAFu3bl3l7pZn797ltUuS1s+yRvEluW+SrbOPJdZ9BnBDVV1ysPWqarKqdlTVjvHx8eWUs2qL5eE656QkaQGD3g/qmUm+Bvwj8ElgN/DhJT72WOCZSXbTfGfqSUnOW3mpa++ss2Bs7MC2sbGmXZI0XIP2oP4n8Cjgq1V1HM1MEp892Aeq6rVVdUxVbQeeB/xtVb1gNcWutYkJmJyEbdsgaZ4nJx0gIUl9MOhURz+oqu8k2ZJkS1V9PMnZXRa2XiYmDCRJ6qNBA+rGJHcDPgVMJbmBZjaJgVTVJ4BPLLs6SdKmNegpvtOB/cArgI8A3wB+vquiJEkatAf1q8B7quo64NwO65EkCRi8B3V34KNJLk7y0iRHd1mUJEkDBVRVvb6qHgK8BLg/8Mkkf9NpZZKkTW25t9u4Afgm8B3gvmtfjiRJjUG/qPtfknwC+BhwH+BFVfWwLguTJG1ugw6SOBY4s6ou67AWSZLuMOjtNl7b3tfpAXM/U1VOqypJ6sSgd9R9KfBbwLeA29vmAjzNJ0nqxKCn+M4ETqiq73RYiyRJdxh0FN8+4LtdFiJJ0lyD9qCuAT6R5K+A7802VtWbO6lKkrTpDRpQe9vHYe1DkqRODTqK7/UAScaqan+3JUmSNPgXdR+d5Crgy+3yyUn+qNPKJEmb2qCDJM4Gfo5miiOq6nLg8R3VJEnS4HPxVdW+eU23rXEtkiTdYdBBEvuSPAaoJHcBXg5c3V1ZkqTNbtAe1ItpbrXxQOA64JR2WZKkTgw6iu/bwETHtUiSdIdB5+J76wLN3wWmq+ovFvnM4cCngLu2+7mgqn5zpYVKkjaXQU/xHU5zWu9r7eNhwDHAC5Ocvchnvgc8qapObj97WpJHraZYSdLmMeggiYcBj62q2wCSvB24GHgccMVCH6iqAv61XbxL+6hVVStJ2jQG7UHdC7jbnOUjgHu3gfW9hT8C7T2kLqO5VfxFVfW5BdbZmWQ6yfTMzMzglUuSRtqgAfUG4LIk705yDvAF4I1JjgD+ZrEPVdVtVXUKzenARyZ56ALrTFbVjqraMT4+vuwDkCSNpkFH8b0ryYeAR7ZNr6uqf2pf//oAn78xyceB04ArV1SpJGlTOWgPKsmJ7fOpwP1p7gu1D7hf23awz44nObJ9/SPAz9LO5SdJ0lKW6kG9EngR8KYF3ivgSQf57P2Bc5McQhOE762qD66oSknSpnPQgKqqF7XPT1zuhqvqi8DDV1iXJGmTW+oU36vnvH7OvPd+p6uiJElaahTf8+a8fu28905b41okSbrDUgGVRV4vtCxJ0ppZKqBqkdcLLUuStGaWGsV3cpKbaHpLP9K+pl0+vNPKJEmb2lKj+A5Zr0IkSZpr4Fu+S5K0ngwoSVIvGVCSpF4yoCRJvWRASZJ6yYCSJPWSASVJ6iUDSpLUSwaUJKmXDChJUi8ZUJKkXjKgJEm9ZEBJknrJgJIk9VJnAZXk2CQfT3JVki8leXlX+5IkjZ6lbli4GrcCr6yqS5PcHbgkyUVVdVWH+5QkjYjOelBVdX1VXdq+vhm4GnhgV/uTJI2WdbkGlWQ78HDgcwu8tzPJdJLpmZmZ9ShHkrQBdB5QSe4G/DlwZlXdNP/9qpqsqh1VtWN8fLzrciRJG0SnAZXkLjThNFVVF3a5L0nSaOlyFF+AdwFXV9Wbu9qPJGk0ddmDeizwi8CTklzWPp7W4f56a2oKtm+HLVua56mpYVckSf3X2TDzqvo0kK62v1FMTcHOnbB/f7O8Z0+zDDAxMby6JKnvnEmiY7t23RlOs/bvb9olSYszoDq2d+/y2iVJDQOqY1u3Lq9dktQwoDp21lkwNnZg29hY0y5JWpwB1bGJCZichG3bIGmeJycdICFJS+lysli1JiYMJElaLntQkqReMqAkSb1kQEmSesmAkiT1kgElSeolA0qS1EsGlCSplwwoSVIvGVCSpF4yoCRJvWRASZJ6yYCSJPWSASVJ6iUDSpLUS50FVJI/TnJDkiu72ockaXR12YM6Bzitw+1LkkZYZwFVVZ8C/rmr7etAU1OwfTts2dI8T00NuyJJWp2h31E3yU5gJ8DWrVuHXM3GNDUFO3fC/v3N8p49zTJ4J19JG9fQB0lU1WRV7aiqHePj48MuZ0PatevOcJq1f3/TLkkb1dADSqu3d+/y2iVpIzCgRsBiZ0Y9YyppI+tymPn5wGeAE5Jcm+SFXe1rszvrLBgbO7BtbKxpl6SNqrNBElX1/K62rQPNDoTYtas5rbd1axNODpCQtJF5im9ETEzA7t1w++3N81qHk8PYJa23oQ8zV/85jF3SMNiD0pIcxi5pGAwoLclh7JKGwYDSktZrGLvXuSTNZUBpSesxjH32OteePVB153UuQ0ravAwoLWliAiYnYds2SJrnycm1HSDhdS5J86Wqhl3DHXbs2FHT09PDLkNDsGVL03OaL2mGzksaXUkuqaod89vtQakXnK5J0nwGlHrB6ZokzWdAqRfW4zqXpI3FmSTUGxMTBpKkO9mD0qbh96ykjcUelDYF5xOUNh57UNoU1ut7VvbSpLVjD0qbwnrMJ2gvTVpb9qC0KazH96zWo5dmD02biQGlTWE9vmfVdS9tveYrNATVFwaUNoX1+J5V17209eqhdR2C6xGAhuyIqKrOHsBpwFeArwOvWWr9RzziESVtVOedVzU2VtX80948xsaa9rWQHLjt2UeyNtuvqtq2beF9bNu2Ntvv+me0XvuY3c+2bc3Pf9u2jbf99drHIIDpWihDFmpciwdwCPAN4EHAYcDlwEkH+4wBpY2uy1/4rsOjqvsQXI9jWI99dB2CoxLkg/4+DCOgHg389Zzl1wKvPdhnDChpcevxD0rX/7ivRy9wFHqaoxDky/n7ulhAdXkN6oHAvjnL17ZtklZgPa6jdT2YZD1GU67HProeELMeX4voeh9rcc106IMkkuxMMp1kemZmZtjlSL02MQG7dzf3yNq9e+2/X9V1CK7HaMr12EfXITgKQb4WAdhlQF0HHDtn+Zi27QBVNVlVO6pqx/j4eIflSBpElyG4Hr3AUehpjkKQr0kALnTeby0eNLNUXAMcx52DJB5ysM94DUrSRuEovqW3vdprUJ3e8j3J04CzaUb0/XFVHTSbveW7JI2OqanmmtPevU3P6ayzFu7JLnbL904DarmSzAB7Fnn7KODb61hOFzyGfvAY+mEUjgFG4ziGfQzbquqHrvH0KqAOJsn0Qgm7kXgM/eAx9MMoHAOMxnH09RiGPopPkqSFGFCSpF7aSAE1OewC1oDH0A8eQz+MwjHAaBxHL49hw1yDkiRtLhupByVJ2kQMKElSL/U+oJKcluQrSb6e5DXDrmclkhyb5ONJrkrypSQvH3ZNK5HkkCRfSPLBYdeyUkmOTHJBki8nuTrJo4dd03IleUX79+jKJOcnOXzYNS0lyR8nuSHJlXPa7p3koiRfa5/vNcwal7LIMbyx/bv0xSTvT3LkEEtc0kLHMOe9VyapJEcNo7aF9DqgkhwC/C/gqcBJwPOTnDTcqlbkVuCVVXUS8CjgJRv0OF4OXD3sIlbpD4CPVNWJwMlssONJ8kDgvwI7quqhNLO0PG+4VQ3kHJobmM71GuBjVfVjwMfa5T47hx8+houAh1bVw4Cv0txWqM/O4YePgSTHAk8B1nC+9NXrdUABjwS+XlXXVNX3gT8DTh9yTctWVddX1aXt65tp/lHcULceSXIM8HTgncOuZaWS3BN4PPAugKr6flXdONSiVuZQ4EeSHAqMAf805HqWVFWfAv55XvPpwLnt63OBf7+eNS3XQsdQVR+tqlvbxc/STIrdW4v8OQC8BXg10KtRc30PqJG7p1SS7cDDgc8NuZTlOpvmL/DtQ65jNY4DZoB3t6cq35nkiGEXtRxVdR3w+zT/070e+G5VfXS4Va3Y0VV1ffv6m8DRwyxmDfxn4MPDLmK5kpwOXFdVlw+7lvn6HlAjJcndgD8Hzqyqm4Zdz6CSPAO4oaouGXYtq3QocCrw9qp6OHAL/T+tdID2Os3pNGH7AOCIJC8YblWr185o3av/vS9Hkl00p/Knhl3LciQZA14H/Pdh17KQvgfUQPeU2giS3IUmnKaq6sJh17NMjwWemWQ3zWnWJyU5b7glrci1wLVVNdt7vYAmsDaSnwH+sapmquoHwIXAY4Zc00p9K8n9AdrnG4Zcz4okOQN4BjBRG++LpcfT/Gfn8vb3+xjg0iT3G2pVrb4H1OeBH0tyXJLDaC4Gf2DINS1bktBc97i6qt487HqWq6peW1XHVNV2mj+Dv62qDfe/9qr6JrAvyQlt05OBq4ZY0krsBR6VZKz9e/VkNthAjzk+APxy+/qXgb8YYi0rkuQ0mlPfz6yq/Uut3zdVdUVV3beqtre/39cCp7a/K0PX64BqLz6+FPhrml/C91bVl4Zb1Yo8FvhFmp7HZe3jacMuapN6GTCV5IvAKcDvDLec5Wl7fxcAlwJX0PwO93KamrmSnA98BjghybVJXgj8HvCzSb5G0zP8vWHWuJRFjuFtwN2Bi9rf63cMtcglLHIMveVUR5KkXup1D0qStHkZUJKkXjKgJEm9ZEBJknrJgJIk9ZIBpQ0vyf2S/FmSbyS5JMmHkjx42HWtRpInJFnwC7hJzkhye5KHzWm7sp1Gay32/a9rsR1ptQwobWjtl1XfD3yiqo6vqkfQzCi90ed1ewIHnyHiWmDX+pQyuHYCW2lNGFDa6J4I/KCq7viCZFVdXlUXp/HGtndxRZLnwh29k08m+Ysk1yT5vSQTSf6hXe/4dr1zkrwjyXSSr7ZzEpLk8CTvbtf9QpIntu1nJLkwyUfaexy9YbamJE9J8pkklyZ5XzsvI0l2J3l9235FkhPbntCLgVe0X/78qQWO+4PAQ+bMinGHuT2gJM9Ocs6c43l7ks+2x/2ENPcHunp2nTmfe0uae059LMl423Z8e2yXJLk4yYnzfk6fA96AtEYMKG10DwUWm8T2P9DMFnEyzUwFb5yd+61tezHw4zSzfDy4qh5JczuRl83Zxnaa2748HXhHmpsDvoRmftOfAJ4PnJs7bxp4CvBc4CeA56a5WeVRwH8DfqaqTgWmgV+bs49vt+1vB15VVbuBdwBvqapTquriBY7tdpoweN1Bfzo/7F7Ao4FX0Ew19BbgIcBPJDmlXecIYLqqHgJ8EvjNtn0SeFnbS30V8EdztnsM8Jiqmntc0qrYHdcoexxwflXdRjMx6SeBnwRuAj4/e6uHJN8AZm9ZcQVNr2zWe6vqduBrSa4BTmy3+4cAVfXlJHuA2WteH6uq77bbvQrYBhxJc8PNv2vOSHIYzXQzs2YnD76EJlQH9afAriTHLeMzf1lVleQK4FtVdUVb65dowvgymvB7T7v+ecCFbY/vMcD72mMAuOuc7b6v/TlLa8aA0kb3JeDZK/jc9+a8vn3O8u0c+Hsxfy6wpeYGm7vd29ptBbioqp6/xGdm1x9IVd2a5E3Abxykxvm3g597nPN/Bovtu2jOttxYVacsss4tSxYsLZOn+LTR/S1w1yQ7ZxuSPKy9bnMxzWm2Q9rrKI8H/mGZ239Oki3tdakHAV9ptzvR7uvBwNa2fTGfBR6b5EfbzxwxwCjDm2kmIV3KOTSnL8fntH0ryY8n2QI8a4BtzLeFO0P/F4BPt/cv+8ckz4FmcEqSk1ewbWlgBpQ2tPb+O88CfqYdZv4l4Hdp7tD6fuCLwOU0QfbqFdxGYC9NqH0YeHFV/T+aay9b2tNk7wHOqKrvLbaBqpoBzgDOTzOL+mdoThUezF8CzzrIIInZbX8feCtw3znNr6EZRPH3NHfdXa5bgEcmuRJ4EvA/2vYJ4IVJLqfpuZ6+gm1LA3M2c2kR7ci2D1bVBcOuRdqM7EFJknrJHpQkqZfsQUmSesmAkiT1kgElSeolA0qS1EsGlCSpl/4/0Gmha41eoSYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = model_pca.plot_scree(log_scale = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9ee91cc",
   "metadata": {},
   "source": [
    "除此之外，我们还可以计算每增加一个主成分后，对信息**增量**是多少？我们可以计算累积贡献率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d827bab6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cummulative percentages of total variance are [0.4562 0.7052 0.8137 0.8707 0.9132 0.9477 0.9642 0.9755 0.9835 0.9898\n",
      " 0.9935 0.9969 0.9986 0.9996 0.9999 1.    ]\n"
     ]
    }
   ],
   "source": [
    "model_pca_ccr = np.cumsum(model_pca_cr)/p # cummulative contribution rate\n",
    "print(\"The cummulative percentages of total variance are\", np.around(model_pca_ccr,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac0c17b",
   "metadata": {},
   "source": [
    "我们选取主成分时需要其累积贡献率达到90%，即"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "bc93846d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_cv = 0.90"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "4d72f6e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The appropriate number of component is 5\n"
     ]
    }
   ],
   "source": [
    "num_pca = np.min(np.where(model_pca_ccr > pca_cv))\n",
    "print(\"The appropriate number of component is\", (num_pca+1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "316f0a2f",
   "metadata": {},
   "source": [
    "第二步，我们利用所提取的主成分构建线性回归模型。具体来说，$\\mathbf{X}$是已标准化的设计矩阵。假定$\\mathbf{X}'\n",
    "\\mathbf{X}$的特征值$\\lambda_1 \\geq \\lambda_2 \\geq \\cdots \\geq \\lambda_p$和其相应单位正交化后的特征向量为$\\mathbf{v}_1,\\mathbf{v}_2,\\cdots,\\mathbf{v}_p$.令\n",
    "$$\n",
    "\\Lambda = \\text{diag}\\{\\lambda_1,\\lambda_2,\\cdots,\\lambda_p\\},\\quad \n",
    "V' = (\\mathbf{v}_1, \\mathbf{v}_2,\\cdots,\\mathbf{v}_p)\n",
    "$$\n",
    "于是，我们所构建的主成分为\n",
    "$$\n",
    "Z = X V'\n",
    "$$\n",
    "在线性回归模型中\n",
    "$$\n",
    "\\mathbf{y} = X\\mathbf{\\beta} + \\mathbf{\\epsilon} = X V'V \\mathbf{\\beta} + \\mathbf{\\epsilon} = Z \\mathbf{\\alpha} + \\mathbf{\\epsilon} \n",
    "$$\n",
    "我们选取前$k$个主成分作为特征构建线性回归模型，即\n",
    "$$\n",
    "\\mathbf{y} = Z_1 \\mathbf{\\alpha}_1 + \\mathbf{\\epsilon}\n",
    "$$\n",
    "其中，$Z = (\\mathbf{z}_1,\\mathbf{z}_2,\\cdots,\\mathbf{z}_{k},\\mathbf{z}_{k+1},\\cdots,\\mathbf{z}_p) = (Z_1,Z_2)$而\n",
    "$\\mathbf{\\alpha} = (\\alpha_1,\\alpha_2,\\cdots,\\alpha_k,\\alpha_{k+1},\\cdots,\\alpha_p)' = (\\mathbf{\\alpha}_1',\\mathbf{\\alpha}_2')'$.\n",
    "\n",
    "我们用最小二乘估计来估计$\\mathbf{\\alpha}_1$即\n",
    "$$\n",
    "\\hat{\\mathbf{\\alpha}_1} = (Z_{1}'Z_{1})^{-1}Z_{1}'\\mathbf{y}.\n",
    "$$\n",
    "而用零来估计$\\mathbf{\\alpha}_2 = \\mathbf{0}$。所以，$\\hat{\\mathbf{\\alpha}} = (\\hat{\\mathbf{\\alpha}_1}',\\mathbf{0}')$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "0de82996",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_pca_selected = PCA(X_train_standardized,  standardize = False,  demean = True)\n",
    "V = model_pca_selected.loadings\n",
    "Z_train_standardized= X_train_standardized @ V[:,0:(num_pca+1)]\n",
    "model_pcr = sm.OLS(Y_train_centered, Z_train_standardized).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adfbd76",
   "metadata": {},
   "source": [
    "第三，在通过线性变换，来估计主成分回归中的参数。具体来说，回归系数$\\mathbf{\\beta}$的主成分估计为\n",
    "$$\n",
    "\\hat{\\mathbf{\\beta}}_{pcr} = V' \\hat{\\mathbf{\\alpha}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "1e2ee467",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The regression coefficients in PCR are [   29.3117  -132.0506  -249.3543  -106.9957  -207.9692  -376.854\n",
      "    85.6962  -316.6594  -218.0032   122.9863  1130.1306 -1116.7174\n",
      "   621.7523  -241.2335  -233.2593    42.4857]\n"
     ]
    }
   ],
   "source": [
    "model_pcr_coef = V.T @ np.pad(model_pcr.params,(0,p-(num_pca+1)))\n",
    "print(\"The regression coefficients in PCR are\", np.around(model_pcr_coef,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25840479",
   "metadata": {},
   "source": [
    "### Task 2: 比较主成分回归估计与最小二乘估计\n",
    "\n",
    "这里我们采用测试集的RMSE来比较两种估计方法的优劣。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "547d8028",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The RMSE in the ordinary regression is 591.1203\n",
      "The RMSE in the principal component regression is 566.4681\n"
     ]
    }
   ],
   "source": [
    "model_ols = sm.OLS(Y_train_centered,X_train_standardized).fit()\n",
    "model_ols_coef = model_ols.params \n",
    "Y_pred_ols = Y_train_mean + X_test_standardized @ model_ols_coef\n",
    "ols_rmse = np.sqrt(np.mean((Y_pred_ols-Y_test)**2))\n",
    "\n",
    "Y_pred_pcr = Y_train_mean + X_test_standardized @ model_pcr_coef\n",
    "pcr_rmse = np.sqrt(np.mean((Y_pred_pcr-Y_test)**2))\n",
    "\n",
    "print(\"The RMSE in the ordinary regression is\", round(ols_rmse,4))\n",
    "print(\"The RMSE in the principal component regression is\",round(pcr_rmse,4))\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}