project.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0STxdZQOsgj"
      },
      "source": [
        "# プロジェクト"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CyMjylZAOsgn"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "\n",
        "from collections import Counter\n",
        "\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.preprocessing import PolynomialFeatures\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.pipeline import make_pipeline\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.metrics import confusion_matrix,  accuracy_score, precision_score, recall_score, f1_score\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2wyXvIAOsgo"
      },
      "source": [
        "あなたは酒造会社のデータ分析部門の責任者です。ワインの理化学検査結果からワインの品質を予測するシステムの検討を行ってほしいとの依頼をCTO（最高技術責任者）から受けました。\n",
        "\n",
        "部門の新人で最近データ分析を勉強した部下が次のような分析を行い、実データ（UCI Machine Learning Repositoryに公開されているワインの品質データセットを使います）を用いて、ワインの理化学検査結果に基づく特徴量からワインの品質予測を以下のように試行しました。\n",
        "\n",
        "***白ワイン***のデータセットを使用する。（赤ワインのデータセットではないことに注意してください）\n",
        "\n",
        "[Wine Quality Data Set ](https://archive.ics.uci.edu/ml/datasets/wine+quality)\n",
        "\n",
        "各ワインのデータは以下の11種類の特徴量からなる。\n",
        "- 1 - fixed acidity \n",
        "- 2 - volatile acidity \n",
        "- 3 - citric acid \n",
        "- 4 - residual sugar \n",
        "- 5 - chlorides \n",
        "- 6 - free sulfur dioxide \n",
        "- 7 - total sulfur dioxide \n",
        "- 8 - density \n",
        "- 9 - pH \n",
        "- 10 - sulphates \n",
        "- 11 - alcohol \n",
        "\n",
        "各ワインには0から10の品質スコアが付与されており、11種類の特徴量を元にワインの品質を高品質（ワインのqualityが6以上）、低品質（ワインのqualityが6未満）の2値にして予測する2クラス分類問題を考える。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1MpidDieOsgp",
        "outputId": "2bf36137-5b0d-4d1d-d778-9dc546853d9f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-01-16 10:26:58--  https://raw.githubusercontent.com/UTDataMining/2021S/master/project/winequality-white.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 264426 (258K) [text/plain]\n",
            "Saving to: ‘winequality-white.csv.1’\n",
            "\n",
            "\rwinequality-white.c   0%[                    ]       0  --.-KB/s               \rwinequality-white.c 100%[===================>] 258.23K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2022-01-16 10:26:58 (11.5 MB/s) - ‘winequality-white.csv.1’ saved [264426/264426]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Colaboratoryでは以下を実行して必要なファイルをダウンロード\n",
        "!wget https://raw.githubusercontent.com/UTDataMining/2021S/master/project/winequality-white.csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PWXf36u1Osgq"
      },
      "outputs": [],
      "source": [
        "# 白ワインデータセットの読み込み\n",
        "wine = pd.read_csv(\"winequality-white.csv\", sep=\";\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TDOtW5ThOsgq",
        "outputId": "d6a3e38b-87dd-47cb-c4ef-a61572fa2131",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3258\n",
            "1640\n"
          ]
        }
      ],
      "source": [
        "# 特徴量\n",
        "X=wine[ ['fixed acidity','volatile acidity','citric acid','residual sugar','chlorides','free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates','alcohol']].values\n",
        "\n",
        "# 正解ラベルを1（高品質：ワインのqualityが6以上）か0（低品質：ワインのqualityが6未満）とする\n",
        "y=wine['quality'].values\n",
        "y = (y>=6).astype(np.int) \n",
        "print(np.sum(y==1, axis=0)) # 高品質ワインデータの数\n",
        "print(np.sum(y==0, axis=0)) # 低品質ワインデータの数"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tokwSCu_Osgr",
        "outputId": "dff81aea-9d81-440c-a377-13aea2c0d852",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[158 170]\n",
            " [ 75 577]]\n",
            "0.75\n",
            "0.7724230254350736\n",
            "0.8849693251533742\n",
            "0.8248749106504645\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-772231d8-e291-4b54-b433-fad71ca9c550\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fixed acidity</th>\n",
              "      <th>volatile acidity</th>\n",
              "      <th>citric acid</th>\n",
              "      <th>residual sugar</th>\n",
              "      <th>chlorides</th>\n",
              "      <th>free sulfur dioxide</th>\n",
              "      <th>total sulfur dioxide</th>\n",
              "      <th>density</th>\n",
              "      <th>pH</th>\n",
              "      <th>sulphates</th>\n",
              "      <th>alcohol</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>weight</th>\n",
              "      <td>-0.195072</td>\n",
              "      <td>-5.553155</td>\n",
              "      <td>-0.159136</td>\n",
              "      <td>0.057788</td>\n",
              "      <td>-0.441644</td>\n",
              "      <td>0.008894</td>\n",
              "      <td>-0.001888</td>\n",
              "      <td>-2.718552</td>\n",
              "      <td>-0.581467</td>\n",
              "      <td>0.957013</td>\n",
              "      <td>0.97454</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-772231d8-e291-4b54-b433-fad71ca9c550')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-772231d8-e291-4b54-b433-fad71ca9c550 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-772231d8-e291-4b54-b433-fad71ca9c550');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "        fixed acidity  volatile acidity  ...  sulphates  alcohol\n",
              "weight      -0.195072         -5.553155  ...   0.957013  0.97454\n",
              "\n",
              "[1 rows x 11 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ],
      "source": [
        "# 訓練データとテストデータに分割\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y) \n",
        "\n",
        "# ロジスティック回帰モデル\n",
        "lr=LogisticRegression(solver='liblinear',  multi_class='auto') \n",
        "lr.fit(X_train, y_train) # 訓練データで学習\n",
        "y_pred = lr.predict(X_test) # テストデータで予測\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred)) # 混同行列\n",
        "print(accuracy_score(y_test, y_pred)) # accuracy\n",
        "print(precision_score(y_test, y_pred)) # precision\n",
        "print(recall_score(y_test, y_pred)) # recall\n",
        "print(f1_score(y_test, y_pred))  # f値\n",
        "\n",
        "# 特徴量ごとのパラメータ重み\n",
        "pd.DataFrame(lr.coef_, index=['weight'], columns=wine.drop(\"quality\", axis=1).columns.values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pja765NYOsgs"
      },
      "source": [
        "部下は、この分析を元にワインの品質予測の（2クラス分類としての）accuracyは75%となりそうです、と報告してきました。駆け出しのデータサイエンティストである部下の上司であり指導的立場にあるあなたはこの分析は適切と考えますか？\n",
        "\n",
        "**上記のワインの理化学検査結果からの品質予測について、あなた自身で（部下に対して模範となるような）分析を行い、その分析の過程と結果についてレポートとしてまとめたものを作成して提出してください。**\n",
        "\n",
        "なお、レポートは定められた様式に従って図表を含めて最大3ページまでとし、少なくとも1ページ以上は本文で構成してください（図表だけを貼り付けただけで説明がないものは不可）。\n",
        "\n",
        "レポートと合わせて分析に用いたコードも提出してください。\n",
        "\n",
        "授業で未習の内容を取り入れてもらっても構いませんが、それを持って大きく加点することはありません。むしろ、授業で学んだ内容を元にまず、適切に分析を行っているか、その上で独自の創意工夫があるか、を主に評価します。"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 分析に用いたコード\n",
        "まず、特徴量選択のため主成分分析による次元削減を行う。"
      ],
      "metadata": {
        "id": "kg8ri-Ve_cFt"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iPXGRQ_9Osgs"
      },
      "outputs": [],
      "source": [
        "# 累積寄与率が90%以上となるように、特徴量をi次元に縮約する関数\n",
        "def reduce_dimension(X):\n",
        "  for i in range(1, X.shape[1]):\n",
        "    pca = PCA(n_components=i, random_state=0)\n",
        "    pca.fit(X)\n",
        "    if np.sum(pca.explained_variance_ratio_) >= 0.9:\n",
        "      X = pca.transform(X)\n",
        "      return X\n",
        "\n",
        "# 標準化\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(X)\n",
        "norm_X = scaler.transform(X)\n",
        "\n",
        "# 主成分分析による次元削減（8次元に縮約）\n",
        "norm_X = reduce_dimension(norm_X)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "PCA で8次元に削減した特徴量を用いて、ロジスティック回帰モデルによる予測を行う。"
      ],
      "metadata": {
        "id": "3m7NLf5oEG8h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 訓練データとテストデータに分割\n",
        "norm_X_train, norm_X_test, y_train, y_test = train_test_split(norm_X, y, test_size=0.2, random_state=0, stratify=y) \n",
        "\n",
        "lr = LogisticRegression(solver='liblinear', multi_class='auto') \n",
        "lr.fit(norm_X_train, y_train)\n",
        "y_pred = lr.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mauQzfZyEWAm",
        "outputId": "67ec3b4f-5fb8-4b98-dd1a-76d83df8bd4a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[126 202]\n",
            " [ 71 581]]\n",
            "0.7214285714285714\n",
            "0.7420178799489144\n",
            "0.8911042944785276\n",
            "0.8097560975609756\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "recall 以外の評価指標が落ちてしまったので、次元数は削減しないで分析を続ける。\n",
        "\n",
        "外れ値を除外してからモデルの学習を行うことを考える。"
      ],
      "metadata": {
        "id": "FTNeIUngFgbJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wine = pd.read_csv(\"winequality-white.csv\", sep=\";\")\n",
        "\n",
        "\n",
        "# 四分位範囲を基準として、外れ値が2つ以上ある行を削除する関数\n",
        "def remove_outliers(df):\n",
        "  outlier_indices = []\n",
        "\n",
        "  for column_name in df.columns.values:\n",
        "    if(column_name == 'quality'):\n",
        "      continue\n",
        "    \n",
        "    q1 = df[column_name].describe()['25%']\n",
        "    q3 = df[column_name].describe()['75%']\n",
        "    iqr = q3 - q1\n",
        "\n",
        "    min = q1 - (iqr * 1.5)\n",
        "    max = q3 + (iqr * 1.5)\n",
        "\n",
        "    outlier_indices.extend(\n",
        "        df[(df[column_name] < min) | (df[column_name] > max)].index\n",
        "    )\n",
        "\n",
        "  outlier_indices = Counter(outlier_indices)\n",
        "  rows_to_be_removed = list(k for k, v in outlier_indices.items() if v >= 2)\n",
        "\n",
        "  return df.drop(rows_to_be_removed)\n",
        "\n",
        "\n",
        "# 白ワインデータセットから外れ値を含む行を除外\n",
        "wine = remove_outliers(wine)\n",
        "\n",
        "X = wine[['fixed acidity','volatile acidity','citric acid','residual sugar','chlorides','free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates','alcohol']].values\n",
        "y = wine['quality'].values\n",
        "y = (y >= 6).astype(np.int)\n",
        "print(np.sum(y==1, axis=0))\n",
        "print(np.sum(y==0, axis=0))\n",
        "\n",
        "# 訓練データとテストデータに分割\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y) \n",
        "\n",
        "# 標準化\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(X_train)\n",
        "norm_X_train = scaler.transform(X_train)\n",
        "norm_X_test = scaler.transform(X_test)\n",
        "\n",
        "lr = LogisticRegression(solver='liblinear', multi_class='auto') \n",
        "lr.fit(norm_X_train, y_train)\n",
        "y_pred = lr.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eUESIS6WGAyt",
        "outputId": "d7378047-f884-4e1d-ac01-b606dae0ab97"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3194\n",
            "1545\n",
            "[[147 162]\n",
            " [ 74 565]]\n",
            "0.7510548523206751\n",
            "0.7771664374140302\n",
            "0.8841940532081377\n",
            "0.8272327964860907\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "recall 以外の評価指標がわずかに向上したため、外れ値は削除して分析を続ける。\n",
        "次にモデルの正則化を考える。5分割交差検証によりaccuracy を評価指標として最適な正則化項の係数を決定する。"
      ],
      "metadata": {
        "id": "1IjWXDHbLTF8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def model_selection1(X_train, y_train):\n",
        "  pipe = make_pipeline(StandardScaler(), LogisticRegression(solver='liblinear', multi_class='auto'))\n",
        "  param_grid = {'logisticregression__C': [10, 100, 1000, 10000, 100000]}\n",
        "  grid = GridSearchCV(pipe, param_grid=param_grid, scoring=\"accuracy\", cv=5)\n",
        "  grid.fit(X_train, y_train)\n",
        "  return grid.best_params_['logisticregression__C'], grid.best_score_\n",
        "\n",
        "model_selection1(X_train, y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cyJQoEq4nh6D",
        "outputId": "7b48c352-7616-45ab-f5d1-b861cd71383e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1000, 0.7562662300416114)"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "交差検証の結果、最適な正則化項の係数は0.001、そのときのaccuracy は0.75...となることがわかった。続いて、多項式を用いた特徴量生成によりモデルを複雑化することを考える。5分割交差検証により、accuracy を評価指標として最適な正則化項の係数と多項式の次数の組み合わせを決定する（かなり時間がかかります）。"
      ],
      "metadata": {
        "id": "CK8f5Nv0sRNl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def model_selection2(X_train, y_train):\n",
        "    pipe = make_pipeline(PolynomialFeatures(), StandardScaler(), LogisticRegression(solver='liblinear', multi_class='auto', max_iter=10000))\n",
        "    param_grid = {'logisticregression__C': [0.01, 0.1, 1], 'polynomialfeatures__degree': [3, 4, 5]}\n",
        "    grid = GridSearchCV(pipe, param_grid=param_grid, scoring=\"accuracy\", cv=5)\n",
        "    grid.fit(X_train, y_train)\n",
        "    return grid.best_params_['logisticregression__C'], grid.best_params_['polynomialfeatures__degree'], grid.best_score_\n",
        "\n",
        "optimal_C, optimal_degree, valid_accuracy = model_selection2(X_train, y_train)\n",
        "print(optimal_C, optimal_degree, valid_accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qxKYfiB7uMYb",
        "outputId": "3b2cf921-3305-4471-8522-13b53029a6d3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.1 4 0.7831722061732386\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "最適な正則化項の係数と多項式の次数の組み合わせは10と4であり、そのときのaccuracy は0.7831722061732386となることが分かった。この正則化項の係数と多項式の次数によりロジスティック回帰モデルを学習し、テストデータで評価を行う。"
      ],
      "metadata": {
        "id": "JTZpjmf07Z_u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipe = make_pipeline(PolynomialFeatures(degree=optimal_degree), StandardScaler(), LogisticRegression(solver='liblinear',  multi_class='auto', C=optimal_C))\n",
        "pipe.fit(X_train,y_train)\n",
        "y_pred = pipe.predict(X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred)) # 混同行列\n",
        "print(accuracy_score(y_test, y_pred)) # accuracy\n",
        "print(precision_score(y_test, y_pred)) # precision\n",
        "print(recall_score(y_test, y_pred)) # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uuxeWmGT79Wh",
        "outputId": "47b773ad-ebfd-4c09-9a58-cff8ae35a0c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[170 139]\n",
            " [ 88 551]]\n",
            "0.760548523206751\n",
            "0.7985507246376812\n",
            "0.8622848200312989\n",
            "0.8291948833709555\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "当初の分析結果に比べ、recall 以外の評価指標については（わずかではあるが）予測精度を向上することができた。しかしながら、依然高精度の予測ができているとは言えないため、別の機械学習モデルを使用することを考える。まずはSVM を使用する。"
      ],
      "metadata": {
        "id": "jXC9nvPN-LzA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "svc = SVC(kernel='rbf', random_state=0)\n",
        "svc.fit(norm_X_train, y_train)\n",
        "y_pred = svc.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uNxuQDoUEGqD",
        "outputId": "7dc488cd-f0a9-43ba-bde4-57a1fc3d8d9c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[174 135]\n",
            " [ 80 559]]\n",
            "0.7732067510548524\n",
            "0.8054755043227666\n",
            "0.8748043818466353\n",
            "0.8387096774193549\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "決定木による予測も行う。"
      ],
      "metadata": {
        "id": "3siQCvm0N1Jk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dtc = DecisionTreeClassifier(random_state=0)\n",
        "dtc = dtc.fit(norm_X_train, y_train)\n",
        "y_pred = dtc.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NCKvHntVOx0g",
        "outputId": "f43ec735-5ae9-4c70-d6e1-88666d3516f4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[202 107]\n",
            " [ 91 548]]\n",
            "0.7911392405063291\n",
            "0.8366412213740458\n",
            "0.8575899843505478\n",
            "0.8469860896445133\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "accuracy に関しては、決定木による予測が一番精度が高いことが分かった。\n",
        "よって、以降では決定木の予測精度を上げることを試みる。複数の決定木を使ったアンサンブル学習（ランダムフォレスト）により、訓練データを学習する。"
      ],
      "metadata": {
        "id": "MYKjDmtXPRR3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rfc = RandomForestClassifier(random_state=0)\n",
        "rfc.fit(norm_X_train, y_train)\n",
        "y_pred = rfc.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eB3ZqJIkQriy",
        "outputId": "719a19b8-ff5d-4012-a912-7bf38a7255b4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[217  92]\n",
            " [ 72 567]]\n",
            "0.8270042194092827\n",
            "0.8603945371775418\n",
            "0.8873239436619719\n",
            "0.8736517719568568\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "単一の決定木で学習したときよりaccuracy が向上した。ロジスティック回帰モデルのときと同様に、主成分分析による次元削減を行った上でランダムフォレストを使用して学習を行う。"
      ],
      "metadata": {
        "id": "AGB253gzUqio"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wine = pd.read_csv(\"winequality-white.csv\", sep=\";\")\n",
        "\n",
        "X = wine[['fixed acidity','volatile acidity','citric acid','residual sugar','chlorides','free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates','alcohol']].values\n",
        "y = wine['quality'].values\n",
        "y = (y >= 6).astype(np.int)\n",
        "print(np.sum(y==1, axis=0))\n",
        "print(np.sum(y==0, axis=0))\n",
        "\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(X)\n",
        "norm_X = scaler.transform(X)\n",
        "\n",
        "# 主成分分析による次元削減（8次元に縮約）\n",
        "norm_X = reduce_dimension(norm_X)\n",
        "\n",
        "# 訓練データとテストデータに分割\n",
        "norm_X_train, norm_X_test, y_train, y_test = train_test_split(norm_X, y, test_size=0.2, random_state=0, stratify=y) \n",
        "\n",
        "rfc = RandomForestClassifier(random_state=0)\n",
        "rfc.fit(norm_X_train, y_train)\n",
        "y_pred = rfc.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ADctQpMiTFTa",
        "outputId": "5a17f3db-0984-4e9b-b3cd-68c70d1cdd33"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3258\n",
            "1640\n",
            "[[216 112]\n",
            " [ 52 600]]\n",
            "0.8326530612244898\n",
            "0.8426966292134831\n",
            "0.9202453987730062\n",
            "0.8797653958944281\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "PCA による次元削減によって、accuracy がさらに向上した。続いて、外れ値を除外してからランダムフォレストによる学習を行う。"
      ],
      "metadata": {
        "id": "iJVdKY-xWOLS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 白ワインデータセットから外れ値を含む行を除外\n",
        "wine = remove_outliers(wine)\n",
        "\n",
        "X = wine[['fixed acidity','volatile acidity','citric acid','residual sugar','chlorides','free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates','alcohol']].values\n",
        "y = wine['quality'].values\n",
        "y = (y >= 6).astype(np.int)\n",
        "print(np.sum(y==1, axis=0))\n",
        "print(np.sum(y==0, axis=0))\n",
        "\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(X)\n",
        "norm_X = scaler.transform(X)\n",
        "\n",
        "# 主成分分析による次元削減（8次元に縮約）\n",
        "norm_X = reduce_dimension(norm_X)\n",
        "\n",
        "# 訓練データとテストデータに分割\n",
        "norm_X_train, norm_X_test, y_train, y_test = train_test_split(norm_X, y, test_size=0.2, random_state=0, stratify=y) \n",
        "\n",
        "rfc = RandomForestClassifier(random_state=0)\n",
        "rfc.fit(norm_X_train, y_train)\n",
        "y_pred = rfc.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tddyZ87YW-CW",
        "outputId": "4e38b7f8-de6c-4811-9fd4-c1c1b2178f02"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3187\n",
            "1538\n",
            "[[199 109]\n",
            " [ 65 572]]\n",
            "0.8158730158730159\n",
            "0.8399412628487518\n",
            "0.8979591836734694\n",
            "0.8679817905918057\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "すべての評価指標が落ちてしまったので、外れ値は削除しない方針で分析を続ける。3分割交差検証によりaccuracy を評価指標として最適な決定木の数（n_estimators）と決定木の深さ（max_depth）の組み合わせを決定する。"
      ],
      "metadata": {
        "id": "ogU26XfmgS2Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wine = pd.read_csv(\"winequality-white.csv\", sep=\";\")\n",
        "\n",
        "X = wine[['fixed acidity','volatile acidity','citric acid','residual sugar','chlorides','free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates','alcohol']].values\n",
        "y = wine['quality'].values\n",
        "y = (y >= 6).astype(np.int)\n",
        "print(np.sum(y==1, axis=0))\n",
        "print(np.sum(y==0, axis=0))\n",
        "\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(X)\n",
        "norm_X = scaler.transform(X)\n",
        "\n",
        "# 主成分分析による次元削減（8次元に縮約）\n",
        "norm_X = reduce_dimension(norm_X)\n",
        "\n",
        "# 訓練データとテストデータに分割\n",
        "norm_X_train, norm_X_test, y_train, y_test = train_test_split(norm_X, y, test_size=0.2, random_state=0, stratify=y) \n",
        "\n",
        "def model_selection3(X_train, y_train):\n",
        "    param_grid = {\"max_features\": [2, 3, 4, 5], \"n_estimators\":[500, 1000, 1500, 2000]}\n",
        "    grid = GridSearchCV(estimator=RandomForestClassifier(random_state=0), param_grid=param_grid, scoring=\"accuracy\", cv=3)\n",
        "    grid.fit(X_train, y_train)\n",
        "    return grid.best_params_['max_features'], grid.best_params_['n_estimators'], grid.best_score_, grid.cv_results_\n",
        "\n",
        "optimal_feature, optimal_estimator, valid_accuracy, cv_result = model_selection3(norm_X_train, y_train)\n",
        "\n",
        "# GridSearchCV 結果の可視化\n",
        "cv_result = pd.DataFrame(cv_result)\n",
        "cv_result = cv_result[['param_max_features', 'param_n_estimators', 'mean_test_score']]\n",
        "cv_result_pivot_table = cv_result.pivot_table('mean_test_score', 'param_max_features', 'param_n_estimators')\n",
        "heatmap = sns.heatmap(cv_result_pivot_table, cmap='Reds', annot=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 317
        },
        "id": "wgvtvPxdh018",
        "outputId": "c89c5186-a53a-419c-cdc4-248c3ac4add6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3258\n",
            "1640\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEHCAYAAABLKzaMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5gUVdbA4d/pngTDkHOSKIqggJgjmAAD6rp+gIppQVdBZU3osoqsec0YweyaMKNEA4ZVlCBIUnQIEoecw4Tu8/1RNUP3CDPd0mG65rzPUw8VblWfKtszt2/duiWqijHGGG/yJTsAY4wx8WNJ3hhjPMySvDHGeJgleWOM8TBL8sYY42FpyQ6gTDu3WNefYvm7kh1BxZFZJdkRVAiBH8YlO4QKxd+tn+zvMa6W6hHnnGd1635/XiJYTd4YYzysYtfkjTEmgbxY67Ukb4wxrjRJiRaYqFiSN8YYl897Od6SvDHGFLPmGmOM8TCfNdcYY4x3WU3eGGM8zNrkjTHGw/zWXGOMMd5lzTXGGONh1lxjjDEeZjV5Y4zxMOtCaYwxHmY1eWOM8bA071XkLckbY0wxH97L8pbkjTHG5cXeNV5sgjLGmD/FF8VUHhHpISILRSRXRIbuZXtzEZkiIrNEZI6I9HLX13HXbxeRJ0vtc7iIzHWP+YRI+XeKLckbY4zLJ5FPZRERP/AU0BNoD/QVkfalig0DxqhqZ6AP8LS7fjfwL+CmvRz6GWAA0NadepR7TuUVMMaYyiJNJOKpHEcCuaq6WFULgLeA3qXKKFDdna8BrAJQ1R2q+j+cZF9CRBoB1VX1e1VV4FXg3HLPqbwCxhhTWcSw1tsEWB6yvAI4qlSZ4cBkERkMZAOnRnDMFaWO2aS8QKwmb4wxrmiaa0RkoIjMCJkGRvlxfYGXVbUp0At4TURinpOtJu/6+tup3POfhwkGg/z13N4MvOLSsO2rVudx6x13sW3bNgLBIDcNvpaTTjguSdHG19dTf+CeR0c61+KcMxnY/6Kw7avy1nDriHvZtn07gUCQm669ipOOPTpJ0cZXZf9efDM/l/vGTCQQDHLBcV0Y0OP4sO2rNm7h9pc/ZOuu3QSDQYaceyondWzL5u07uWHUO8z9fSXnHd2JYX17JekMohNNF0pVHQWM2sfmlUCzkOWm7rpQV+K2qavqVBHJAuoCa8s4ZtNyjvkHluSBQCDAiPsf5KVnnqRBg/pccNGldD/pBNq0blVS5pnnX6TnaafQ78ILyF20mIGDh/DFCR8lMer4CAQCjHjoMV564mEa1K/HBZdfRfcTjqNNyxYlZZ556VV6ntKNfn85l9wlSxk45Fa++PDtpMUcL5X9exEIBrn7zfE8f/0lNKhVnf+7bzTdDm1Hm8b1Sso8N/5rehzenj4nHUHuqnVc/eTrnNTxBjLS0xh8Tjd+W7WW3JX7ylkVTwy7UE4H2opIS5xE3AfoV6rMMuAU4GURORjIAtbt64CqulpEtorI0cAPQH9gZHmBWHMNMGfefA5o1pRmTZuQkZ7OmWeczudffh1WRkTYvmMHANu2b6d+vbrJCDXu5iz4mQOaNqFZk8bOtTitO59//b+wMs612AkUX4s6yQg17ir792Lu0pU0r1+bZvVqkZHmp+cRh/DFnF/CCwls350PwPbdu6lfMweAqpkZHN6mOZlpqVWPlCimsqhqETAImAT8jNOLZr6IjBCRc9xiNwIDROQn4E3gMveGKiKyFHgEuExEVoT0zLkGeB7IBRYBE8o7p9T6LxAna9auo2GDBiXLDRrUZ868+WFlBl01gCuvGcx/33qHXbt28dKzT5Y+jCesWbeehvXrlyw3qF+POfN/Disz6G+Xc+X1N/Lfd95n1+5dvDTykUSHmRCV/XuxZtM2GtaqXrLcsGZ15iwJbx0YdNbJ/O3x//L6lGnsKijkhesvSXSYMRVBr5mIqep4YHypdXeEzC8A9tq2p6ot9rF+BtAhmjjiWpMXkYNE5BQRqVZqfbl9OyuacRMncd7ZZ/H1pE8YNfJRbhk2nGAwmOywkmLc5M84r1dPvv74XUY98gC3DL+n8l6LSv69GDd9HucecxhT7v8Hzw7qx60vfUAwqMkO60+LVT/5iiRuSV5ErgM+AgYD80QktI/ovWXsV3LHetSLL8crvDAN6tcjb82akuU1a9bSoF69sDLvfjiWnqc7PZw6H3Yo+QX5bNq8OSHxJVKDenXJW7unDXXN2nU0KNUE8e7H4+l5ajcAOnfsQH5BAZs2b0lonIlQ2b8XDWrlkLdpa8ly3uat1K+VE1bmvW9n0ePwQwDo1KoZBUVFbNq+M6FxxlIsn3itKOIZ6wDgcFU9FzgZ+JeIXO9u2+ffQVUdpapdVbXrwCsui2N4e3Q8pD1Lly1n+cqVFBQWMm7SZLqffEJYmUYNGzJ12nQAFi1eQn5+AbVr1UpIfInU8eCDWLp8BctXrXauxadf0L1Ub5FGDeozdfpMABYtWUp+QQG1a9VMRrhxVdm/Fx0OaMLvazewYv0mCooCTJg+n26Htgsr06h2Db7/ZQkAi1avI7+wiNo5VZMRbkzEqk2+IhG3nT/2BxaZr6qHhCxXA94FFgDdVbVTuQfZuSVhv/u++uZb7n3oEQLBIH/pfTZ//9sVPP70c3RofzCnnHwiuYsWM+zf97Jz505EhJtvGMzxxySw22D+roR91Ffffc+9j450rsVZvfj75Zfw+KgX6HDQQZxy4nHkLlnKsHv/w85du5xrMehqjj/qiITFR2aVhH1URf5eBH4YF/fP+Grub9z/zkSCQeW8Yztxda8TGTl2Cocc0Jjuh7Ujd9U67vzvx+zMLwCBm84/jePatwbg1NsfY/vufAoDAapXyWL0dZeE9cyJNX+3fvude9+u1SDinPN/m9akRK6PZ5L/AviHqs4OWZcGvAhcpKr+cg+SwCRf4SUwyVd4CUzyFVkiknwqiUWSHxNFkr8wRZJ8PJtr+gN5oStUtUhV+wMnxvFzjTHmT/Fim3zculCq6ooytn0br881xpg/K4KRe1OO9ZM3xhiX91K8JXljjCmRSs0wkbIkb4wxLg+21liSN8aYYvYib2OM8TDvpXhL8sYYUyKVxqSJlCV5Y4xxiQfr8pbkjTHG5b0Ub0neGGNKWO8aY4zxML8H6/KW5I0xxuW9FG9J3hhjSlhzjTHGeJgHc7wleWOMKWZdKI0xxsPsYShjjPEwG4XSGGM8zJprjDHGw7zYu8aLv06MMeZPieU7XkWkh4gsFJFcERm6l+3NRWSKiMwSkTki0itk223ufgtF5IyQ9deLyDwRmS8iN0R6TsYYY3C6UEY6lXkcET/wFNATaA/0FZH2pYoNA8aoamegD/C0u297d/kQoAfwtIj4RaQDMAA4EjgMOEtE2pR3ThW6uSbww7hkh1BhzOh7a7JDqDA6n3lwskOoMHYv25DsECqM6t367fcxfLFrrzkSyFXVxQAi8hbQG1gQUkaB6u58DWCVO98beEtV84ElIpLrHq8p8IOq7nSP+RVwPvBgWYFYTd6YFGUJPvZiVZMHmgDLQ5ZXuOtCDQcuFpEVwHhgcDn7zgNOEJE6IlIV6AU0Ky8QS/LGGOMSkWimgSIyI2QaGOXH9QVeVtWmOAn7NRHZZ05W1Z+BB4DJwERgNhAo70MqdHONMcYkUjQPQ6nqKGDUPjavJLyW3dRdF+pKnDZ3VHWqiGQBdcvaV1VfAF4AEJF7cWr5ZbKavDHGuMQnEU/lmA60FZGWIpKBcyN1bKkyy4BTAETkYCALWOeW6yMimSLSEmgLTHPL1Xf/bY7THv9GeYFYTd4YY1yxuu+qqkUiMgiYBPiBF1V1voiMAGao6ljgRmC0iAzBuQl7maoqMF9ExuDcpC0CrlXV4maZ90SkDlDort9cXiwRJ3kR+SswUVW3icgwoAtwt6r+GOkxjDGmIvPFcPAaVR2Pc0M1dN0dIfMLgOP2se89wD17WX9CtHFE01zzLzfBHw+citMu9Ey0H2iMMRWVSORTqogmyRf/XDgTGKWq44CM2IdkjDHJEU3vmlQRTZJfKSLPAf8HjBeRzCj3N8aYCq2y1+QvxLmJcIbb2F8buDkuURljTBL4RCKeUkXESd59lHYtcLy7qgj4LR5BGWNMMnixJh9N75o7ga5AO+AlIB34L/u4O2yMManG50+h7B2haPrJnwd0Bn4EUNVVIpITl6iMMSYJUumGaqSiSfIFqqoiogAikh2nmIwxJik8mOOjuvE6xu1dU1NEBgCfAaPjE5YxxiSeF7tQRlSTF+eM3gYOArbitMvfoaqfxjE2Y4xJqBTK3RGLKMm7zTTjVbUjYIndGONJ/hgOa1BRRNNc86OIHBG3SIwxJskqbXON6yjgIhH5HdiB83IUVdVD4xKZMcYkWArl7ohFk+TPKL+IMcakrsqe5DVuURhjTAUQwctAUk40SX4cTqIXnDeYtAQWAofEIS5jjEm4Sl2Td3vWlBCRLsA1MY/IGGOSxIu9a/706/9U9UcROSqWwSTaN/NzuW/MRALBIBcc14UBPY4P275q4xZuf/lDtu7aTTAYZMi5p3JSx7Zs3r6TG0a9w9zfV3Le0Z0Y1rdXks4gdmp0O5kD/j0c8ftZ+/qbrH7y6bDtGU0a0/qJR/FXr474/Sy75z62fD6FtFo1afv8c2R3Oox1b7/D77f/K0lnEDvSoStpff8O4iPwzUSCE94OL1C7HmlX3gxVqzll3nsBnTsdsnNIu+ZfSIt2BL+dTOCNp5ISfyz5ux5D1jU3IT4/BRM+pODtl8O2S72GVLnlLqRaNfD5yX9hJEXTvkVyalDljgfxt2tP4eSP2f3kg8k5gSilUq+ZSEUzQNk/QhZ9OK//WxXziBIkEAxy95vjef76S2hQqzr/d99ouh3ajjaN65WUeW781/Q4vD19TjqC3FXruPrJ1zmp4w1kpKcx+Jxu/LZqLbkr1ybxLGLE56PFfXfzy4X9KFi9mkMmfsLmyZ+y69c9g4w2ueE6Noz9hLWvvEaVA9vS7vVXmH3EsQTz81n+wENUPagdVQ5ql8STiBHxkXbRIAofHgqb1pP2r5EEZ0+F1ctKivjPuojg9K8JfvkJNGpO+g13U3hrfygsJPDBK0iTFkiTFsk7h1jx+agyeCg7br0GXb+G7Cdfo2jqVwSXLSkpknnRlRR+9SmFn7yLr3lLqt7zBNsvORstzCf/5WfwtWyNv0XrJJ5EdDyY46PqJ58TMmXitNH3jkdQiTB36Uqa169Ns3q1yEjz0/OIQ/hizi/hhQS2784HYPvu3dSv6YzHVjUzg8PbNCczzRvvQa/WuRO7lywlf9kytLCQjR+OpdYZp4eVUVX8OdUA8OfkUJC3BoDgzl1snzadYH5+wuOOB2nVDl27CtbnQaCI4LSv8HU+NryQKlSp6pSvmo1u3uCsL9iN5s6HooIERx0f/naHEFy1HM1bCUVFFH45mbRjTw4vpIpkO8NYSXY1ghvWOet37yYwfzYUpNa1qOz95Beo6juhK9yXe7+zj/KIyJE4femni0h7oAfwi/uC26Ras2kbDWtVL1luWLM6c5asDCsz6KyT+dvj/+X1KdPYVVDIC9dfkugwEyKjUUMKVu35UVawejXZXTqHlVn50KMc9PbrNLzicnxVq/Dzhf0SHWZi1KyLbly3Z3nTOqTlQWFFAmNfI+0f9+Hv3hsysyh6eGiCg0wMqVuf4Lo1Jcu6fg3+gzqElcl/bRRV73+KjN7/h2RVYcetf090mDGVQrk7YtHU5G+LcB1QMv78E8AzInIf8CSQDQwVkX+Wsd9AEZkhIjNGf/JFFOHF3rjp8zj3mMOYcv8/eHZQP2596QOCwcrZk7TOeb1Z9/Y7zOpyJAsvupQ2Tz7mzf8jIuA7qhvBbydTePNFFD0+jLS/3VJpr0V6tzMonPwx2/v1Yuc/r6PKrf9O6Wvh80vEU6ootyYvIj2BXkATEXkiZFN1nLdD7csFQCecpp08oKmqbhWRh4AfgHv2tpOqjgJGAQSmvBG3jNqgVg55m7aWLOdt3kr9WuHD47/37SxGDb4IgE6tmlFQVMSm7TupU91boywXrM4jo3HjkuWMRo0oXJ0XVqZev/9jYV/nl8z2mT8imZmk1alN0foNCY017javR2rvuS9DrXp7mmNcvuPPoOhRp56ii36G9AyoVgO2bU5kpHGn69fiq9egZFnqNiC4fl1YmfQevdl5+2AAAj/PRTIykBo10c2bEhprrKRSM0ykIqnJrwJmALuBmSHTWMp+CrZIVQPuawMXqepWAFXdBQT3K+oY6HBAE35fu4EV6zdRUBRgwvT5dDs0/MZho9o1+P4X5ybTotXryC8sonZO1WSEG1fbZ/9EVqsWZDZvhqSnU/vcc9g0OXwcuoKVq6h+gtP7KKttG3yZWd5L8IAuWYg0aAJ1G4I/Dd+RJ6Gzp4YX2rgOX/tOznyjZk6S91iCBwgsXICvSTOkYWNISyP95NMpmvpVWBldm0da5yMB8DVvARmZKZvgAfBJ5FOKENXIKssikq6qhREfWOQHoJuq7hQRn6oG3fU1gCmq2qW8Y8SzJg/w1dzfuP+diQSDynnHduLqXicycuwUDjmgMd0Pa0fuqnXc+d+P2ZlfAAI3nX8ax7V3egqcevtjbN+dT2EgQPUqWYy+7pKwnjmxNqPvrXE7NkCNU7pxwAinC+W6N99m1eMjaXLLjeyYPYfNkz+lyoFtafnQA/iys0GV5f++ly1ffQ1Ap+nf4a+Wg2SkE9iylV/6XBTWMyfWOp95cNyODSAdjyCtz9/B5yPwv0kEx72Jv3d/gkt/RX/6Hho1J+3SIZCVBQqBd59H588EIP2BV52bsv502LmdwkduC+uZE0u7l8X/j2zakceR+fcbnS6Ukz6i4I0Xybz0agK/LqBo6tf4mrck6x/DkKyqgLJ79BMEZn4PQLXXPkaqZkN6Orp9GzuHXhvWMyfWqn86c78z75ZunSLOOTWmzE6JTB9Nkm8L3Ae0x3niFQBVbbWP8pmq+ocuFyJSF2ikqnPL+8x4J/lUEu8kn0rineRTRSKSfCqJRZLf2r1zxDmn+hezyvw8EekBPA74gedV9f5S25sDrwA13TJDizuliMhtwJVAALhOVSe564cAf8MZfWAucLmq7i4rjmhuvL4EPIPTDt8NeBXnRd57tbcE765fH0mCN8aYhItRc42I+IGngJ44FeO+bg/DUMOAMaraGegDPO3u295dPgSnR+LTIuIXkSbAdUBXVe2A84ehT7mnFMXpV1HVz3Fq/7+r6nDgzCj2N8aYCk38voinchwJ5KrqYlUtAN7ij88VKU4HFoAa7Hm4tDfwlqrmq+oSINc9HjidZaqISBpQlQgeSI0myeeLiA/4TUQGich5QLUo9jfGmIpNJPKpbE2A5SHLK9x1oYYDF4vICmA8MLisfVV1JfAQsAxYDWxR1cnlBRJNkr8e5y/HdcDhwMXApVHsb4wxFZr4JPIp5JkedxoY5cf1BV5W1aY43dRfcyvSe49NpBZOLb8l0BjIFpGLy/uQaEahnO5+UFBVL490P2OMSRlR9JMPfaZnL1YCzUKWm7rrQl2J0+aOqk4VkSygbhn7ngosUdV1TqjyPnAsZdwbhShq8iJyjIgsAH5xlw8TkafL2c0YY1JH7PrJTwfaikhLEcnAuUE6tlSZZcApACJyME6vxXVuuT4ikikiLYG2wDS3/NEiUlWcp7ZOAX4uL5Boxq55DOfhp7EAqvqTiJwYxf7GGFOhxeqJV1UtEpFBwCScXjAvqup8ERkBzFDVscCNwGi3W6QCl6nTp32+iIwBFuD0ZrxWVQPADyLyLvCju34W+/4lUSKqYRRVdXmpixCIZn9jjKnQyu81EzG3z/v4UuvuCJlfABy3j33vYS9Dv6jqncCd0cQRTZJfLiLHAioi6Tg3Ysv9qWCMMali37c9U1c0p3Q1cC1O956VOIOPXRuPoIwxJili14WywohkFMoHVPVWnHFoLkpATMYYkxSSQgOPRSqSmnwv907uPseON8YYT6iMNXlgIrAJqCYiWwHBuRMsOG99ql7WzsYYkyoiGK4g5ZR7Rqp6s6rWBMapanVVzQn9NwExGmNMYnhwPPmI/2ypapkv7RaRqWVtN8aYCq+SNtdEKqv8IsYYU3F58fV/sUzy9oIPY0xqS6FmmEjFMskbY0xKs5p82bx3dYwxlUtl7F1TbC+vrkJETg5ZvCQWARljTLJEM558qojmz9YYEblVHFVEZCTOi70BUNV5sQ/PGGMSyIO9a6JJ8kfhDGT/Hc5YyavYxwhqxhiTkjzYTz6aNvlCYBdQBae75BJVDcYlKteywcPjefiUcni/I5IdQoUx6NHPkh1ChTCoRZ1kh1ChdIjBMbx44zWamvx0nCR/BHAC0FdE3olLVMYYkwyVvCZ/parOcOdXA71FxG62GmO8w+e93jXRvMh7BoCI1GfP061fxSMoY4xJCg8210Sc5EXkbOARoDGwFjgA581Qh8QnNGOMSTAP1uSjOaO7gaOBX1W1Jc6bwr+PS1TGGJMMlbwLZaGqbgB8IuJT1SlA1zjFZYwxiefBJB/NjdfNIlIN+Bp4XUTWAjviE5YxxiSB35/sCGIumpp8b5wulENw3ha1CDg7HkEZY0xSVOaavKruABCR6sDHcYvIGGOSJYWSd6Si6V1zFXAXsBsIsuddr63iE5oxxiSYB5N8NM01NwEdVLWFqrZS1ZaqagneGOMdPl/kUzlEpIeILBSRXBEZupftzUVkiojMEpE5ItIrZNtt7n4LReQMd107EZkdMm0VkRvKiyOaG6+LgJ1RlDfGmNQSo5q8iPiBp4DTgBXAdBEZq6oLQooNA8ao6jPuUO7jgRbufB+cZ5AaA5+JyIGquhDoFHL8lcAH5cUSTZK/DfhORH4A8otXqup1URzDGGMqrtg9DHUkkKuqiwFE5C2cziuhSV6B6u58DZyRfXHLvaWq+cASEcl1jzc1ZN9TgEWq+nt5gUST5J8DvgDm4rTJG2OMp0jsknwTYHnI8gqc4dpDDQcmi8hgIBs4NWTf0AdNV7jrQvUB3owkkGiSfLqq/iOK8sYYk1qiaK4RkYHAwJBVo1R1VBSf1hd4WVUfFpFjgNdEpNwRk0UkAzgHp3WlXNEk+QnuSX1MeHPNxiiOYYwxFVcUSd5N6PtK6itxXrJUrKm7LtSVQA/3WFNFJAuoG8G+PYEfVXVNJHFG89ukL267PDDTnWaUuYcxxqSS2D0MNR1oKyIt3Zp3H2BsqTLLcNrWEZGDcUb3XeeW6yMimSLSEmgLTAvZry8RNtVAdA9DtSxru4icpqqfRno8Y4ypcGLUJq+qRSIyCJgE+IEXVXW+iIwAZqjqWOBGYLSIDMG5CXuZqiowX0TG4NykLQKuVdUAgIhk4/TYuSrSWKJprinPA4AleWNM6orhUMOqOh6nW2ToujtC5hewj/dkq+o9wD17Wb8DiOq9j7FM8t57VMwYU7l48InXWCZ5jeGxjDEm8Tz40pBYJnljjEltHqzJx/LP1tIYHishqhx3Ak0+nkjT8Z9S48qBf9jub9iIhi++SuN3PqTJ+2OpcsJJAPhq1KThi69ywLRZ1Ln9jj/sl4rkoM74h47Ef/tTSPfz/ligZl1819yF/x8P4b/pEeTgLs76qtWc9fe9ju/8vyU26Dhpf8apDP9lJiN+m80Ztw75w/ZazZoy5ItPuP3Hbxj203d06Hk6ANm1azPki094bNsq+ox8KNFhx0W1E0+k7eef03bKFOpeffUftqc3bkyLN96g9Sef0GbCBKqdfDIA/po1afHGGxw8bx6N7rorwVHvh8o81LA7VsKZQIvQ/VT1Efff82MdXFz5fNQZdid5Ay6nKC+Pxm+/x84pn1O4eFFJkZpXXcOOSRPY9vabpLdqTYNnRrPijO5oQT6bRj5ORtu2ZLQ5MIknESPiw3f+AALP3gVbNuAf8iCB+dNhzYqSIr7TLkBnf0fwu0nQoCn+AcMI3H01FBUSnPAm0rA50qh5Ek8iNsTno+9TD/P4ab3ZtGIlt03/kjljx7P654UlZXoNu5mZYz7g62dfoNHB7Rg0/l3+2bIjhbt3M/Zfd9O4Q3uadGifxLOIEZ+PxiNGsOSSSyjKy6PVRx+x7bPPyM/NLSlSb9Agto4bx8bXXyezTRsOeOklfj3hBIL5+ax95BEyDzyQrHbtkngSUarkLw35GLgM585uTsiUkjI7Hkrhst8pWrEcigrZMWEcVbufGl5IFV92NQB8OTkE1q11Vu/aRf6smWh+funDpqbmbdD1q2HjGggUEZz1P6TDkX8sl1V1z79b3GfgCvJhyS9QVJi4eOOoxZFdWZu7mPVLlhIoLGT6W+9xaO8zw8qoKlnVna9+Vo0abF6VB0DBzp0s+vZ7inbvTnjc8VDlsMPI//13CpcvRwsL2fLxx+Scdlp4IVV81fb8P1K0xnk+R3ftYueMGan3/0hlrskDTVX10P35MBF5VVX7788xYsVfvwGBvLyS5cCaPDI7HhZWZvPTI2k46kWq97sEqVKFvAGXJTjKxJAadWDzhj0rNm9ADmgbdic9OPFt/FffAcf3goxMAs8OT3SYCVGrSSM2Ld/zC2bzilW0PCr8VcafDL+P6yd/SLfBV5GRXZXHT+2d6DATIr1hQwpXry5ZLsrLo0qnTmFl1j72GC1efZU6l16Kr2pVllx8caLDjK0USt6RiqYmP0FETo+0sIiMLTV9DJxfvFzGfgNFZIaIzHhz45Yowou97F5nse2jD1h+6omsuWYA9e77jye/BJGQLscTnDaFwIgBBEbfjb/f9ZX2WhzR9wKmvvw6tzU7mCd7/ZXLXxuFVNJrUeOcc9j03nssPPZYll5+OU0feSS1vxcxHE++oogm0u+BD0RklztY/TYR2VpG+abAVuAR4GF32hYyv1eqOkpVu6pq1761a0QRXnQCa9fgb9iwZNnfoCFFa8OHgsg5/wJ2THKeZcj/aTaSkYmvVq24xZQsumUD1Ax5vqJmHXRL+JBEvqNOQX/61ln4/VdIT4fs6njNppWrqdWsaclyzaaN2bRyVViZ467sz8wxzjDeS76fRlpWJtXqRvV8SkoozMsjvVGjkuW0hj9z9rgAABXFSURBVA0pDPn1C1DrwgvZOm4cALtmzcKXmYm/du2ExhlTHmyuiSbJPwIcA1RV1eqqmqOqZf1f3hVnfJt/AltU9Utgl6p+papf/emIYyR/3lzSm7cgrUlTSEsnu+eZ7JzyeViZotWrqXLUMQCkt2qNZGYQ3OjB8diW5yL1GkHt+uBPw9f5eHTe9LAiumk90tZtravfBNIyYHtyf2nFw+/TZ1K/bSvqtDgAf3o6R/T5C3PGhj20yMZlKzjoFKenVcODDiQ9K4tt69YnI9y42jVnDpktWpDetCmSnk6Ns89m22efhZUpXLWK7GOPBSCzdWskM5PAhg17O1xq8GCSF2eohAgKinwNnKyqUY0lLyJNgUeBNcA5qhpxF4wlHQ6M6wNWVU44iTq33g5+P9s+eJcto56l5rXXUTB/Hju//IL0Vq2pe9fdSNVsUGXTIw+y6zunNtt00hf4qlVD0tMJbt1G3sDLw3rmxFqz08sdgXS/yMFd8PW+Anw+gtM+Rz97D1+PPujyRej86U6PmguvgcwsUCX48avorz8B4B/2LGRVAX8a7NpJ4Lm7wnrmxNqgRz8rv9B+6NDzdP762P34/H6+e/E1Jtz7EGff9U9+n/Ejcz6eQKOD23Hx6JFkVstGVXn/ljv4+dMvALhnyVyyqlfHn5HOrs1beOL0c8N65sTSoBbx//VQ7eSTaXTHHYjPx6Z33mHdU09Rf8gQds2dy7bPPiOzTRsa33cfvmzn/5E199/P9m++AeDAb74J+X9kK0v79w/rmRNrHZYs2e/MG3h2aMQ5x3/1/SmR6aNJ8i/jvLR7AuFDDT8S4f5nAsep6u2RBhfvJJ9K4p3kU0m8k3yqSESSTyUxSfLP3R55kr/q3pRI8tH0rlniThnuFBVVHQeMi3Y/Y4xJmBRqholUNEMNp9Bja8YY8yekUK+ZSEXzxGs94BacN4hnFa9X1e5xiMsYYxLPgzX5aP5svQ78ArQE7sIZq2Z6WTsYY0xK8fkjn1JENEm+jqq+ABS63SCvAKwWb4zxDg8+DBXNjdfiwUlWuz1lVgEp/NSDMcaU4sHmmmiS/N0iUgPnvYQjgerAH8dhNcaYVCWpU0OPVERJ3h1muK2qfgJsAbrFNSpjjEkGn/dq8hH92XLfFN43zrEYY0xyiS/yKUVE01zzrYg8CbwN7Cheqao/xjwqY4xJhhTqNROpaJJ88UDSI0LWKdbDxhjjFR5sronmiVdrhzfGeFsl711TPMhY6SdeR+x7D2OMSSEp1NYeqWiGNXgWqIrTs+Z54AJgWpziMsaYxPNgc000f7aOdd/PuskdrOwY4MD4hGWMMUkQw941ItJDRBaKSK6IDN3L9uYiMkVEZonIHBHpFbLtNne/hSJyRsj6miLyroj8IiI/i8gx5cURTXPNLvffnSLSGNgANCqjvDHGpJYY9a5xny16CjgNWAFMF5GxqrogpNgwYIyqPiMi7YHxQAt3vg9O03hj4DMROdDtyv44MFFVLxCRDJzWlTJFk+Q/EZGawIM4r/UDp9nGGGO8IXbNNUcCuaq6GEBE3gJ6A6FJXnFGDgCogTNUDG65t1Q1H1giIrnAkSKyADgRuAxAVQuAgvICiSbJPwT8HTgBmAp8AzwTxf7GGFOxRXHjVUQGAgNDVo1S1VHufBNgeci2FcBRpQ4xHJgsIoOBbODUkH2/L7VvE5zWlHXASyJyGE5l+3pV3UEZommTfwXn58MTOGPXtAdejWJ/Y4yp2KJ4kbeqjlLVriHTqPI/IExf4GVVbQr0Al4TKfOvTBrQBXhGVTvjPJT6h7b+ve0UqQ6q2j5keYr788EYY7whdkMIrwSahSw3ddeFuhLoAaCqU0UkC6hbxr4rgBWq+oO7/l0iSPLRnNGPInJ08YKIHAXMiGJ/Y4yp2GL30pDpQFsRaeneIO0DjC1VZhlwCoCIHIzz/NE6t1wfEckUkZZAW2CaquYBy0Wknbv/KYS38e9VNDX5w4HvRGSZu9wcWCgicwFV1UOjOJYxxlQ8MXriVVWLRGQQMAnwAy+q6nwRGQHMUNWxOMO2jxaRITg3YS9TVQXmi8gYnAReBFzr9qwBGAy87v7hWAxcXu4pOccsn4gcUM5J/R7RgaJQdEf/yIKrBJa//335hSqJZucfXX6hSkBycpIdQoXiv/mp/c7Qgc9ejTjn+E/tnxJPTkUzdk3Mk7gxxlQolX3sGmOM8bTKPHaNMcZ4ntXkjTHGw/yV+6UhxhjjbdZcY4wxHmbNNcYY42FWkzfGGO8Sq8kbY4yHWU3eGGM8zHrXGGOMh1lN3hhjPMza5I0xxsOsJm+MMR5mNXljjPEwu/FqjDEeZs01xhjjYdZcY4wxHmY1eWOM8TCryRtjjIdZTd4YY7xLrHeNMcZ4mNXkvUXadMTX62IQH8Efv0K/+SS8QI06+M4fgGRlgwjBT8egv82BKtXw9RmENG6Fzv6G4LjXknMCMVTluBOoPfSfiN/PtvfeYcsLo8K2+xs2ot69D+DLqY74fWx89GF2ffMVvho1qf/oE2R26Mj2Dz9gw70jknQGsWPfixAt2uM75QIQHzrnW3Tap+Hbc2rh69UfMquAz0fwq49gyXzIysbX+2/Q8AB03vfo52OSE3+0rE3eQ0TwndWfwCsPwtaN+K+6i8AvP8K6VSVFfCedg86bRnD6F1CvMf6LbyTw6I1QVEDw8/eR+k2QBk2TeBIx4vNRZ9id5A24nKK8PBq//R47p3xO4eJFJUVqXnUNOyZNYNvbb5LeqjUNnhnNijO6owX5bBr5OBlt25LR5sAknkSM2PdiDxF8p11IcMxI2LYZ3yW3oIvmwoa8PUWO6YEu/BGd/Q3UaYjvL9cQHHUHBAoJ/u8TpG4jqNs4iScRJQ/W5L13RpFq2hrduBY2rYNAgODc75GDuoSXUXVqKABZVWHbZme+sACW/QpFhYmNOU4yOx5K4bLfKVqxHIoK2TFhHFW7nxpeSBVfdjUAfDk5BNatdVbv2kX+rJlofn6iw44P+17s0aiFcx22bIBgAP1lJtLm0D+Wy8hy/s2sAtu3OPOFBbByERQVJSzcmBCJfEoRlbYmLzm1nC9vsa0bkaat0ZAywSkf4L/0FjjqNMjIJPDyAwmPMxH89RsQyNtTOwusySOz42FhZTY/PZKGo16ker9LkCpVyBtwWYKjTAz7XoSoVhPdtmnP8rbNTuIPod+Ow/fXQUiXkyA9k+CYJxIbY6z5YlfvFZEewOOAH3heVe8vtb058ApQ0y0zVFXHu9tuA64EAsB1qjrJXb8U2OauL1LVruXFkbAkLyLHA0cC81R1cqI+d3/IoccQnPUN+t1EaNYG/1+uIvDU7U5NrpLJ7nUW2z76gK2vvEjmYZ2od99/WHnumZXyWtj3Yg85uCs67wd0xufQuCW+XpcSfOkeIEWvhS82vWtExA88BZwGrACmi8hYVV0QUmwYMEZVnxGR9sB4oIU73wc4BGgMfCYiB6pqwN2vm6qujzSWuDXXiMi0kPkBwJNADnCniAwtY7+BIjJDRGaM/vHXeIXn1FBq1NmzonptdOumsDK+Liei89zTWJ4LaelQtVrcYkqWwNo1+Bs2LFn2N2hI0do1YWVyzr+AHZPGA5D/02wkIxNfrVoJjTMR7HsRYvtm55dNsZyasH1zWBHpeCy6cKazsGqJey2yExhkjMWuueZIIFdVF6tqAfAW0LtUGQWqu/M1gOIbP72Bt1Q1X1WXALnu8f6UeLbJp4fMDwROU9W7gNOBi/a1k6qOUtWuqtp1QJc43shbuRip3QBq1gW/H1/Ho9FfZoXHsmUD0qq9s1C3sfMF3rEtfjElSf68uaQ3b0Fak6aQlk52zzPZOeXzsDJFq1dT5ahjAEhv1RrJzCC4cWMywo0v+17ssfp3qFXf+aPn8yMHHY7mzg0vs3Uj0vwgZ752A0hLg53bEx9rzEjEU2iF1J0GhhyoCbA8ZHmFuy7UcOBiEVmBU4sfHMG+CkwWkZmlPm+f4tlc4xORWjh/SERV1wGo6g4RSf7dmGCQ4LhX8fe/BXxC8MevYd1KfN3PR1cuQRfOIjjxTfy9r4Bje4AqwQ9Gl+zuH/Kwc6PJn4b/oMMJvPpgWA+MlBIIsOHeETR87gXw+9n2wbsULsql5rXXUTB/Hju//IKN/7mPunfdTfX+l4Mq64ft+THWdNIX+KpVQ9LTqdr9VPIGXh7WMyel2PdiDw0S/GwMvguuBZ8PnTsVNqxGjjsTzVsGi+YS/PJ9fGf0Q7p2AyA4YU+3Ud/AEc5NWX8a0vZQgu88GdYzp0KK4oaqqo4CRpVbcN/6Ai+r6sMicgzwmoh0KGef41V1pYjUBz4VkV9U9euydhCNUzuie4MgiPNnT4HjVHW1iFQD/qeqnco7RtEd/VO0YS/2lr//fbJDqDCanX90skOoECQnJ9khVCj+m5/a7y4vuuLniHOOND14n5/nJu3hqnqGu3wbgKreF1JmPtBDVZe7y4uBo3FuuJaUFZFJ7rGmlvqM4cB2VX2orDjj1lyjqi1UtZWqtnT/Xe1uCgLnxetzjTHmTxNf5FPZpgNtRaSliGTg3EgdW6rMMuAUABE5GMgC1rnl+ohIpoi0BNoC00QkW0Ry3PLZOE3f88oLJOFdKFV1J7Ak0Z9rjDHlilH3d1UtEpFBwCSc7pEvqup8ERkBzFDVscCNwGgRGYLT2nGZOk0r80VkDLAAKAKuVdWAiDQAPhCnSSkNeENVJ5YXS6XtJ2+MMX8Uu4ec3D7v40utuyNkfgFw3D72vQe4p9S6xcBheytfFkvyxhhTLIWeZI2UJXljjClmSd4YY7zMkrwxxniXB0ehtCRvjDHFrLnGGGM8zJK8McZ4mSV5Y4zxLLGavDHGeJgleWOM8TDrXWOMMR5mNXljjPEyS/LGGONdVpM3xhgP816OtyRvjDEl7MarMcZ4mDXXGGOMl1mSN8YY77KavDHGeJgleWOM8TLvJXlxXg5uyiIiA1V1VLLjqAjsWuxh18Jh16Fi815/ofgYmOwAKhC7FnvYtXDYdajALMkbY4yHWZI3xhgPsyQfGWtv3MOuxR52LRx2HSowu/FqjDEeZjV5Y4zxMEvyxhjjYZbkARFZKiJzRWS2iMxw19UWkU9F5Df331ruehGRJ0QkV0TmiEiX5Ea/f0TkRRFZKyLzQtZFfe4icqlb/jcRuTQZ57K/9nEthovISve7MVtEeoVsu829FgtF5IyQ9T3cdbkiMjTR5xELItJMRKaIyAIRmS8i17vrK+V3I6WpaqWfgKVA3VLrHgSGuvNDgQfc+V7ABJxH444Gfkh2/Pt57icCXYB5f/bcgdrAYvffWu58rWSfW4yuxXDgpr2UbQ/8BGQCLYFFgN+dFgGtgAy3TPtkn9ufuBaNgC7ufA7wq3vOlfK7kcqT1eT3rTfwijv/CnBuyPpX1fE9UFNEGiUjwFhQ1a+BjaVWR3vuZwCfqupGVd0EfAr0iH/0sbWPa7EvvYG3VDVfVZcAucCR7pSrqotVtQB4yy2bUlR1tar+6M5vA34GmlBJvxupzJK8Q4HJIjJTRIqf3mugqqvd+TyggTvfBFgesu8Kd52XRHvuXr8mg9wmiBeLmyeoRNdCRFoAnYEfsO9GyrEk7zheVbsAPYFrReTE0I3q/O6slH1NK/O5u54BWgOdgNXAw8kNJ7FEpBrwHnCDqm4N3WbfjdRgSR5Q1ZXuv2uBD3B+cq8pboZx/13rFl8JNAvZvam7zkuiPXfPXhNVXaOqAVUNAqNxvhtQCa6FiKTjJPjXVfV9d7V9N1JMpU/yIpItIjnF88DpwDxgLFDcE+BS4CN3fizQ3+1NcDSwJeTnq1dEe+6TgNNFpJbbnHG6uy7llbrfch7OdwOca9FHRDJFpCXQFpgGTAfaikhLEckA+rhlU4qICPAC8LOqPhKyyb4bqSbZd36TPeH0gvjJneYD/3TX1wE+B34DPgNqu+sFeAqnB8VcoGuyz2E/z/9NnGaIQpz20iv/zLkDV+DcfMwFLk/2ecXwWrzmnuscnETWKKT8P91rsRDoGbK+F05vlEXF36dUm4DjcZpi5gCz3alXZf1upPJkwxoYY4yHVfrmGmOM8TJL8sYY42GW5I0xxsMsyRtjjIdZkjfGGA+zJG+MMR5mSd54kojUFJFrQpYbi8i7MTr2uSLSPhbHMibeLMmbiIhIWrJjiFJNoCTJq+oqVb0gRsc+F2fY3Yil4PUzHmEPQ1Ui7miCE4GZOOOmzwf6AzcBZwNVgO+Aq1RVReRLnCcdj8d5GvRXYBjOOOkbgItUdY2IDMcZU70V0BwYgjOmeE+ccUrOVtXCfcS0FGfI2rOBdOCvqvrLPspmAyOBDm7Z4ar6kYgcArzkxuUD/gL8G2f424U4w9s+BXyiqh1E5DKcRJ2NMxzBQ+6+lwD5QC9V3SgiA4CB7rZcd3sn4BNgizv9BWe89WeBqjhPfF6hqpv2cv2WAXcCAZzH/sMGwjMmLpL9yK1NiZuAFjiPqh/nLr+Ik+Brh5R5DScpA3wJPB2yrRZ7KgZ/Ax5254cD/8NJvIcBO3Ef88cZ8O3cMmJaCgx2568Bni+j7L3Axe58TZw/OsWJ/yJ3fQbOH6sWhL/8o2QZuAwnaecA9XCS9dXutkdxRlwEqBOy/90hcb4MXBCybQ5wkjs/AnhsH9dvLtCkOP5kfx9sqhyTNddUPstV9Vt3/r84tcxuIvKDiMwFugOHhJR/O2S+KTDJLXdzqXIT1Kmtz8V5O9JEd/1cnARbluIRDmeWU/Z0YKiIzMZJoFk4vxymAreLyK3AAaq6q5zPA5iiqttUdR1Okv94L/F2EJFv3PO9iPDzBUBEauAk7K/cVa/gvGGqWOj1+xZ42f2F4I8gRmP2myX5yqd0+5wCT+PUTDviDKebFbJ9R8j8SOBJt9xVpcrlA6gzJG+hqhZ/ThAorz063/03UE5ZAf6iqp3cqbmq/qyqbwDnALuA8SLSvZzPC/3M4hjzQ+aLY3gZGOSe712En2+kSq6fql6N09zVDJgpInX+xPGMiYol+cqnuYgc4873w2lmAVjvviCirJuTNdgzFngyXsg8CRjsDoOLiHR2/20FLFbVJ3CGvj0U2IbTHLM/coDV7rjqF4WsLzm2qm4BNonICe62S4Cv2AsRaa2qP6jqHcA6wsdZNyYuLMlXPgtx3n71M04b+zM4tfd5OEl0ehn7DgfeEZGZwPo4x7k3/8Zp958jIvPdZYALgXluM04HnHeNbgC+FZF5IvKfP/l5/8J55d23QOjN4LeAm0Vkloi0xvmD9x8RmYNzY3bEPo73HxGZKyLzcG5w//Qn4zImYta7phJxe9d8oqodkhyKMSZBrCZvjDEeZjV5kxAi8gFOX/pQt6rqH14FJyKXA9eXWv2tql4br/iM8SpL8sYY42HWXGOMMR5mSd4YYzzMkrwxxniYJXljjPGw/wfHiVhmtvnK4gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "最適な決定木の数は1500、決定木の深さは4であることが分かった。この決定木の数と決定木の深さでランダムフォレストモデルを学習させ、テストデータによる評価を行う。"
      ],
      "metadata": {
        "id": "H7oKk1o5RF_N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rfc = RandomForestClassifier(random_state=0, max_features=4, n_estimators=1500)\n",
        "rfc.fit(norm_X_train, y_train)\n",
        "y_pred = rfc.predict(norm_X_test)\n",
        "\n",
        "print(confusion_matrix(y_test, y_pred))  # 混同行列\n",
        "print(accuracy_score(y_test, y_pred))  # accuracy\n",
        "print(precision_score(y_test, y_pred))  # precision\n",
        "print(recall_score(y_test, y_pred))  # recall\n",
        "print(f1_score(y_test, y_pred))  # f値"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zAwjHjjNAVul",
        "outputId": "30f9ba9c-7b4a-414a-bd20-9c9711a95ce0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[220 108]\n",
            " [ 54 598]]\n",
            "0.8346938775510204\n",
            "0.8470254957507082\n",
            "0.9171779141104295\n",
            "0.8807069219440353\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "最終的なaccuracy は83%となった。"
      ],
      "metadata": {
        "id": "v5kT4yFHRbnH"
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "project.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}