FS-ReliefF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Selection using ReliefF\n",
    "https://epistasislab.github.io/scikit-rebate/using/\n",
    "`pip install skrebate`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from skrebate import ReliefF\n",
    "from sklearn.feature_selection import mutual_info_classif\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import matplotlib.pyplot as plt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seg_data = pd.read_csv('segmentation-all.csv')\n",
    "print(seg_data.shape)\n",
    "seg_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seg_data['Class'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data, scale it and divide into train and test sets.  \n",
    "The filters are *trained* using the training data and then a classifier is trained on the feature subset and tested on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = seg_data.pop('Class').values\n",
    "X_raw = seg_data.values\n",
    "\n",
    "X_tr_raw, X_ts_raw, y_train, y_test = train_test_split(X_raw, y, \n",
    "                                                       random_state=42, test_size=1/2)\n",
    "scaler = MinMaxScaler()\n",
    "X_train = scaler.fit_transform(X_tr_raw)\n",
    "X_test = scaler.transform(X_ts_raw)\n",
    "\n",
    "feature_names = seg_data.columns\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ReliefF\n",
    "- `ReliefF` will produce scores for all features.\n",
    "- `n_features_to_select` controls the transform behaviour, if a dataset is transformed this number of features will be retained. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reliefFS = ReliefF(n_features_to_select=11, n_neighbors=100, n_jobs = -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reliefFS.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relief_scores = reliefFS.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reliefFS.transform(X_train).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also calcuate I-Gain scores: to be used for comparision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i_scores = mutual_info_classif(X_train,y_train)\n",
    "i_scores\n",
    "# The i-gain scores for the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "stats.spearmanr(relief_scores, i_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store the ReliefF and I-Gain scores in a dataframe.  \n",
    "**Note:** The mutual information score returned by `mutual_info_classif` is effectively an information gain score.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=pd.DataFrame({'Mutual Info.':i_scores,'ReliefF':relief_scores,'Feature':feature_names})\n",
    "df.set_index('Feature', inplace = True)\n",
    "df.sort_values('Mutual Info.', inplace = True, ascending = False)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting the ReliefF and I-Gain scores\n",
    "We see that the two scores are fairly well correlated.  \n",
    "The Spearman correlation is 0.81."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "rr = range(0,len(feature_names))\n",
    "ax2 = ax.twinx()\n",
    "ax.plot(df.index, df[\"Mutual Info.\"], label='I-Gain')\n",
    "ax2.plot(df.index, df[\"ReliefF\"], color='red', label='Relief')\n",
    "ax.set_xticks(rr)\n",
    "\n",
    "ax.set_xticklabels(list(df.index), rotation = 90)\n",
    "ax.set_xlabel('Features', fontsize=12, fontweight='bold')\n",
    "ax.set_ylabel('I-Gain')\n",
    "ax2.set_ylabel('ReliefF')\n",
    "fig.legend(loc=\"upper right\", bbox_to_anchor=(1,1), bbox_transform=ax.transAxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection\n",
    "Compare  \n",
    "- Baseline: all features\n",
    "- Visual inspection of the ReliefF plot suggests we select the top 11 features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = KNeighborsClassifier(n_neighbors=3)\n",
    "model = model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "acc_all = accuracy_score(y_pred,y_test)\n",
    "acc_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_features = X_train.shape[1]\n",
    "n_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### After feature selection\n",
    "We produce a reduced dataset with the 11 top ranking features selected by ReliefF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tr_relief = reliefFS.transform(X_train)\n",
    "X_ts_relief = reliefFS.transform(X_test)\n",
    "X_tr_relief.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kNN_relief = model.fit(X_tr_relief,y_train)\n",
    "y_pred = kNN_relief.predict(X_ts_relief)\n",
    "acc_11 = accuracy_score(y_pred,y_test)\n",
    "acc_11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "import numpy as np\n",
    "%matplotlib inline \n",
    "\n",
    "fig, ax = plt.subplots(figsize=(2.5,3.5))\n",
    "width = 0.5\n",
    "sb = 'skyblue'\n",
    "\n",
    "options = ['All', 'ReliefF 11']\n",
    "scores = [acc_all,acc_11]\n",
    "\n",
    "y_pos = np.arange(len(options))\n",
    "\n",
    "p1 = ax.bar(y_pos, scores, width, align='center', \n",
    "            color=['red', 'blue'],alpha=0.5)\n",
    "\n",
    "ax.set_ylim([0.5, 1])\n",
    "plt.grid(axis = 'y')\n",
    "plt.yticks(np.arange(0.5,1.05,0.05))\n",
    "ax.text(0,acc_all, '%0.3f' % acc_all, ha='center', va = 'top')\n",
    "ax.text(1,acc_11, '%0.3f' % acc_11, ha='center',va = 'top')\n",
    "\n",
    "plt.xticks(y_pos, options)\n",
    "plt.ylabel('Test Set Accuracy')\n",
    "plt.xlabel('Features')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}