examples/housing_price_feature_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WEWW_CD4owNL"
   },
   "source": [
    "# Housing price prediction feature engineering\n",
    "\n",
    "In general, price of a house is determined under many factors and the location always plays a paramount role in making value of the property. In this notebook, we will discover how geo-related aspects affect housing price in the USA. We will consider a house based on the information provided in ***an existing dataset*** with some addtional spatial attributes extracted from its location using xarray-spatial ***(and probably some elevation dataset, and census-parquet as well?)***.\n",
    "\n",
    "Existing features:\n",
    "- ...\n",
    "\n",
    "New features:\n",
    "- ***Slope?*** (from an elevation dataset)\n",
    "- ***Population density, ...?*** (from Census if none available in the existing features)\n",
    "- Distance to nearest hospital (or grocery store / university / pharmacy)\n",
    "\n",
    "We'll first build a machine learning model and train it with all existing features. For each newly added feature, we'll retrain it and compare the results to find out which features help enrich the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LWdCWgs31tjo"
   },
   "source": [
    "## Imports\n",
    "\n",
    "First, let's import all neccesary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import rasterio\n",
    "\n",
    "import datashader as ds\n",
    "from datashader.transfer_functions import shade\n",
    "from datashader.transfer_functions import stack\n",
    "from datashader.transfer_functions import dynspread\n",
    "from datashader.transfer_functions import set_background\n",
    "from datashader.colors import Elevation\n",
    "\n",
    "from xrspatial import slope\n",
    "from xrspatial import proximity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eHISReTx1zmb"
   },
   "source": [
    "## Load the existing dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assume the data contain lat lon coords with some additional values\n",
    "df = pd.DataFrame({\n",
    "    'id': [0, 1, 2, 3, 4],\n",
    "    'x': [0, 1, 2, 0, 4],\n",
    "    'y': [2, 0, 1, 3, 1],\n",
    "    'column_1': [2, 3, 4, 2, 6],\n",
    "    'price': [1, 3, 4, 3, 7]\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NoXJhz-7153G"
   },
   "source": [
    "## Build and train housing price model\n",
    "\n",
    "We'll split the data into train data and test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mIm7gBOf9Mi-"
   },
   "source": [
    "Now let's build the model to predict housing price. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t7KiSyr29ROE"
   },
   "source": [
    "After tuning the hyper parameters, we selected the best model as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AL8mMHzI9-Hz"
   },
   "source": [
    "Prediction accuracy on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "18o3Swi-6Fjg"
   },
   "source": [
    "## Calculated spatial attributes\n",
    "\n",
    "As stated above, we'll calculate spatial attributes (slope, ...?) of each location and its proximities to some nearest services. \n",
    "\n",
    "**TBD**: What is the format of additional data? Is it in vector or raster format?\n",
    "- If raster (preferred), load directly as 2D xarray DataArrays\n",
    "- If vector, load into a pandas/geopandas DataFrame and rasterize with datashader.\n",
    "\n",
    "Assume that the data is in vector format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bounding box of the raster\n",
    "xmin, xmax, ymin, ymax = (\n",
    "    df.x.min(),\n",
    "    df.x.max(),\n",
    "    df.x.min(),\n",
    "    df.x.max()\n",
    ")\n",
    "xrange = (xmin, xmax)\n",
    "yrange = (ymin, ymax)\n",
    "\n",
    "# width and height of the raster image\n",
    "W, H = 800, 600\n",
    "\n",
    "# canvas object to rasterize the houses\n",
    "cvs = ds.Canvas(plot_width=W, plot_height=H, x_range=xrange, y_range=yrange)\n",
    "raster = cvs.points(df, x='x', y='y', agg=ds.min('id'))\n",
    "\n",
    "# visualize the raster\n",
    "points_shaded = dynspread(shade(raster, cmap='salmon', min_alpha=0, span=(0,1), how='linear'), threshold=1, max_px=5)\n",
    "set_background(points_shaded, 'black')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6k9E72q9Ge_d"
   },
   "source": [
    "Identify location in pixel space of houses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kmkbu2e2LrQo"
   },
   "source": [
    "Calculate new feature value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b15dhXjXLxMj"
   },
   "source": [
    "Retrain the model with new feature and compute test accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YEFwSg5Q6W1Z"
   },
   "source": [
    "## Feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}