3_feature_views_and_training_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7eb83ff8",
   "metadata": {},
   "source": [
    "# <span style=\"font-width:bold; font-size: 3rem; color:#1EB182;\"><img src=\"images/icon102.png\" width=\"38px\"></img> **Hopsworks Feature Store** </span><span style=\"font-width:bold; font-size: 3rem; color:#333;\">- Part 03: Training Data & Feature views</span>\n",
    "\n",
    "<span style=\"font-width:bold; font-size: 1.4rem;\">This is the second part of the quick start series of tutorials about Hopsworks Feature Store. This notebook explains how to read from a feature group and create training dataset within the feature store</span>\n",
    "\n",
    "## 🗒️ In this notebook we will see how to create a training dataset from the feature groups: \n",
    "\n",
    "1. Retrieving Feature Groups.\n",
    "2. Defining Transformation functions.\n",
    "4. Feature View creation.\n",
    "5. Training Dataset with training, validation and test data.\n",
    "\n",
    "![part2](images/02_training-dataset.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b3bcd1",
   "metadata": {},
   "source": [
    "## <span style=\"color:#ff5f27;\"> 🔮 Connecting to Hopsworks Feature Store </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89ad779f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hopsworks\n",
    "\n",
    "project = hopsworks.login()\n",
    "\n",
    "fs = project.get_feature_store() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2eaa049",
   "metadata": {},
   "source": [
    "## <span style=\"color:#ff5f27;\">🪄 Retrieving Feature Groups</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "735a083e",
   "metadata": {},
   "outputs": [],
   "source": [
    "air_quality_fg = fs.get_or_create_feature_group(\n",
    "    name = 'air_quality_fg',\n",
    "    version = 1\n",
    ")\n",
    "weather_fg = fs.get_or_create_feature_group(\n",
    "    name = 'weather_fg',\n",
    "    version = 1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1c1f115",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = air_quality_fg.select_all().join(weather_fg.select_all())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8946c204",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "query.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be427dca",
   "metadata": {},
   "source": [
    "## <span style=\"color:#ff5f27;\">💼 Query Preparation</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc3192d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = air_quality_fg.select_all().join(weather_fg.select_all())\n",
    "\n",
    "query_show = query.show(5)\n",
    "col_names = query_show.columns\n",
    "\n",
    "query_show"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fff1e11",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## <span style=\"color:#ff5f27;\">🧑🏻‍🔬 Transformation functions</span>\n",
    "\n",
    "Hopsworks Feature Store provides functionality to attach transformation functions to training datasets.\n",
    "\n",
    "Hopsworks Feature Store also comes with built-in transformation functions such as `min_max_scaler`, `standard_scaler`, `robust_scaler` and `label_encoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "423c4293",
   "metadata": {},
   "outputs": [],
   "source": [
    "[t_func.name for t_func in fs.get_transformation_functions()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28563812",
   "metadata": {},
   "source": [
    "We can retrieve transformation function we need .\n",
    "\n",
    "To attach transformation function to training dataset provide transformation functions as dict, where key is feature name and value is online transformation function name.\n",
    "\n",
    "Also training dataset must be created from the Query object. Once attached transformation function will be applied on whenever save, insert and get_serving_vector methods are called on training dataset object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad29953e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load transformation functions.\n",
    "standard_scaler = fs.get_transformation_function(name = 'standard_scaler')\n",
    "label_encoder = fs.get_transformation_function(name = 'label_encoder')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aad7489",
   "metadata": {},
   "outputs": [],
   "source": [
    "category_cols = ['city','date','conditions','aqi']\n",
    "\n",
    "mapping_transformers = {col_name:standard_scaler for col_name in col_names if col_name not in category_cols}\n",
    "category_cols = {col_name:label_encoder for col_name in category_cols if col_name not in ['date','aqi']}\n",
    "\n",
    "mapping_transformers.update(category_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d83a1681",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## <span style=\"color:#ff5f27;\"> ⚙️ Feature View Creation </span>\n",
    "\n",
    "`Feature Views` stands between **Feature Groups** and **Training Dataset**. Сombining **Feature Groups** we can create **Feature Views** which store a metadata of our data. Having **Feature Views** we can create **Training Dataset**.\n",
    "\n",
    "The Feature Views allows schema in form of a query with filters, define a model target feature/label and additional transformation functions.\n",
    "\n",
    "In order to create Feature View we can use `FeatureStore.create_feature_view()` method.\n",
    "\n",
    "We can specify next parameters:\n",
    "\n",
    "- `name` - name of a feature group.\n",
    "\n",
    "- `version` - version of a feature group.\n",
    "\n",
    "- `labels`- our target variable.\n",
    "\n",
    "- `transformation_functions` - functions to transform our features.\n",
    "\n",
    "- `query` - query object with data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403df0b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_view = fs.create_feature_view(\n",
    "    name = 'air_quality_fv',\n",
    "    version = 1,\n",
    "    transformation_functions = mapping_transformers,\n",
    "    query = query\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c723c54",
   "metadata": {},
   "source": [
    "For now `Feature View` is saved in Hopsworks and we can retrieve it using `FeatureStore.get_feature_view()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "721c4bcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_view = fs.get_feature_view(\n",
    "    name = 'air_quality_fv',\n",
    "    version = 1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e1187a2",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## <span style=\"color:#ff5f27;\"> 🏋️ Training Dataset Creation</span>\n",
    "\n",
    "In Hopsworks training data is a query where the projection (set of features) is determined by the parent FeatureView with an optional snapshot on disk of the data returned by the query.\n",
    "\n",
    "**Training Dataset  may contain splits such as:** \n",
    "* Training set - the subset of training data used to train a model.\n",
    "* Validation set - the subset of training data used to evaluate hparams when training a model\n",
    "* Test set - the holdout subset of training data used to evaluate a mode\n",
    "\n",
    "To create training dataset we use `FeatureView.create_training_data()` method.\n",
    "\n",
    "Here are some importand things:\n",
    "\n",
    "- It will inherit the name of FeatureView.\n",
    "\n",
    "- The feature store currently supports the following data formats for\n",
    "training datasets: **tfrecord, csv, tsv, parquet, avro, orc**.\n",
    "\n",
    "- We can choose necessary format using **data_format** parameter.\n",
    "\n",
    "- **start_time** and **end_time** in order to filter dataset in specific time range."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f3e7b2",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#ff5f27;\"> ⛳️ Dataset with train, validation and test splits</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "299bd245",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_view.create_training_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d51507c0",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#ff5f27;\"> Next Steps</span>\n",
    "\n",
    "In the next notebook, we will train a model on the Training Dataset we created in this notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}