FeaturePreview/Enterprise_Feature_Store/EFS_Getting_Started_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       Using the Enterprise Feature Store Functions\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Teradata Enterprise Feature Store (EFS) Functions are designed to handle feature management within the Vantage environment. While inspired by the syntax of Feast, Teradata EFS Functions stands out, offering efficiency and robustness in data management and feature handling tailored specifically for the use of Teradata Vantage.  \n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    " Teradata EFS Functions use Teradata Dataframes for Feature management, to the contrary of the pandas dataframe of Feast. With Teradata Dataframes we avoid extracting the data to create or use Features from the Enterprise Feature Store (EFS). The EFS Functions are crafted to empower Data Science teams for effective and streamlined feature management. This notebook will walk you through the capabilities of EFS Functions, demonstrating how it integrates seamlessly with your data models and processes.\n",
    "\n",
    "</p>\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'>\n",
    "<b>Key Concepts of the Enterprise Feature Store (EFS) SDK\n",
    "</b>\n",
    "</p>\n",
    "The Enterprise Feature Store (EFS) SDK is designed with a totally object-oriented approach, focusing on intuitive interaction with feature stores. Central to this design are several core objects: Feature, Entity, DataSource, FeatureGroup. Together, these objects facilitate the efficient management and utilization of features within your data ecosystem, leveraging Teradata Vantage for metadata storage. Here's a closer look at each of these objects and their roles:\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"EFS_key_concepts.png\" alt=\"efs\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<div style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'>\n",
    "<b>Feature\n",
    "</b>\n",
    "\n",
    "A Feature represents a single, distinct piece of data that can be used in machine learning models. Features are the fundamental building blocks of the EFS, designed to encapsulate specific data types, validation rules, and metadata essential for downstream analysis and modeling.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'>\n",
    "<b>Entity\n",
    "</b>\n",
    "</p>\n",
    "<p>\n",
    "An Entity serves as the anchor for one or more Features, grouping them by a common identifier. This shared identifier ensures that features within an entity relate to the same logical unit, such as a customer or transaction. The Entity concept ensures data consistency and simplifies the management of feature relationships.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'>\n",
    "<b>Data Source\n",
    "</b>\n",
    "</p>\n",
    "\n",
    "The DataSource object provides a flexible mapping between the results of a SQL query or DataFrame and Features. It describes how raw data from Teradata Vantage can be transformed into structured features ready for machine learning. This abstraction allows for the separation of data retrieval logic from feature management, promoting modularity and reuse.\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'>\n",
    "<b>Feature Group\n",
    "</b>\n",
    "</p>\n",
    "\n",
    "A FeatureGroup represents a collection of Features that are related by a common Entity and originate from the same DataSource. By grouping features this way, the EFS SDK encourages logical organization of features and simplifies batch operations like updates, retrievals, and analysis.\n",
    "\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'>\n",
    "<b>Repository\n",
    "</b>\n",
    "</p>\n",
    "A Repository is a logical workspace to enable the user to work in their Feature Groups, with the possibility to promote features between repositories. This enables the possibility to have personal repositories (for example, a Lab), team repositories to collaborate and a central production repository.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>What You Will Do in This Notebook</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "This notebook is designed to guide you through a series of practical exercises that demonstrate the use of Teradata's Enterprise Feature Store capabilities. By the end of this tutorial, you will have a comprehensive understanding of how to manage and utilize feature stores for machine learning workflows. Here's what you'll learn:\n",
    "</p>\n",
    "\n",
    "<div style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "<ol>\n",
    "<li>\n",
    "<b>Setup a Feature Store Repository and Grant access on it to users</b>\n",
    "<ul>\n",
    "<li>Learn how to set up a new feature store repository, using Feature Groups, which serves as the foundational environment for storing and managing your data features.</li>\n",
    "<li>Owner of the FeatureStore can grant/revoke read only, write only or read and write authorization to other user(s) </li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "<li>\n",
    "<b>Create and Register objects with FeatureStore</b>\n",
    "<ul>\n",
    "    <li>Discover <span style=\"color: #FF4500\">how to create a Feature</span> from Teradata DataFrame and register the feature with the Teradata Enterprise Feature Store.</li>\n",
    "    <li>Discover <span style=\"color: #FF4500\">how to create an Entity</span> from Teradata DataFrame and register the Entity with the Teradata Enterprise Feature Store.</li>\n",
    "    <li>Discover <span style=\"color: #FF4500\">different ways to create DataSource</span> and register DataSource with the Teradata Enterprise Feature Store.</li>\n",
    "    <li>Discover <span style=\"color: #FF4500\">different ways to create FeatureGroup</span> and register FeatureGroup with the Teradata Enterprise Feature Store.</li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "<li>\n",
    "<b>Searching inside Teradata Enterprise Feature Store</b>\n",
    "<ul>\n",
    "<li>Explore methods to search in Features, Entities, DataSources and FeatureGroups. </li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "\n",
    "<li>\n",
    "<b>Modifying FeatureStore Objects</b>\n",
    "<ul>\n",
    "<li>Explore methods to modify existing features and other objects within your Enterprise Feature Store to adapt to changes in your data or analysis requirements. </li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "<li>\n",
    "<b>Combining multiple FeatureGroups.</b>\n",
    "<ul>\n",
    "<li>Explore a way to combine multiple FeatureGroups to a single FeatureGroup and store the combined FeatureGroup within your Enterprise Feature Store. </li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "\n",
    "<li>\n",
    "<b>Archive and Delete objects in FeatureStore.</b>\n",
    "<ul>\n",
    "<li>Explore method to archive and delete different objects from FeatureStore. </li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "\n",
    "<li>\n",
    "<b>Creating Datasets and historic Datasets for ML models</b>\n",
    "<ul>\n",
    "<li>Teradata EFS approach to get Datasets or historic Datasets to feed your ML Model.</li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "\n",
    "<li>\n",
    "<b>Use Enterprise Feature Store with teradataml anaylytic functions.</b>\n",
    "<ul>\n",
    "<li>Apply Teradata EFS Functions to create suitable for training a machine learning model, ensuring it is clean, well-structured, and aligned with your model's requirements.</li>\n",
    "</ul>\n",
    "</li>\n",
    "\n",
    "\n",
    "<li>\n",
    "<b>Repository Governance</b>\n",
    "<ul>\n",
    "<li>Promote features from one repository to another repository.</li>\n",
    "</ul>\n",
    "</li>\n",
    "</ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>1. Connect to Vantage</b>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install --upgrade teradataml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Enterprise Feature Store is new feature added in teradataml 20.0.0.3 so we are upgrading the installed teradataml version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Note: </b><i>The above statements may need to be uncommented if you run the notebooks on a platform other than ClearScape Analytics Experience that does not have the libraries installed. If you uncomment those installs, be sure to restart the kernel after executing those lines to bring the installed libraries into memory. The simplest way to restart the Kernel is by typing zero zero: <b> 0 0</b></i></p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "from teradataml import (\n",
    "    create_context,\n",
    "    execute_sql,\n",
    "    DataFrame,\n",
    "    DataSource,\n",
    "    Entity,\n",
    "    FeatureGroup,\n",
    "    FeatureStore,\n",
    "    FeatureType,\n",
    "    FeatureStatus,\n",
    "    load_example_data,\n",
    "    remove_context,\n",
    "    Feature,\n",
    "    XGBoost\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=PP_EFS_Getting_Started_Python.ipynb;' UPDATE FOR SESSION; ''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>2. Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "In this tutorial, we will use the <span style=\"background-color: #eee; font-style: italic; \"> load_example_data() </span> function provided by teradataml, which is responsible to load the data to Vantage.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "This will create two tables in Vantage. \n",
    "    <ul>\n",
    "        <li style = 'font-size:16px;font-family:Arial;color:#00233C'>patient_profile</li>\n",
    "        <li style = 'font-size:16px;font-family:Arial;color:#00233C'>medical_readings</li></ul></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_example_data('dataframe', 'patient_profile')\n",
    "load_example_data('dataframe', 'medical_readings')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patient_profile_df = DataFrame('patient_profile')\n",
    "patient_profile_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "medical_readings_df = DataFrame('medical_readings')\n",
    "medical_readings_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>3. Setup a Feature Store Repository and Grant access on it to different users. </b>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's first setup the FeatureStore with repo name as <span style=\"background-color: #eee; font-style: italic; \"> LabRepoOne </span><i>. Please Look at Prerequisite for setting up FeatureStore in teradataml user guide.</i>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before creating Repo, let's check existing FeatureStores.\n",
    "FeatureStore.list_repos()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FeatureStore is not setup for repo LabRepoOne. Let's setup.\n",
    "fs = FeatureStore('LabRepoOne')\n",
    "fs.setup(perm_size='10e8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's verify by listing the repo's.\n",
    "FeatureStore.list_repos()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now we can see the repo we created 'LabRepoOne' </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We can authorise the repos to be used by different users. As in this demo environment we have only one user 'demo_user', we are listing the commands that can be used to give access to other users in actual project environments. <br> Let's Look at the ways to authorize access for FeatureStore to user <span style=\"background-color: #eee; font-style: italic; \"> user1 </span>. <br>\n",
    "<p style = 'font-size:14px;font-family:Courier'><b>Grant Access</b><br>\n",
    "  <code>user='user1'</code> <br>\n",
    " <p style = 'font-size:14px;font-family:Courier;color:#355E3B'>   # Grant read only access to user1. user1 can able to only see the all objects in FeatureStore but cannot modify these objects.\n",
    "<p style = 'font-size:14px;font-family:Courier'><code>fs.grant.read(user)</code><br>\n",
    "  <p style = 'font-size:14px;font-family:Courier;color:#355E3B'> # Grant write only access to user1. user1 can able to modify all objects in FeatureStore but cannot see these objects.\n",
    "<p style = 'font-size:14px;font-family:Courier'><code>fs.grant.write(user)</code><br>   \n",
    "      <p style = 'font-size:14px;font-family:Courier;color:#355E3B'> # Grant read and write to user1. user1 will get full access on all objects of FeatureStore.\n",
    "<p style = 'font-size:14px;font-family:Courier'><code>fs.grant.read_write(user)</code><br>\n",
    "    <p style = 'font-size:14px;font-family:Courier'><b> Revoke Access</b><br>\n",
    "        <p style = 'font-size:14px;font-family:Courier;color:#355E3B'> # Revoke read access from user1 on FeatureStore LabRepoOne.\n",
    "<p style = 'font-size:14px;font-family:Courier'><code>fs.revoke.read(user)</code><br>\n",
    "      <p style = 'font-size:14px;font-family:Courier;color:#355E3B'> # Revoke write access from user1 on FeatureStore LabRepoOne.\n",
    "<p style = 'font-size:14px;font-family:Courier'><code>fs.revoke.write(user)</code><br>\n",
    "          <p style = 'font-size:14px;font-family:Courier;color:#355E3B'># Revoke read and write access from user1 on FeatureStore LabRepoOne.\n",
    "<p style = 'font-size:14px;font-family:Courier'><code>fs.revoke.write(user)</code><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Create and Register objects with FeatureStore </b></p>\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>4.1 Create and Register Feature </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.1.1 Creating a Feature</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating Feature for Column 'age' from Teradata DataFrame 'patient_profile_df'.\n",
    "f1 = Feature(name='PatientAge', \n",
    "             column=patient_profile_df.age, \n",
    "             feature_type=FeatureType.CONTINUOUS, \n",
    "             description=None, \n",
    "             tags=[\"PatientProfile\", \"PatientDetails\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at underlying properties.\n",
    "print(f\"\\033[1mName:\\033[0m {f1.name}\")\n",
    "print(f\"\\033[1mColumn Name:\\033[0m {f1.column_name}\")\n",
    "print(f\"\\033[1mData Type:\\033[0m {f1.data_type}\")\n",
    "print(f\"\\033[1mDescription:\\033[0m {f1.description}\")\n",
    "print(f\"\\033[1mTags:\\033[0m {f1.tags}\")\n",
    "print(f\"\\033[1mStatus:\\033[0m {f1.status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.1.2 Register Feature with FeatureStore `fs` </b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before even register the Feature, let's look at available Features.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FeatureStore.apply() register every object.\n",
    "fs.apply(f1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at available Features again.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>4.2 Create and Register Entity </b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.2.1 Creating an Entity </b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create entity for DataFrame 'patient_profile_df'\n",
    "entity=Entity(name='PatientEntity', columns=patient_profile_df.patient_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at Entity properties.\n",
    "entity.name, entity.columns, entity.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.2.2 Register Entity with FeatureStore `fs` </b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before even registering Entity, let's look at existing Entities.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register the Entity.\n",
    "fs.apply(entity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at existing Entities after registering the Entity.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>4.3 Create and Register DataSource </b></p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>DataSource can be either created from a SQL Query or from Teradata DataFrame.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>DataSource has argument `timestamp_col_name` which accepts the name of Column in DataSource which indicates when the corresponding record is created. This is much helpfull to get historic dataset. </li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.3.1 Creating a DataSource from Teradata DataFrame</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create DataSource from DataFrame `patient_profile_df`.\n",
    "ds = DataSource(name='PatientProfileSource', source=patient_profile_df, timestamp_col_name='record_timestamp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at properties of DataSource.\n",
    "ds.name, ds.source, ds.description, ds.timestamp_col_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.3.2 Creating a DataSource from SQL Query</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create DataSource from DataFrame `patient_profile_df`.\n",
    "ds = DataSource(name='PatientProfileSource', source=\"SELECT * FROM PATIENT_PROFILE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at properties of DataSource.\n",
    "ds.name, ds.source, ds.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.3.3 Register DataSource with FeatureStore `fs`</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before registering let's look at existing DataSources.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register DataSource with repo.\n",
    "fs.apply(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at available DataSources after registration.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>4.4 Create and Register FeatureGroup </b></p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>FeatureGroup can be created using Teradata DataFrame.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>FeatureGroup can be created using SQL Query. </li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>FeatureGroup can be created using objects of Feature, Entity, DataSource.  </li>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.4.1 Creating a FeatureGroup from Teradata DataFrame\n",
    "</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fg = FeatureGroup.from_DataFrame(\n",
    "    name='PatientProfileDF', \n",
    "    entity_columns='patient_id', \n",
    "    df=patient_profile_df, \n",
    "    timestamp_col_name='record_timestamp'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Properties.\n",
    "fg.features, fg.entity, fg.data_source, fg.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.4.2 Creating a FeatureGroup from SQL Query\n",
    "</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fg = FeatureGroup.from_query(\n",
    "    name='PatientProfileQuery', \n",
    "    entity_columns='patient_id', \n",
    "    query=\"select * from patient_profile\", \n",
    "    timestamp_col_name='record_timestamp'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Properties.\n",
    "fg.features, fg.entity, fg.data_source, fg.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.4.3 Creating a FeatureGroup using objects of Feature, Entity and DataSource\n",
    "</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fg = FeatureGroup(name='PatientProfileObjs', features=[f1], entity=entity, data_source=ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Properties.\n",
    "fg.features, fg.entity, fg.data_source, fg.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>4.4.4 Register FeatureGroup with FeatureStore `fs`\n",
    "</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at underlying FeatureGroups first.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Available Features also. Notice: Feature is not associated with any group.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register FeatureGroup with FeatureStore.\n",
    "fs.apply(fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at FeatureGroups after registration.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Available Features again. Notice: Feature is now associated with group.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>5. Searching inside Teradata Enterprise Feature Store </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>How to search for Features: <b>FeatureStore.list_features()</b> returns Teradata DataFrame. All the filter options available on Teradata DataFrame can be used for searching. Look at example below.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>How to search for Entities: <b>FeatureStore.list_entities()</b> returns Teradata DataFrame. All the filter options available on Teradata DataFrame can be used for searching.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>How to search for DataSources: <b>FeatureStore.list_data_sources()</b> returns Teradata DataFrame. All the filter options available on Teradata DataFrame can be used for searching.  </li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>How to search for FeatureGroups: <b>FeatureStore.list_feature_groups()</b> returns Teradata DataFrame. All the filter options available on Teradata DataFrame can be used for searching.   </li>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's first create some more Features and register with repo. Then we can use same for searching.\n",
    "f1=Feature(name='PatientBMI', column=patient_profile_df.bmi)\n",
    "fs.apply(f1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First list the Features.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter the Features registerd at day 18. \n",
    "# Note: One can use all the filter options available on Teradata DataFrame. Look at user guide to look at available filter options.\n",
    "features_df = fs.list_features()\n",
    "features_df[features_df.creation_time.day_of_month()==15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>6. Modifying FeatureStore Objects </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Teradata EFS exposed below API's to get the objects from FeatureStore</p>\n",
    "<ul >\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>FeatureStore.get_feature()</b>to get the <b>Feature</b> object from FeatureStore.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>FeatureStore.get_entity()</b> to get the <b>Entity</b> object from FeatureStore.  </li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>FeatureStore.get_data_source()</b> to get the <b>DataSource</b> object from FeatureStore.   </li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>FeatureStore.get_feature_group()</b> to get the <b>FeatureGroup</b> object from FeatureStore. </li></ul>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Use these API's to get corresponding object, modify the corresponding property, then again register object with repository using <b>FeatureStore.apply()</b>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>6.1 Update description for Feature PatientAge </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature=fs.get_feature('PatientAge')\n",
    "feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update Description and tags.\n",
    "feature.description=\"Patient's age for patient profile.\"\n",
    "feature.tags = ['PatientProfile']\n",
    "fs.apply(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at features again. Look for description column. \n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>6.2 Update description for Entity PatientEntity</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before updating description, let's look at Entities.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Entity from FeatureStore.\n",
    "entity = fs.get_entity('PatientEntity')\n",
    "entity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update Entity description.\n",
    "entity.description = \"Entity for Patient Profile.\"\n",
    "fs.apply(entity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After updating description, let's look at Entities.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>6.3 Update timestamp_col_name for DataSource PatientProfileSource</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before updating time stamp column, let's look at DataSource.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First get the DataSource.\n",
    "data_source = fs.get_data_source('PatientProfileSource')\n",
    "data_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update time stamp column.\n",
    "data_source.timestamp_col_name = 'record_timestamp'\n",
    "fs.apply(data_source)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After updating time stamp column, let's look at DataSource.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>6.4 Update FeatureGroup</b>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'> Note: Updating FeatureGroup will update the underlying Feature(s), DataSource, Entity.</li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before updating description, let's look at FeatureGroup.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before updating description, let's look at DataSources.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get FeatureGroup.\n",
    "fg = fs.get_feature_group('PatientProfileObjs')\n",
    "\n",
    "# Update DataSource description and FeatureGroup description.\n",
    "fg.data_source.description = \"Data Source for Patient Profile.\"\n",
    "fg.description = \"FeatureGroup for Patient Profile.\"\n",
    "\n",
    "# Register FeatureGroup with FeatureStore.\n",
    "fs.apply(fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After updating description, let's look at FeatureGroup.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After updating description, let's look at DataSource.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>6.5 How to add a new Feature or change Entity or DataSource to an Existing FeatureGroup</b>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'> You can always modify FeatureGroup with `FeatureGroup.apply()` method.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'> Note: `FeatureGroup.apply()` will not update details to `repo`. You should do `FeatureStore.apply()` to update repo.</li>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before adding Feature, let's look at available Features.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's add a new Feature for FeatureGroup PatientProfileObjs\n",
    "f2 = fs.get_feature('PatientBMI')\n",
    "# First register the Feature with FeatureGroup.\n",
    "fg.apply(f2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Then, Register FeatureGroup with FeatureStore.\n",
    "fs.apply(fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Features.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>6.6 How to remove a new Feature from an Existing FeatureGroup</b>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'> You can use `FeatureGroup.remove()` method to remove object from FeatureGroup.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'> Note: `FeatureGroup.remove()` will not update details to `repo`. You should do `FeatureStore.apply()` to update repo.</li>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's remove Feature `PatientBMI` from FeatureGroup `PatientProfileObjs`.\n",
    "fg.remove(f2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update FeatureGroup with FeatureStore.\n",
    "fs.apply(fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Features.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at FeatureGroups.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>7. Combining multiple FeatureGroups </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "One can combine multiple FeatureGroups to a single group using `+` operator. Once you combine multiple FeatureGroups, you will again get a new FeatureGroup. The name of new FeatureGroup is combined name of all FeatureGroups. For example, if you are combining FeatureGroups `group1`, `group2`, `group3`, then the new name is `group1_group2_group3` . You can change the name if you are looking to.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>New FeatureGroup will have Features from all the individual FeatureGroups.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>When you are combining multiple FeatureGroups, corresponding `Entity` and `time_stamp_column` should be same for all individual FeatureGroups.</li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's first create individual FeatureGroups first.\n",
    "patient_profile_fg = FeatureGroup.from_DataFrame(\n",
    "    name='PatientProfile', \n",
    "    df=patient_profile_df, \n",
    "    entity_columns='patient_id', \n",
    "    timestamp_col_name='record_timestamp'\n",
    ")\n",
    "medical_readings_fg = FeatureGroup.from_DataFrame(\n",
    "    name='MedicalReadings', \n",
    "    df=medical_readings_df, \n",
    "    entity_columns='patient_id', \n",
    "    timestamp_col_name='record_timestamp'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at Features first for FeatureGroups.\n",
    "patient_profile_fg.features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "medical_readings_fg.features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new FeatureGroup.\n",
    "combined_fg = patient_profile_fg + medical_readings_fg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at new FeatureGroup name.\n",
    "combined_fg.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at combined features.\n",
    "combined_fg.features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Push individual FeatureGroups and also Combined FeatureGroup.\n",
    "fs.apply(patient_profile_fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Push individual FeatureGroups and also Combined FeatureGroup.\n",
    "fs.apply(medical_readings_fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Push individual FeatureGroups and also Combined FeatureGroup.\n",
    "fs.apply(combined_fg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at FeatureGroups.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at DataSources.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Entities.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at Features after pushing all FeatureGroups.\n",
    "fs.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter the features to understand the data more. Note that, Feature `blood_pressure` is mapped to two FeatureGroups.\n",
    "features_df = fs.list_features()\n",
    "features_df = features_df[features_df.name == 'blood_pressure']\n",
    "features_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. Archive and Delete objects in FeatureStore </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Archive and Delete are two different operations and they are not same in FeatureStore.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Archive stages objects instead of removing it completly from FeatureStore. Archived objects will not be part of any further processing.</li>\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "<li> use `FeatureStore.archive_feature()` to archive a Feature. `FeatureStore.list_features(archived=True)` lists archived Features.</li>\n",
    "<li>use `FeatureStore.archive_feature()` to archive a Entity. `FeatureStore.list_entities(archived=True)` lists archived Entities.</li>\n",
    "<li>use `FeatureStore.archive_data_source()` to archive a DataSource. `FeatureStore.list_data_sources(archived=True)` to list archived DataSources.</li>\n",
    "<li>use `FeatureStore.archive_feature_group()` to archive a FeatureGroup. `FeatureStore.list_feature_groups(archived=True)` to list archived FeatureGroups.</li>\n",
    "    </ul></li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Archiving FeatureGroup will archive the corresponding Feature, Entity and DataSource.</li>  \n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>If a Feature is associated with a FeatureGroup, it can not be archived. First the Feature should be removed from FeatureGroup and then archive it.</li> \n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>If an Entity is associated with a FeatureGroup, it can not be archived. First the Entity should be removed from FeatureGroup and then archive it.</li> \n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>If a DataSource is associated with a FeatureGroup, it can not be archived. First the DataSource should be removed from FeatureGroup and then archive it.</li> \n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Delete will remove the archived objects.</li> \n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>use `FeatureStore.delete_feature()` to delete a Feature.</li>\n",
    "    <li>use `FeatureStore.delete_entity()` to delete a Entity.</li>\n",
    "    <li>use `FeatureStore.delete_data_source()` to delete a DataSource.</li>\n",
    "    <li>use `FeatureStore.delete_feature_group()` to delete a FeatureGroup.</li></ul></li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>Deleting FeatureGroup <u><b> will not </b></u> remove corresponding archived Features or archived Entities or archived DataSources. You should delete these with corresponding API's.</li>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>8.1 Archive a Feature. Delete the archived Feature.</b>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's first look at Features which are not associated with any FeatureGroup.\n",
    "features_df = fs.list_features()\n",
    "features_df[features_df.group_name == None]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's archive the Feature `PatientBMI`. Before archiving Feature, let's look at features which are archived.\n",
    "fs.list_features(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Archive it.\n",
    "fs.archive_feature('PatientBMI')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived Features.\n",
    "fs.list_features(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the archived Feature.\n",
    "fs.delete_feature('PatientBMI')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived Features again.\n",
    "fs.list_features(archived=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\"> \n",
    "<b style = 'font-size:18px;font-family:Arial;color:#00233C'>8.2 Archive a FeatureGroup. Delete the archived objects.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before archiving group, let's look at Features for FeatureGroup. \n",
    "features_df[features_df.group_name == 'PatientProfile']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: These Features are mapped to other FeatureGroups also.\n",
    "features_df[(\n",
    "    (features_df.name == 'skin_thickness') | (features_df.name == 'pregnancies') | (features_df.name == 'age') | (features_df.name == 'bmi')\n",
    ")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before archiving group, let's look at DataSources.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived DataSources.\n",
    "fs.list_data_sources(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before archiving group, let's look at Entities.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived Entities.\n",
    "fs.list_entities(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before archiving group, let's look at FeatureGroups. \n",
    "# Notice, FeatureGroup `PatientProfileObjs` is associated with DataSource `PatientProfileSource` and Entity `PatientEntity`\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived FeatureGroups.\n",
    "fs.list_feature_groups(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's archive FeatureGroup `PatientProfileObjs`.\n",
    "fs.archive_feature_group('PatientProfile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at FeatureGroups after archive.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at archived FeatureGroup.\n",
    "fs.list_feature_groups(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After archiving group, let's look at Features for FeatureGroup.\n",
    "features_df[features_df.group_name == 'PatientProfile']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at Features and observe group_name.\n",
    "features_df[(\n",
    "    (features_df.name == 'skin_thickness') | (features_df.name == 'pregnancies') | (features_df.name == 'age') | (features_df.name == 'bmi')\n",
    ")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived Features. No Feature is archived because these Features are mapped to other FeatureGroup also.\n",
    "fs.list_features(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After archiving group, let's look at DataSources. \n",
    "# Notice, DataSource `PatientProfileSource`, which is associated with FeatureGroup `PatientProfileObjs` is also archived.\n",
    "fs.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at archived DataSources.\n",
    "fs.list_data_sources(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After archiving group, let's look at Entities. \n",
    "# Notice, Entity `PatientEntity`, which is associated with FeatureGroup `PatientProfileObjs` is also archived.\n",
    "fs.list_entities()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fs.list_entities(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete archived FeatureGroup. \n",
    "fs.delete_feature_group('PatientProfile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived FeatureGroups after delete.\n",
    "fs.list_feature_groups(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete archived DataSources.\n",
    "fs.delete_data_source('PatientProfile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived DataSources after delete.\n",
    "fs.list_data_sources(archived=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete archived Entities.\n",
    "fs.delete_entity('PatientProfile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at archived Entities after delete.\n",
    "fs.list_entities(archived=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>8.3 Creating Datasets and historic Datasets for ML models </b>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Since FeatureStore stores DataSource also, you can retrive Teradata DataFrame from FeatureStore. <br> `FeatureStore.get_dataset()` get's Teradata DataFrame from FeatureGroup.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at available FeatureGroups first.\n",
    "fs.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get DataSet for FeatureGroup PatientProfile. \n",
    "fs.get_dataset('MedicalReadings')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's get DataSet for combined FeatureGroup. \n",
    "# Interesting point to observe:\n",
    "#     DataSet will have all the combined Features of both FeatureGroups.\n",
    "#     patient_id and record_timestamp will remain as it is.\n",
    "fs.get_dataset('PatientProfile_MedicalReadings')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'> In some cases, you need the historic DataSet to perform ML Model. In such cases, use API `FeatureStore.get_dataset()` and filter the data using filter options. Let's look at an example.</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = fs.get_dataset('PatientProfile_MedicalReadings')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assume you want to feed only week 15 data to your model. \n",
    "week15_df = df[df.record_timestamp.week()==15]\n",
    "week15_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>9. Use Enterprise Feature Store with teradataml analytic functions </b>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>teradataml analytic functions accepts Features as input.\n",
    "<br> Let's look at Diabetes prediction using teradataml analytic function `XGBoost()`.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First get the Dataset.\n",
    "medical_readings_df = fs.get_dataset('MedicalReadings')\n",
    "medical_readings_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split DataSet in to two samples.\n",
    "sampled_df = medical_readings_df.sample(frac=[0.7, 0.3])\n",
    "train_df = sampled_df[sampled_df.sampleid==2]\n",
    "test_df = sampled_df[sampled_df.sampleid==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the FeatureGroup. Notice the Feature `outcome` should be set as label. \n",
    "medical_readings_fg=fs.get_feature_group('MedicalReadings')\n",
    "medical_readings_fg.labels='outcome'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import XGBoost\n",
    "model = XGBoost(data=train_df,\n",
    "                input_columns=medical_readings_fg.features,\n",
    "                response_column = medical_readings_fg.labels,\n",
    "                max_depth=3,\n",
    "                lambda1 = 1000.0,\n",
    "                model_type='Classification',\n",
    "                seed=-1,\n",
    "                shrinkage_factor=0.1,\n",
    "                iter_num=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Score the model using test data.\n",
    "XGBoostPredict_out_1 = model.predict(newdata=test_df,\n",
    "                                     id_column='patient_id',\n",
    "                                     model_type='Classification'\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "XGBoostPredict_out_1.result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>10. Repository Governance </b>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "So far, we have been working in a repo called \"LabRepoOne\", with Teradata EFS Functions, you can manage your Feature Store Repos and \"Promote\" them to work as a production repo.\n",
    "</p> \n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "<b>Note : </b> The Feature Store Functions are not materializing the data of your features into production, this only for the metadata of the feature repo. Make sure that your ETL processes are executed in the production datasources. \n",
    "</p> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, create a new repo to 'ProdRepoOne' to move Features.\n",
    "ProdLabRepoOne = FeatureStore(\"ProdLabRepoOne\")\n",
    "# Setup prod repo if it is not setup.\n",
    "ProdLabRepoOne.setup()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assume you want to promote FeatureGroup 'MedicalReadings' from 'LabRepoOne' to 'ProdLabRepoOne'.\n",
    "# First get the FeatureGroup from LabRepoOne. Then apply it to 'ProdLabRepoOne'.\n",
    "ProdLabRepoOne.apply(fs.get_feature_group('MedicalReadings'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's verify ProdLabRepoOne FeatureGroups.\n",
    "ProdLabRepoOne.list_feature_groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's verify ProdLabRepoOne Features.\n",
    "ProdLabRepoOne.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's verify ProdLabRepoOne DataSources.\n",
    "ProdLabRepoOne.list_data_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's verify ProdLabRepoOne Entities.\n",
    "ProdLabRepoOne.list_entities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>11. Cleanup </b></p>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We need to clean up our work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tables = ['patient_profile','medical_readings']\n",
    "\n",
    "# Loop through the list of tables and execute the drop table command for each table\n",
    "for table in tables:\n",
    "    try:\n",
    "        db_drop_table(table_name=table)\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2024. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}