OGUZ_Ismail-Can_1_notebook_exploratoire_062023.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e91d1e33",
   "metadata": {},
   "source": [
    "<center><font size=\"6\"><b>Anticipez les besoins en consommation de bâtiments</b></font></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bfbbf3d",
   "metadata": {},
   "source": [
    "![Seattle City](Seattle-city.jpeg)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "354f4d65",
   "metadata": {},
   "source": [
    "# Exploration des données de consommation énergétique des bâtiments non résidentiels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6618eaa",
   "metadata": {},
   "source": [
    "# Préambule"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de1dc95",
   "metadata": {},
   "source": [
    "Ce notebook a pour objectif d'explorer les données de consommation énergétique des bâtiments non résidentiels de la ville, ainsi que les données structurelles associées à ces bâtiments. Les relevés minutieux de 2016 effectués par les agents de la ville seront utilisés comme données de référence. L'objectif est de tenter de prédire les émissions de CO2 et la consommation totale d'énergie de bâtiments non destinés à l'habitation pour lesquels ces données n'ont pas encore été mesurées. Nous chercherons également à évaluer l'intérêt de l'\"ENERGY STAR Score\" pour la prédiction d'émissions. Ce notebook comprendra une analyse exploratoire des données, ainsi que des tests de différents modèles de prédiction afin de répondre au mieux à la problématique. Nous ferons également attention au traitement des différentes variables, à la fois pour trouver de nouvelles informations et pour optimiser les performances en appliquant des transformations simples aux variables. Enfin, nous mettrons en place une évaluation rigoureuse des performances de la régression et optimiserons les hyperparamètres et le choix d'algorithmes de ML à l'aide d'une validation croisée.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2c3220b",
   "metadata": {},
   "source": [
    "<a id='sec:introduction'></a>\n",
    "## I. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8152aca1",
   "metadata": {},
   "source": [
    "Ce premier notebook se concentre sur une étape essentielle du projet, à savoir l'analyse exploratoire des données et le prétraitement des données. Avant de plonger dans les différentes méthodes de prédiction, il est primordial de comprendre en profondeur les données de consommation énergétique des bâtiments non résidentiels de la ville de Seattle. Dans ce notebook, nous effectuerons une analyse approfondie des caractéristiques des bâtiments, des variables structurales et des relevés de consommation réalisés en 2016. Nous examinerons également les différentes étapes de prétraitement des données, telles que le traitement des valeurs manquantes et des données aberrantes. Cette étape est cruciale pour garantir des résultats fiables dans les étapes ultérieures de modélisation et de prédiction. Passons maintenant à la table des matières pour explorer en détail le contenu de ce premier notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d21198fa",
   "metadata": {},
   "source": [
    "### Table des Matières\n",
    "* [I. Introduction](#sec:introduction)\n",
    "* [II. Analyse exploratoire des données](#sec:analyse-exploratoire)\n",
    "    * A. Analyse univariée\n",
    "    * B. Analyse bivariée\n",
    "* [III. Prétraitement des données](#sec:pretraitement-donnees)\n",
    "    * A. Traitement des valeurs manquantes\n",
    "    * B. Traitement des données aberrantes\n",
    "* [IV. Sélection de variables](#sec:selection-variables)\n",
    "    * A. Analyse de la corrélation entre les variables\n",
    "    * B. Analyse de la variance des variables\n",
    "    * C. Feature Enginneering \n",
    "* [V. Conclusion](#sec:conc)\n",
    "    * Analyse des corrélations entre les variables après Feature Engineering\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f168659f",
   "metadata": {},
   "source": [
    "<a id='sec:analyse-exploratoire'></a>\n",
    "# II. Analyse exploratoire des données"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51fb6f6c",
   "metadata": {},
   "source": [
    "\n",
    "Dans cette section, nous allons effectuer une analyse exploratoire des données pour mieux comprendre les caractéristiques des bâtiments de Seattle et la consommation d'énergie. Nous allons examiner les données en utilisant des techniques d'analyse univariée et bivariée, ainsi que des visualisations de données pour mettre en évidence les relations entre les variables. L'objectif de cette section est de déterminer les caractéristiques les plus importantes des bâtiments de Seattle qui influencent la consommation d'énergie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c63904c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import missingno as msno\n",
    "import plotly.express as px\n",
    "import plotly.figure_factory as ff\n",
    "from plotly import __version__\n",
    "from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\n",
    "import chart_studio.plotly as py\n",
    "import cufflinks as cf \n",
    "import plotly.graph_objects as go\n",
    "from plotly.subplots import make_subplots\n",
    "import importlib.metadata\n",
    "\n",
    "# For Notebooks\n",
    "init_notebook_mode(connected=True)\n",
    "# For offline use\n",
    "cf.go_offline()\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1064f424",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Modification des affichages de colonnes, lignes et largeurs de colonnes pour avoir un maximum d'information\n",
    "pd.set_option('display.max_columns', 200)\n",
    "pd.set_option('display.max_rows', 1000)\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e5890868",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.8.3\n",
      "pandas - 2.0.2\n",
      "numpy - 1.23.5\n",
      "seaborn - 0.12.1\n",
      "missingno - 0.4.2\n",
      "cufflinks - 0.17.3\n",
      "matplotlib -  3.6.2\n"
     ]
    }
   ],
   "source": [
    "!python --version\n",
    "\n",
    "# Version des librairies utilisées\n",
    "print('\\n'.join(f'{m.__name__} - {m.__version__}' \n",
    "                for m in globals().values() \n",
    "                if getattr(m, '__version__', None)))\n",
    "\n",
    "print(\"matplotlib - \", importlib.metadata.version('matplotlib'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e7b92fd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('2016_Building_Energy_Benchmarking.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7b4b8744",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OSEBuildingID</th>\n",
       "      <th>DataYear</th>\n",
       "      <th>BuildingType</th>\n",
       "      <th>PrimaryPropertyType</th>\n",
       "      <th>PropertyName</th>\n",
       "      <th>Address</th>\n",
       "      <th>City</th>\n",
       "      <th>State</th>\n",
       "      <th>ZipCode</th>\n",
       "      <th>TaxParcelIdentificationNumber</th>\n",
       "      <th>CouncilDistrictCode</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>NumberofBuildings</th>\n",
       "      <th>NumberofFloors</th>\n",
       "      <th>PropertyGFATotal</th>\n",
       "      <th>PropertyGFAParking</th>\n",
       "      <th>PropertyGFABuilding(s)</th>\n",
       "      <th>ListOfAllPropertyUseTypes</th>\n",
       "      <th>LargestPropertyUseType</th>\n",
       "      <th>LargestPropertyUseTypeGFA</th>\n",
       "      <th>SecondLargestPropertyUseType</th>\n",
       "      <th>SecondLargestPropertyUseTypeGFA</th>\n",
       "      <th>ThirdLargestPropertyUseType</th>\n",
       "      <th>ThirdLargestPropertyUseTypeGFA</th>\n",
       "      <th>YearsENERGYSTARCertified</th>\n",
       "      <th>ENERGYSTARScore</th>\n",
       "      <th>SiteEUI(kBtu/sf)</th>\n",
       "      <th>SiteEUIWN(kBtu/sf)</th>\n",
       "      <th>SourceEUI(kBtu/sf)</th>\n",
       "      <th>SourceEUIWN(kBtu/sf)</th>\n",
       "      <th>SiteEnergyUse(kBtu)</th>\n",
       "      <th>SiteEnergyUseWN(kBtu)</th>\n",
       "      <th>SteamUse(kBtu)</th>\n",
       "      <th>Electricity(kWh)</th>\n",
       "      <th>Electricity(kBtu)</th>\n",
       "      <th>NaturalGas(therms)</th>\n",
       "      <th>NaturalGas(kBtu)</th>\n",
       "      <th>DefaultData</th>\n",
       "      <th>Comments</th>\n",
       "      <th>ComplianceStatus</th>\n",
       "      <th>Outlier</th>\n",
       "      <th>TotalGHGEmissions</th>\n",
       "      <th>GHGEmissionsIntensity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Mayflower park hotel</td>\n",
       "      <td>405 Olive way</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0659000030</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61220</td>\n",
       "      <td>-122.33799</td>\n",
       "      <td>1927</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12</td>\n",
       "      <td>88434</td>\n",
       "      <td>0</td>\n",
       "      <td>88434</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>88434.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>81.699997</td>\n",
       "      <td>84.300003</td>\n",
       "      <td>182.500000</td>\n",
       "      <td>189.000000</td>\n",
       "      <td>7226362.5</td>\n",
       "      <td>7456910.0</td>\n",
       "      <td>2.003882e+06</td>\n",
       "      <td>1.156514e+06</td>\n",
       "      <td>3946027.0</td>\n",
       "      <td>12764.529300</td>\n",
       "      <td>1276453.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>249.98</td>\n",
       "      <td>2.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Paramount Hotel</td>\n",
       "      <td>724 Pine street</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0659000220</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61317</td>\n",
       "      <td>-122.33393</td>\n",
       "      <td>1996</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11</td>\n",
       "      <td>103566</td>\n",
       "      <td>15064</td>\n",
       "      <td>88502</td>\n",
       "      <td>Hotel, Parking, Restaurant</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>83880.0</td>\n",
       "      <td>Parking</td>\n",
       "      <td>15064.0</td>\n",
       "      <td>Restaurant</td>\n",
       "      <td>4622.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>61.0</td>\n",
       "      <td>94.800003</td>\n",
       "      <td>97.900002</td>\n",
       "      <td>176.100006</td>\n",
       "      <td>179.399994</td>\n",
       "      <td>8387933.0</td>\n",
       "      <td>8664479.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>9.504252e+05</td>\n",
       "      <td>3242851.0</td>\n",
       "      <td>51450.816410</td>\n",
       "      <td>5145082.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>295.86</td>\n",
       "      <td>2.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>5673-The Westin Seattle</td>\n",
       "      <td>1900 5th Avenue</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0659000475</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61393</td>\n",
       "      <td>-122.33810</td>\n",
       "      <td>1969</td>\n",
       "      <td>1.0</td>\n",
       "      <td>41</td>\n",
       "      <td>956110</td>\n",
       "      <td>196718</td>\n",
       "      <td>759392</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>756493.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>43.0</td>\n",
       "      <td>96.000000</td>\n",
       "      <td>97.699997</td>\n",
       "      <td>241.899994</td>\n",
       "      <td>244.100006</td>\n",
       "      <td>72587024.0</td>\n",
       "      <td>73937112.0</td>\n",
       "      <td>2.156655e+07</td>\n",
       "      <td>1.451544e+07</td>\n",
       "      <td>49526664.0</td>\n",
       "      <td>14938.000000</td>\n",
       "      <td>1493800.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2089.28</td>\n",
       "      <td>2.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>HOTEL MAX</td>\n",
       "      <td>620 STEWART ST</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0659000640</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61412</td>\n",
       "      <td>-122.33664</td>\n",
       "      <td>1926</td>\n",
       "      <td>1.0</td>\n",
       "      <td>10</td>\n",
       "      <td>61320</td>\n",
       "      <td>0</td>\n",
       "      <td>61320</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>61320.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>56.0</td>\n",
       "      <td>110.800003</td>\n",
       "      <td>113.300003</td>\n",
       "      <td>216.199997</td>\n",
       "      <td>224.000000</td>\n",
       "      <td>6794584.0</td>\n",
       "      <td>6946800.5</td>\n",
       "      <td>2.214446e+06</td>\n",
       "      <td>8.115253e+05</td>\n",
       "      <td>2768924.0</td>\n",
       "      <td>18112.130860</td>\n",
       "      <td>1811213.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>286.43</td>\n",
       "      <td>4.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>WARWICK SEATTLE HOTEL (ID8)</td>\n",
       "      <td>401 LENORA ST</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98121.0</td>\n",
       "      <td>0659000970</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61375</td>\n",
       "      <td>-122.34047</td>\n",
       "      <td>1980</td>\n",
       "      <td>1.0</td>\n",
       "      <td>18</td>\n",
       "      <td>175580</td>\n",
       "      <td>62000</td>\n",
       "      <td>113580</td>\n",
       "      <td>Hotel, Parking, Swimming Pool</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>123445.0</td>\n",
       "      <td>Parking</td>\n",
       "      <td>68009.0</td>\n",
       "      <td>Swimming Pool</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>75.0</td>\n",
       "      <td>114.800003</td>\n",
       "      <td>118.699997</td>\n",
       "      <td>211.399994</td>\n",
       "      <td>215.600006</td>\n",
       "      <td>14172606.0</td>\n",
       "      <td>14656503.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.573449e+06</td>\n",
       "      <td>5368607.0</td>\n",
       "      <td>88039.984380</td>\n",
       "      <td>8803998.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>505.01</td>\n",
       "      <td>2.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9</td>\n",
       "      <td>2016</td>\n",
       "      <td>Nonresidential COS</td>\n",
       "      <td>Other</td>\n",
       "      <td>West Precinct</td>\n",
       "      <td>810 Virginia St</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0660000560</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61623</td>\n",
       "      <td>-122.33657</td>\n",
       "      <td>1999</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>97288</td>\n",
       "      <td>37198</td>\n",
       "      <td>60090</td>\n",
       "      <td>Police Station</td>\n",
       "      <td>Police Station</td>\n",
       "      <td>88830.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>136.100006</td>\n",
       "      <td>141.600006</td>\n",
       "      <td>316.299988</td>\n",
       "      <td>320.500000</td>\n",
       "      <td>12086616.0</td>\n",
       "      <td>12581712.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.160444e+06</td>\n",
       "      <td>7371434.0</td>\n",
       "      <td>47151.816410</td>\n",
       "      <td>4715182.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>301.81</td>\n",
       "      <td>3.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>10</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Camlin</td>\n",
       "      <td>1619 9th Avenue</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0660000825</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61390</td>\n",
       "      <td>-122.33283</td>\n",
       "      <td>1926</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11</td>\n",
       "      <td>83008</td>\n",
       "      <td>0</td>\n",
       "      <td>83008</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>81352.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27.0</td>\n",
       "      <td>70.800003</td>\n",
       "      <td>74.500000</td>\n",
       "      <td>146.600006</td>\n",
       "      <td>154.699997</td>\n",
       "      <td>5758795.0</td>\n",
       "      <td>6062767.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>8.239199e+05</td>\n",
       "      <td>2811215.0</td>\n",
       "      <td>29475.800780</td>\n",
       "      <td>2947580.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>176.14</td>\n",
       "      <td>2.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Other</td>\n",
       "      <td>Paramount Theatre</td>\n",
       "      <td>911 Pine St</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98101.0</td>\n",
       "      <td>0660000955</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.61327</td>\n",
       "      <td>-122.33136</td>\n",
       "      <td>1926</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "      <td>102761</td>\n",
       "      <td>0</td>\n",
       "      <td>102761</td>\n",
       "      <td>Other - Entertainment/Public Assembly</td>\n",
       "      <td>Other - Entertainment/Public Assembly</td>\n",
       "      <td>102761.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>61.299999</td>\n",
       "      <td>68.800003</td>\n",
       "      <td>141.699997</td>\n",
       "      <td>152.300003</td>\n",
       "      <td>6298131.5</td>\n",
       "      <td>7067881.5</td>\n",
       "      <td>2.276286e+06</td>\n",
       "      <td>1.065843e+06</td>\n",
       "      <td>3636655.0</td>\n",
       "      <td>3851.890137</td>\n",
       "      <td>385189.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>221.51</td>\n",
       "      <td>2.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>12</td>\n",
       "      <td>2016</td>\n",
       "      <td>NonResidential</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>311wh-Pioneer Square</td>\n",
       "      <td>612 2nd Ave</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98104.0</td>\n",
       "      <td>0939000080</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.60294</td>\n",
       "      <td>-122.33263</td>\n",
       "      <td>1904</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>163984</td>\n",
       "      <td>0</td>\n",
       "      <td>163984</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>Hotel</td>\n",
       "      <td>163984.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>43.0</td>\n",
       "      <td>83.699997</td>\n",
       "      <td>86.599998</td>\n",
       "      <td>180.899994</td>\n",
       "      <td>187.199997</td>\n",
       "      <td>13723820.0</td>\n",
       "      <td>14194054.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.138898e+06</td>\n",
       "      <td>7297919.0</td>\n",
       "      <td>64259.000000</td>\n",
       "      <td>6425900.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>392.16</td>\n",
       "      <td>2.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>13</td>\n",
       "      <td>2016</td>\n",
       "      <td>Multifamily MR (5-9)</td>\n",
       "      <td>Mid-Rise Multifamily</td>\n",
       "      <td>Lyon Building</td>\n",
       "      <td>607 - 3rd Ave.</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>WA</td>\n",
       "      <td>98104.0</td>\n",
       "      <td>0939000105</td>\n",
       "      <td>7</td>\n",
       "      <td>DOWNTOWN</td>\n",
       "      <td>47.60284</td>\n",
       "      <td>-122.33184</td>\n",
       "      <td>1910</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6</td>\n",
       "      <td>63712</td>\n",
       "      <td>1496</td>\n",
       "      <td>62216</td>\n",
       "      <td>Multifamily Housing</td>\n",
       "      <td>Multifamily Housing</td>\n",
       "      <td>56132.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>81.500000</td>\n",
       "      <td>85.599998</td>\n",
       "      <td>182.699997</td>\n",
       "      <td>187.399994</td>\n",
       "      <td>4573777.0</td>\n",
       "      <td>4807679.5</td>\n",
       "      <td>1.039735e+06</td>\n",
       "      <td>7.420912e+05</td>\n",
       "      <td>2532015.0</td>\n",
       "      <td>10020.259770</td>\n",
       "      <td>1002026.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Compliant</td>\n",
       "      <td>NaN</td>\n",
       "      <td>151.12</td>\n",
       "      <td>2.37</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   OSEBuildingID  DataYear          BuildingType   PrimaryPropertyType  \\\n",
       "0              1      2016        NonResidential                 Hotel   \n",
       "1              2      2016        NonResidential                 Hotel   \n",
       "2              3      2016        NonResidential                 Hotel   \n",
       "3              5      2016        NonResidential                 Hotel   \n",
       "4              8      2016        NonResidential                 Hotel   \n",
       "5              9      2016    Nonresidential COS                 Other   \n",
       "6             10      2016        NonResidential                 Hotel   \n",
       "7             11      2016        NonResidential                 Other   \n",
       "8             12      2016        NonResidential                 Hotel   \n",
       "9             13      2016  Multifamily MR (5-9)  Mid-Rise Multifamily   \n",
       "\n",
       "                  PropertyName          Address     City State  ZipCode  \\\n",
       "0         Mayflower park hotel    405 Olive way  Seattle    WA  98101.0   \n",
       "1              Paramount Hotel  724 Pine street  Seattle    WA  98101.0   \n",
       "2      5673-The Westin Seattle  1900 5th Avenue  Seattle    WA  98101.0   \n",
       "3                    HOTEL MAX   620 STEWART ST  Seattle    WA  98101.0   \n",
       "4  WARWICK SEATTLE HOTEL (ID8)    401 LENORA ST  Seattle    WA  98121.0   \n",
       "5                West Precinct  810 Virginia St  Seattle    WA  98101.0   \n",
       "6                       Camlin  1619 9th Avenue  Seattle    WA  98101.0   \n",
       "7            Paramount Theatre      911 Pine St  Seattle    WA  98101.0   \n",
       "8         311wh-Pioneer Square      612 2nd Ave  Seattle    WA  98104.0   \n",
       "9                Lyon Building   607 - 3rd Ave.  Seattle    WA  98104.0   \n",
       "\n",
       "  TaxParcelIdentificationNumber  CouncilDistrictCode Neighborhood  Latitude  \\\n",
       "0                    0659000030                    7     DOWNTOWN  47.61220   \n",
       "1                    0659000220                    7     DOWNTOWN  47.61317   \n",
       "2                    0659000475                    7     DOWNTOWN  47.61393   \n",
       "3                    0659000640                    7     DOWNTOWN  47.61412   \n",
       "4                    0659000970                    7     DOWNTOWN  47.61375   \n",
       "5                    0660000560                    7     DOWNTOWN  47.61623   \n",
       "6                    0660000825                    7     DOWNTOWN  47.61390   \n",
       "7                    0660000955                    7     DOWNTOWN  47.61327   \n",
       "8                    0939000080                    7     DOWNTOWN  47.60294   \n",
       "9                    0939000105                    7     DOWNTOWN  47.60284   \n",
       "\n",
       "   Longitude  YearBuilt  NumberofBuildings  NumberofFloors  PropertyGFATotal  \\\n",
       "0 -122.33799       1927                1.0              12             88434   \n",
       "1 -122.33393       1996                1.0              11            103566   \n",
       "2 -122.33810       1969                1.0              41            956110   \n",
       "3 -122.33664       1926                1.0              10             61320   \n",
       "4 -122.34047       1980                1.0              18            175580   \n",
       "5 -122.33657       1999                1.0               2             97288   \n",
       "6 -122.33283       1926                1.0              11             83008   \n",
       "7 -122.33136       1926                1.0               8            102761   \n",
       "8 -122.33263       1904                1.0              15            163984   \n",
       "9 -122.33184       1910                1.0               6             63712   \n",
       "\n",
       "   PropertyGFAParking  PropertyGFABuilding(s)  \\\n",
       "0                   0                   88434   \n",
       "1               15064                   88502   \n",
       "2              196718                  759392   \n",
       "3                   0                   61320   \n",
       "4               62000                  113580   \n",
       "5               37198                   60090   \n",
       "6                   0                   83008   \n",
       "7                   0                  102761   \n",
       "8                   0                  163984   \n",
       "9                1496                   62216   \n",
       "\n",
       "               ListOfAllPropertyUseTypes  \\\n",
       "0                                  Hotel   \n",
       "1             Hotel, Parking, Restaurant   \n",
       "2                                  Hotel   \n",
       "3                                  Hotel   \n",
       "4          Hotel, Parking, Swimming Pool   \n",
       "5                         Police Station   \n",
       "6                                  Hotel   \n",
       "7  Other - Entertainment/Public Assembly   \n",
       "8                                  Hotel   \n",
       "9                    Multifamily Housing   \n",
       "\n",
       "                  LargestPropertyUseType  LargestPropertyUseTypeGFA  \\\n",
       "0                                  Hotel                    88434.0   \n",
       "1                                  Hotel                    83880.0   \n",
       "2                                  Hotel                   756493.0   \n",
       "3                                  Hotel                    61320.0   \n",
       "4                                  Hotel                   123445.0   \n",
       "5                         Police Station                    88830.0   \n",
       "6                                  Hotel                    81352.0   \n",
       "7  Other - Entertainment/Public Assembly                   102761.0   \n",
       "8                                  Hotel                   163984.0   \n",
       "9                    Multifamily Housing                    56132.0   \n",
       "\n",
       "  SecondLargestPropertyUseType  SecondLargestPropertyUseTypeGFA  \\\n",
       "0                          NaN                              NaN   \n",
       "1                      Parking                          15064.0   \n",
       "2                          NaN                              NaN   \n",
       "3                          NaN                              NaN   \n",
       "4                      Parking                          68009.0   \n",
       "5                          NaN                              NaN   \n",
       "6                          NaN                              NaN   \n",
       "7                          NaN                              NaN   \n",
       "8                          NaN                              NaN   \n",
       "9                          NaN                              NaN   \n",
       "\n",
       "  ThirdLargestPropertyUseType  ThirdLargestPropertyUseTypeGFA  \\\n",
       "0                         NaN                             NaN   \n",
       "1                  Restaurant                          4622.0   \n",
       "2                         NaN                             NaN   \n",
       "3                         NaN                             NaN   \n",
       "4               Swimming Pool                             0.0   \n",
       "5                         NaN                             NaN   \n",
       "6                         NaN                             NaN   \n",
       "7                         NaN                             NaN   \n",
       "8                         NaN                             NaN   \n",
       "9                         NaN                             NaN   \n",
       "\n",
       "  YearsENERGYSTARCertified  ENERGYSTARScore  SiteEUI(kBtu/sf)  \\\n",
       "0                      NaN             60.0         81.699997   \n",
       "1                      NaN             61.0         94.800003   \n",
       "2                      NaN             43.0         96.000000   \n",
       "3                      NaN             56.0        110.800003   \n",
       "4                      NaN             75.0        114.800003   \n",
       "5                      NaN              NaN        136.100006   \n",
       "6                      NaN             27.0         70.800003   \n",
       "7                      NaN              NaN         61.299999   \n",
       "8                      NaN             43.0         83.699997   \n",
       "9                      NaN              1.0         81.500000   \n",
       "\n",
       "   SiteEUIWN(kBtu/sf)  SourceEUI(kBtu/sf)  SourceEUIWN(kBtu/sf)  \\\n",
       "0           84.300003          182.500000            189.000000   \n",
       "1           97.900002          176.100006            179.399994   \n",
       "2           97.699997          241.899994            244.100006   \n",
       "3          113.300003          216.199997            224.000000   \n",
       "4          118.699997          211.399994            215.600006   \n",
       "5          141.600006          316.299988            320.500000   \n",
       "6           74.500000          146.600006            154.699997   \n",
       "7           68.800003          141.699997            152.300003   \n",
       "8           86.599998          180.899994            187.199997   \n",
       "9           85.599998          182.699997            187.399994   \n",
       "\n",
       "   SiteEnergyUse(kBtu)  SiteEnergyUseWN(kBtu)  SteamUse(kBtu)  \\\n",
       "0            7226362.5              7456910.0    2.003882e+06   \n",
       "1            8387933.0              8664479.0    0.000000e+00   \n",
       "2           72587024.0             73937112.0    2.156655e+07   \n",
       "3            6794584.0              6946800.5    2.214446e+06   \n",
       "4           14172606.0             14656503.0    0.000000e+00   \n",
       "5           12086616.0             12581712.0    0.000000e+00   \n",
       "6            5758795.0              6062767.5    0.000000e+00   \n",
       "7            6298131.5              7067881.5    2.276286e+06   \n",
       "8           13723820.0             14194054.0    0.000000e+00   \n",
       "9            4573777.0              4807679.5    1.039735e+06   \n",
       "\n",
       "   Electricity(kWh)  Electricity(kBtu)  NaturalGas(therms)  NaturalGas(kBtu)  \\\n",
       "0      1.156514e+06          3946027.0        12764.529300         1276453.0   \n",
       "1      9.504252e+05          3242851.0        51450.816410         5145082.0   \n",
       "2      1.451544e+07         49526664.0        14938.000000         1493800.0   \n",
       "3      8.115253e+05          2768924.0        18112.130860         1811213.0   \n",
       "4      1.573449e+06          5368607.0        88039.984380         8803998.0   \n",
       "5      2.160444e+06          7371434.0        47151.816410         4715182.0   \n",
       "6      8.239199e+05          2811215.0        29475.800780         2947580.0   \n",
       "7      1.065843e+06          3636655.0         3851.890137          385189.0   \n",
       "8      2.138898e+06          7297919.0        64259.000000         6425900.0   \n",
       "9      7.420912e+05          2532015.0        10020.259770         1002026.0   \n",
       "\n",
       "   DefaultData  Comments ComplianceStatus Outlier  TotalGHGEmissions  \\\n",
       "0        False       NaN        Compliant     NaN             249.98   \n",
       "1        False       NaN        Compliant     NaN             295.86   \n",
       "2        False       NaN        Compliant     NaN            2089.28   \n",
       "3        False       NaN        Compliant     NaN             286.43   \n",
       "4        False       NaN        Compliant     NaN             505.01   \n",
       "5        False       NaN        Compliant     NaN             301.81   \n",
       "6        False       NaN        Compliant     NaN             176.14   \n",
       "7        False       NaN        Compliant     NaN             221.51   \n",
       "8        False       NaN        Compliant     NaN             392.16   \n",
       "9        False       NaN        Compliant     NaN             151.12   \n",
       "\n",
       "   GHGEmissionsIntensity  \n",
       "0                   2.83  \n",
       "1                   2.86  \n",
       "2                   2.19  \n",
       "3                   4.67  \n",
       "4                   2.88  \n",
       "5                   3.10  \n",
       "6                   2.12  \n",
       "7                   2.16  \n",
       "8                   2.39  \n",
       "9                   2.37  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6a06377d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['OSEBuildingID',\n",
       " 'DataYear',\n",
       " 'BuildingType',\n",
       " 'PrimaryPropertyType',\n",
       " 'PropertyName',\n",
       " 'Address',\n",
       " 'City',\n",
       " 'State',\n",
       " 'ZipCode',\n",
       " 'TaxParcelIdentificationNumber',\n",
       " 'CouncilDistrictCode',\n",
       " 'Neighborhood',\n",
       " 'Latitude',\n",
       " 'Longitude',\n",
       " 'YearBuilt',\n",
       " 'NumberofBuildings',\n",
       " 'NumberofFloors',\n",
       " 'PropertyGFATotal',\n",
       " 'PropertyGFAParking',\n",
       " 'PropertyGFABuilding(s)',\n",
       " 'ListOfAllPropertyUseTypes',\n",
       " 'LargestPropertyUseType',\n",
       " 'LargestPropertyUseTypeGFA',\n",
       " 'SecondLargestPropertyUseType',\n",
       " 'SecondLargestPropertyUseTypeGFA',\n",
       " 'ThirdLargestPropertyUseType',\n",
       " 'ThirdLargestPropertyUseTypeGFA',\n",
       " 'YearsENERGYSTARCertified',\n",
       " 'ENERGYSTARScore',\n",
       " 'SiteEUI(kBtu/sf)',\n",
       " 'SiteEUIWN(kBtu/sf)',\n",
       " 'SourceEUI(kBtu/sf)',\n",
       " 'SourceEUIWN(kBtu/sf)',\n",
       " 'SiteEnergyUse(kBtu)',\n",
       " 'SiteEnergyUseWN(kBtu)',\n",
       " 'SteamUse(kBtu)',\n",
       " 'Electricity(kWh)',\n",
       " 'Electricity(kBtu)',\n",
       " 'NaturalGas(therms)',\n",
       " 'NaturalGas(kBtu)',\n",
       " 'DefaultData',\n",
       " 'Comments',\n",
       " 'ComplianceStatus',\n",
       " 'Outlier',\n",
       " 'TotalGHGEmissions',\n",
       " 'GHGEmissionsIntensity']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3b281c",
   "metadata": {},
   "source": [
    "__*OSE Building ID:*__ identifiant unique de chaque bâtiment \n",
    "\n",
    "__*Data Year :*__ année des données\n",
    "\n",
    "__*Building Type :*__ type de bâtiment (commercial, multifamilial, etc.)\n",
    " \n",
    "__*Primary Property Type :*__ type de propriété principale (bureau, hôtel, etc.)\n",
    "\n",
    "__*Property Name :*__ nom du bâtiment\n",
    "\n",
    "__*Address :*__ adresse du bâtiment\n",
    "\n",
    "__*City :*__ ville où se trouve le bâtiment\n",
    "\n",
    "__*State :*__ état où se trouve le bâtiment\n",
    "\n",
    "__*ZipCode :*__ code postal du bâtiment\n",
    "\n",
    "__*Tax Parcel Identification Number :*__ numéro d'identification fiscale du bâtiment\n",
    "\n",
    "__*Council District Code :*__ code du district municipal où se trouve le bâtiment\n",
    "\n",
    "__*Neighborhood :*__ quartier où se trouve le bâtiment\n",
    "\n",
    "__*Latitude :*__ latitude géographique du bâtiment\n",
    "\n",
    "__*Longitude :*__ longitude géographique du bâtiment\n",
    "\n",
    "__*YearBuilt :*__ année de construction du bâtiment\n",
    "\n",
    "__*Number of Buildings :*__ nombre de bâtiments sur le site\n",
    "\n",
    "__*Number of Floors :*__ nombre d'étages dans le bâtiment\n",
    "\n",
    "__*Property GFA Total :*__ superficie totale du bâtiment (en pieds carrés)\n",
    "\n",
    "__*Property GFA Parking :*__ superficie de stationnement du bâtiment (en pieds carrés)\n",
    "\n",
    "__*Property GFA Building(s) :*__ superficie du ou des bâtiments (en pieds carrés)\n",
    "\n",
    "__*List Of All Property Use Types :*__ liste de tous les types d'utilisation de la propriété\n",
    "\n",
    "__*Largest Property Use Type :*__ le plus grand type d'utilisation de la propriété (par superficie)\n",
    "\n",
    "__*Largest Property Use Type GFA :*__ superficie de l'utilisation principale de la propriété (en pieds carrés)\n",
    "\n",
    "__*Second Largest Property Use Type :*__ deuxième plus grand type d'utilisation de la propriété (par superficie)\n",
    "\n",
    "__*Second Largest Property Use Type GFA :*__ superficie de la deuxième utilisation la plus importante de la propriété (en pieds carrés)\n",
    "\n",
    "__*Third Largest Property Use Type :*__ troisième plus grand type d'utilisation de la propriété (par superficie)\n",
    "\n",
    "__*Third Largest Property Use Type GFA :*__ superficie de la troisième utilisation la plus importante de la propriété (en pieds carrés)\n",
    "\n",
    "__*Years ENERGY STAR Certified :*__ nombre d'années où le bâtiment a été certifié ENERGY STAR\n",
    "\n",
    "__*ENERGY STAR Score :*__ score ENERGY STAR pour le bâtiment\n",
    "\n",
    "__*Site EUI (kBtu/sf) :*__ indicateur de performance énergétique du site (énergie utilisée par pied carré de surface)\n",
    "\n",
    "__*Site EUIWN (kBtu/sf) :*__ indicateur de performance énergétique du site ajusté en fonction des conditions météorologiques (énergie utilisée par pied carré de surface)\n",
    "\n",
    "__*Source EUI (kBtu/sf) :*__ indicateur de performance énergétique de la source (énergie utilisée par pied carré de surface)\n",
    "\n",
    "__*Source EUIWN (kBtu/sf) :*__ indicateur de performance énergétique de la source ajusté en fonction des conditions météorologiques (énergie utilisée par pied carré de surface)\n",
    "\n",
    "__*Site Energy Use (kBtu) :*__ consommation d'énergie du site (en milliers de British Thermal Units)\n",
    "\n",
    "__*Site Energy Use WN (kBtu) :*__ consommation d'énergie du site ajustée en fonction des conditions météorologiques (en milliers de British Thermal Units)\n",
    "\n",
    "__*Electricity (kWh) :*__ consommation d'électricité (en kilowattheures)\n",
    "\n",
    "__*Electricity (kBtu) :*__ consommation d'électricité (en milliers de British Thermal Units)\n",
    "\n",
    "__*Natural Gas (therms) :*__ consommation de gaz naturel en thermies\n",
    "\n",
    "__*Natural Gas (kBtu) :*__ consommation de gaz naturel (en milliers de British Thermal Units)\n",
    "\n",
    "__*Default Data :*__ indication si les données sont les données par défaut (True/False)\n",
    "\n",
    "__*Comments :*__ commentaires sur les données\n",
    "\n",
    "__*Compliance Status :*__ état de conformité de l'immeuble avec la réglementation de benchmarking de l'énergie de Seattle\n",
    "\n",
    "__*Outlier :*__ indication si les données sont des valeurs aberrantes (True/False)\n",
    "\n",
    "__*Total GHG Emissions :*__ total des émissions de gaz à effet de serre en kilogrammes d'équivalent dioxyde de carbone (CO2e)\n",
    "\n",
    "__*GHG Emissions Intensity :*__ intensité des émissions de gaz à effet de serre en kilogrammes d'équivalent dioxyde de carbone (CO2e) par pied carré (unité de mesure de superficie)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "341d8a4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3376, 46)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "067cdd5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total de valeurs manquantes</th>\n",
       "      <th>Pourcentage de valeurs manquantes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Comments</th>\n",
       "      <td>3376</td>\n",
       "      <td>100.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Outlier</th>\n",
       "      <td>3344</td>\n",
       "      <td>99.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsENERGYSTARCertified</th>\n",
       "      <td>3257</td>\n",
       "      <td>96.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ThirdLargestPropertyUseTypeGFA</th>\n",
       "      <td>2780</td>\n",
       "      <td>82.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ThirdLargestPropertyUseType</th>\n",
       "      <td>2780</td>\n",
       "      <td>82.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SecondLargestPropertyUseType</th>\n",
       "      <td>1697</td>\n",
       "      <td>50.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SecondLargestPropertyUseTypeGFA</th>\n",
       "      <td>1697</td>\n",
       "      <td>50.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENERGYSTARScore</th>\n",
       "      <td>843</td>\n",
       "      <td>24.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LargestPropertyUseType</th>\n",
       "      <td>20</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LargestPropertyUseTypeGFA</th>\n",
       "      <td>20</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ZipCode</th>\n",
       "      <td>16</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SteamUse(kBtu)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotalGHGEmissions</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NaturalGas(kBtu)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NaturalGas(therms)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Electricity(kBtu)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Electricity(kWh)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SourceEUI(kBtu/sf)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SourceEUIWN(kBtu/sf)</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ListOfAllPropertyUseTypes</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GHGEmissionsIntensity</th>\n",
       "      <td>9</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NumberofBuildings</th>\n",
       "      <td>8</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SiteEUI(kBtu/sf)</th>\n",
       "      <td>7</td>\n",
       "      <td>0.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SiteEnergyUseWN(kBtu)</th>\n",
       "      <td>6</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SiteEUIWN(kBtu/sf)</th>\n",
       "      <td>6</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SiteEnergyUse(kBtu)</th>\n",
       "      <td>5</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Total de valeurs manquantes  \\\n",
       "Comments                                                3376   \n",
       "Outlier                                                 3344   \n",
       "YearsENERGYSTARCertified                                3257   \n",
       "ThirdLargestPropertyUseTypeGFA                          2780   \n",
       "ThirdLargestPropertyUseType                             2780   \n",
       "SecondLargestPropertyUseType                            1697   \n",
       "SecondLargestPropertyUseTypeGFA                         1697   \n",
       "ENERGYSTARScore                                          843   \n",
       "LargestPropertyUseType                                    20   \n",
       "LargestPropertyUseTypeGFA                                 20   \n",
       "ZipCode                                                   16   \n",
       "SteamUse(kBtu)                                             9   \n",
       "TotalGHGEmissions                                          9   \n",
       "NaturalGas(kBtu)                                           9   \n",
       "NaturalGas(therms)                                         9   \n",
       "Electricity(kBtu)                                          9   \n",
       "Electricity(kWh)                                           9   \n",
       "SourceEUI(kBtu/sf)                                         9   \n",
       "SourceEUIWN(kBtu/sf)                                       9   \n",
       "ListOfAllPropertyUseTypes                                  9   \n",
       "GHGEmissionsIntensity                                      9   \n",
       "NumberofBuildings                                          8   \n",
       "SiteEUI(kBtu/sf)                                           7   \n",
       "SiteEnergyUseWN(kBtu)                                      6   \n",
       "SiteEUIWN(kBtu/sf)                                         6   \n",
       "SiteEnergyUse(kBtu)                                        5   \n",
       "\n",
       "                                 Pourcentage de valeurs manquantes  \n",
       "Comments                                                    100.00  \n",
       "Outlier                                                      99.05  \n",
       "YearsENERGYSTARCertified                                     96.48  \n",
       "ThirdLargestPropertyUseTypeGFA                               82.35  \n",
       "ThirdLargestPropertyUseType                                  82.35  \n",
       "SecondLargestPropertyUseType                                 50.27  \n",
       "SecondLargestPropertyUseTypeGFA                              50.27  \n",
       "ENERGYSTARScore                                              24.97  \n",
       "LargestPropertyUseType                                        0.59  \n",
       "LargestPropertyUseTypeGFA                                     0.59  \n",
       "ZipCode                                                       0.47  \n",
       "SteamUse(kBtu)                                                0.27  \n",
       "TotalGHGEmissions                                             0.27  \n",
       "NaturalGas(kBtu)                                              0.27  \n",
       "NaturalGas(therms)                                            0.27  \n",
       "Electricity(kBtu)                                             0.27  \n",
       "Electricity(kWh)                                              0.27  \n",
       "SourceEUI(kBtu/sf)                                            0.27  \n",
       "SourceEUIWN(kBtu/sf)                                          0.27  \n",
       "ListOfAllPropertyUseTypes                                     0.27  \n",
       "GHGEmissionsIntensity                                         0.27  \n",
       "NumberofBuildings                                             0.24  \n",
       "SiteEUI(kBtu/sf)                                              0.21  \n",
       "SiteEnergyUseWN(kBtu)                                         0.18  \n",
       "SiteEUIWN(kBtu/sf)                                            0.18  \n",
       "SiteEnergyUse(kBtu)                                           0.15  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compter le nombre de valeurs manquantes et calculer le pourcentage de NaN dans le dataframe\n",
    "missing_values = pd.DataFrame(df.isna().sum(), columns = ['Total de valeurs manquantes'])\n",
    "missing_values['Pourcentage de valeurs manquantes'] = round((missing_values['Total de valeurs manquantes'] / len(df)) * 100, 2)\n",
    "\n",
    "# Sélectionner uniquement les colonnes avec des valeurs manquantes\n",
    "missing_values = missing_values.query(\"`Total de valeurs manquantes` > 0\")\n",
    "\n",
    "# Trier les colonnes par ordre décroissant de nombre de valeurs manquantes\n",
    "missing_values = missing_values.sort_values(by = ['Total de valeurs manquantes'], ascending=False)\n",
    "display(missing_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a55900",
   "metadata": {},
   "source": [
    "## A. Analyse univariée"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5624331",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.hist(figsize=(45,45))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e917eed9",
   "metadata": {},
   "source": [
    "Dans le graphique ci-dessus, vous pouvez voir les histogrammes de toutes les variables numériques. Certaines variables sont réparties dans une plage de valeurs spécifique, tandis que d'autres ont une distribution en forme de cloche avec des valeurs qui augmentent de manière logarithmique. Les variables Zipcode et Coincil District Code prennent des valeurs catégorielles spécifiques. Il est important d'examiner attentivement ces variables et de les préparer pour des méthodes de machine learning à l'aide de techniques de feature engineering appropriées.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62564468",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Neighborhood'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1926ba9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Convertir toutes les chaînes de caractères en majuscules en utilisant la méthode \n",
    "df['Neighborhood'] = df['Neighborhood'].str.upper()\n",
    "df['Neighborhood'] = df['Neighborhood'].str.replace('DELRIDGE NEIGHBORHOODS', 'DELRIDGE')\n",
    "df['Neighborhood'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d5d8e3",
   "metadata": {},
   "source": [
    " Il semble que la majorité des données proviennent des quartiers centraux de la ville. Les cinq premiers quartiers représentent 60% des données totales. Cela peut être pris en compte lors de l'analyse et de la modélisation des données, car cela peut entraîner un biais dans les prévisions pour les quartiers moins représentés dans les données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13c4f89e",
   "metadata": {},
   "outputs": [],
   "source": [
    "color_var = pd.factorize(df['Neighborhood'])[0]\n",
    "\n",
    "fig = px.scatter_mapbox(df, lat=\"Latitude\", lon=\"Longitude\", hover_name=\"Neighborhood\",\n",
    "                        color=color_var, zoom=9, height=300)\n",
    "fig.update_layout(mapbox_style=\"open-street-map\")\n",
    "fig.update_layout(margin={\"r\":0,\"t\":0,\"l\":0,\"b\":0})\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc9d04fa",
   "metadata": {},
   "source": [
    "Comme vous pouvez le voir dans la figure ci-dessus, vous pouvez visualiser la répartition des noms de quartiers sur la carte. De cette manière, vous pouvez voir où se situent les différents quartiers sur la carte. Il est probable que la consommation d'énergie dans les bâtiments puisse varier dans différents quartiers, et ces différences peuvent être utilisées dans l'ingénierie des fonctionnalités."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d2d6831",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "sns.countplot(x=df['BuildingType'])\n",
    "plt.xticks(rotation=90, fontsize=16)\n",
    "plt.yticks(fontsize=16)\n",
    "plt.xlabel('Building Type', fontsize=18)\n",
    "plt.ylabel('Count', fontsize=18)\n",
    "plt.title('Distribution of Building Types', fontsize=22)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60cd6eaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['BuildingType'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7928b20",
   "metadata": {},
   "source": [
    "Lorsqu'on examine la colonne \"BuildingType\", on peut voir qu'il y a plusieurs types de bâtiments différents, notamment \"NonResidential\", \"Multifamily\", \"SPS-District K-12\", \"Nonresidential COS\" et \"Campus\". Étant donné que ce projet se concentre uniquement sur les bâtiments \"NonResidential\", les types \"Multifamily\" peuvent être exclus de l'ensemble de données. De plus, puisqu'il n'y a qu'un seul exemple de données appartenant à \"Nonresidential WA\", ces données peuvent être supprimées car elles ne pourront pas être utilisées dans la phase de modélisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0980a8c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[[ 'BuildingType',\n",
    " 'PrimaryPropertyType',\n",
    " 'SecondLargestPropertyUseType',\n",
    " 'ThirdLargestPropertyUseType',\n",
    " 'LargestPropertyUseType',\n",
    " 'ListOfAllPropertyUseTypes'\n",
    "    ]].head(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbc04f92",
   "metadata": {},
   "source": [
    "Effectivement, chaque type de bâtiment peut comporter différents types d'espaces ou de locaux tels que des restaurants, des bureaux, des entrepôts, etc. Par conséquent, la colonne \"ListOfAllPropertyUseTypes\" fournit des informations plus détaillées sur les différents types de bâtiments. Je vais utiliser la colonne de données \"PrimaryPropertyType\" pour la visualisation car ces données sont plus propres et généralisées que les autres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a13e9f20",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PrimaryPropertyType'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff13fe3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "property_type_colors = {\n",
    "    'Low-Rise Multifamily': 'purple',\n",
    "    'Mid-Rise Multifamily': 'violet',\n",
    "    'Small- and Mid-Sized Office': 'blue',\n",
    "    'Other': 'black',\n",
    "    'Warehouse': 'gray',\n",
    "    'Large Office': 'navy',\n",
    "    'K-12 School': 'green',\n",
    "    'Mixed Use Property': 'orange',\n",
    "    'High-Rise Multifamily': 'deep pink',\n",
    "    'Retail Store': 'red',\n",
    "    'Hotel': 'brown',\n",
    "    'Worship Facility': 'olive',\n",
    "    'Distribution Center': 'dark gray',\n",
    "    'Senior Care Community': 'light pink',\n",
    "    'Supermarket / Grocery Store': 'maroon',\n",
    "    'Medical Office': 'teal',\n",
    "    'Self-Storage Facility': 'dark orange',\n",
    "    'University': 'forest green',\n",
    "    'Residence Hall': 'magenta',\n",
    "    'Refrigerated Warehouse': 'light gray',\n",
    "    'Restaurant': 'crimson',\n",
    "    'Hospital': 'dark red',\n",
    "    'Laboratory': 'lime',\n",
    "    'Office': 'sky blue'\n",
    "}\n",
    "\n",
    "\n",
    "# Create the scattermapbox plot using the property_type_colors dictionary\n",
    "fig = px.scatter_mapbox(df, lat=\"Latitude\", lon=\"Longitude\", color=\"PrimaryPropertyType\",\n",
    "                        color_discrete_map=property_type_colors, zoom=10, height=600)\n",
    "\n",
    "# Update the layout with the mapbox style\n",
    "fig.update_layout(mapbox_style=\"open-street-map\")\n",
    "\n",
    "# Show the plot\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5ca8b42",
   "metadata": {},
   "source": [
    "Dans la carte ci-dessus, en examinant les emplacements des données, on peut voir que les riches familles habitent principalement dans le centre-ville et que la plupart des données sont des logements multifamiliaux. La plupart des données se concentrent dans les quartiers du centre-ville. Les centres de distribution de Seattle se trouvent plutôt au sud et près de l'aéroport, tandis que les écoles sont réparties dans toute la ville."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "488fd05a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Créer une liste contenant les noms des colonnes à convertir\n",
    "cols_to_convert = ['PropertyGFATotal', 'PropertyGFAParking', 'PropertyGFABuilding(s)', 'LargestPropertyUseTypeGFA', 'SecondLargestPropertyUseTypeGFA', 'ThirdLargestPropertyUseTypeGFA']\n",
    "\n",
    "# Définir la constante de conversion (de pieds carrés à mètres carrés)\n",
    "SQ_FT_TO_SQ_M = 0.092903\n",
    "\n",
    "# Parcourir la liste des colonnes à convertir\n",
    "for col in cols_to_convert:\n",
    "    # Convertir les valeurs de la colonne en mètres carrés en multipliant par la constante de conversion\n",
    "    df[col] = df[col] * SQ_FT_TO_SQ_M\n",
    "\n",
    "# rename the columns to indicate that they are now in square meters\n",
    "df = df.rename(columns={\n",
    "    'PropertyGFATotal': 'PropertyGFATotal_m2',\n",
    "    'PropertyGFAParking': 'PropertyGFAParking_m2',\n",
    "    'PropertyGFABuilding(s)': 'PropertyGFABuilding(s)_m2',\n",
    "    'LargestPropertyUseTypeGFA': 'LargestPropertyUseTypeGFA_m2',\n",
    "    'SecondLargestPropertyUseTypeGFA': 'SecondLargestPropertyUseTypeGFA_m2',\n",
    "    'ThirdLargestPropertyUseTypeGFA': 'ThirdLargestPropertyUseTypeGFA_m2'\n",
    "})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d60a228",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate 90th percentile of PropertyGFATotal_m2\n",
    "threshold = df['PropertyGFATotal_m2'].quantile(0.90)\n",
    "# create a subset of the dataframe with values below the threshold\n",
    "df_no_outliers = df[df['PropertyGFATotal_m2'] < threshold]\n",
    "\n",
    "# create a histogram of PropertyGFATotal_m2 without outliers\n",
    "df_no_outliers['PropertyGFATotal_m2'].iplot(kind='hist', bins=40)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e636cb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['PropertyGFATotal_m2'].idxmax()]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a50d1db",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['SiteEnergyUse(kBtu)'].idxmax()]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "355c856f",
   "metadata": {},
   "source": [
    "\n",
    "D'après les données, le complexe universitaire est l'endroit avec la consommation d'énergie la plus élevée et la superficie de bâtiment la plus élevée. Il est composé de 111 bâtiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dbef554",
   "metadata": {},
   "outputs": [],
   "source": [
    "zero_floors = df[df['NumberofFloors'] == 0]\n",
    "for index, row in zero_floors.iterrows():\n",
    "    print(\"Building Type: \", row['BuildingType'], ':', row['ListOfAllPropertyUseTypes'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b712ca9e",
   "metadata": {},
   "source": [
    "Comme il est intéressant de constater, ce complexe de logements est enregistré avec un nombre d'étages de 0. En examinant de plus près les bâtiments avec ce nombre d'étages, on constate que la plupart d'entre eux sont des dortoirs d'étudiants et certains ont été mal enregistrés. Il est donc nécessaire de nettoyer ces données en conséquence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0739eac",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"YearBuilt\"].iplot(kind='hist')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "841bf94f",
   "metadata": {},
   "source": [
    "\n",
    "Pour convertir la colonne \"YearBuilt\" en âge du bâtiment en considérant l'année actuelle comme 2023, nous pouvons utiliser la formule suivante :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa3e67f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "current_year = datetime.datetime.now().year\n",
    "df[\"BuildingAge\"] = current_year - df[\"YearBuilt\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1e94ae4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['BuildingAge'].iplot(kind='box')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b7562c",
   "metadata": {},
   "source": [
    "La moyenne d'âge des bâtiments est de 46 ans et la plupart des valeurs se situent entre 26 et 75 ans. Le bâtiment le plus ancien a 123 ans"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5ee742f",
   "metadata": {},
   "source": [
    "## B. Analyse bivariée"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8ff483f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,10))\n",
    "sns.boxplot(x='BuildingType', y='NumberofFloors', data=df)\n",
    "plt.xticks(rotation=60, fontsize=16)\n",
    "plt.yticks(fontsize=16)\n",
    "plt.xlabel('Building Type', fontsize=18)\n",
    "plt.ylabel('Number of Floors', fontsize=18)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7820053",
   "metadata": {},
   "source": [
    "\n",
    "Comme on peut le voir sur ce boxplot, les gratte-ciels à plusieurs étages sont généralement de type de bâtiment non résidentiel. Les résidences multifamiliales et les campus contiennent des bâtiments moins élevés."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8df1064",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['NumberofBuildings'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fadbfb11",
   "metadata": {},
   "source": [
    "Dans la colonne \"Nombre de bâtiments\", il y a une valeur de 0 qui n'est pas logique. Par conséquent, j'ai corrigé cela en la remplaçant par 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f82ed1b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['NumberofBuildings'] = df['NumberofBuildings'].replace(0, 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dfdef84",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['NumberofBuildings'] != 111]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "398bb30d",
   "metadata": {},
   "source": [
    "J'ai également supprimé la ligne de données qui comprend 111 bâtiments. Comme nous l'avons constaté, il s'agit d'une zone du campus et cela ne correspond pas au projet. Cela pourrait entraîner une erreur lors de l'utilisation de méthodes ML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a321348",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.box(df, x='BuildingType', y='NumberofBuildings')\n",
    "fig.update_layout(\n",
    "    xaxis_title='Building Type',\n",
    "    yaxis_title='Number of Buildings',\n",
    "    font=dict(size=18),\n",
    "    width=1000,\n",
    "    height=600\n",
    ")\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23af40e3",
   "metadata": {},
   "source": [
    "\n",
    "Le nombre de bâtiments sur le campus est plus élevé que celui des autres types de bâtiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14000bb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(50,50))\n",
    "sns.set(font_scale=2.5)\n",
    "numeric_columns = df.select_dtypes(include=['int32', 'float64']).columns\n",
    "\n",
    "plt.title('Matrice de corrélation de Pearson entre les différentes features', fontsize=50)\n",
    "\n",
    "corr = df[numeric_columns].corr()\n",
    "\n",
    "mask = np.zeros_like(corr, dtype=bool)\n",
    "mask[np.triu_indices_from(mask)] = True \n",
    "\n",
    "ax = sns.heatmap(corr, mask=mask, vmin=-1, cmap='coolwarm', annot=True, annot_kws={'size':35})\n",
    "ax.tick_params(axis='both', labelsize=50)\n",
    "plt.xticks(rotation=90, fontsize=50,fontweight='bold')\n",
    "plt.yticks(rotation=0, fontsize=50,fontweight='bold')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a389f995",
   "metadata": {},
   "source": [
    "Le score Energystars est négativement corrélé avec les valeurs de consommation des bâtiments. D'autre part, la quantité annuelle d'énergie consommée par la propriété est fortement corrélée avec la consommation d'électricité et la valeur totale d'émissions de gaz à effet de serre (GES). Sa corrélation avec l'utilisation de gaz naturel et de vapeur n'est également pas négligeable. L'autre corrélation élevée est liée à la surface de plancher brute et à la consommation d'énergie."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70469cb3",
   "metadata": {},
   "source": [
    "<a id='sec:pretraitement-donnees'></a>\n",
    "# III. Prétraitement des données"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c7202d",
   "metadata": {},
   "source": [
    "### A. Traitement des valeurs manquantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf71c33b",
   "metadata": {},
   "outputs": [],
   "source": [
    "msno.bar(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b5285b5",
   "metadata": {},
   "source": [
    "La plupart des données sont complètes. Les données manquantes sont pour la plupart inutiles à traiter. La superficie brute des bâtiments (GFA) n'est pas définie pour les bâtiments de type de propriété secondaire. Je vais laisser la colonne \"EnergyStarScore\" intacte, car j'ai besoin des valeurs de l'ENERGY STAR Score pour pouvoir prédire certaines valeurs énergétiques à l'avenir. Maintenant, je vais commencer le nettoyage des données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09cd9d4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ZipCode'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532c299a",
   "metadata": {},
   "source": [
    "Dans ce code, nous utilisons la bibliothèque Geopy pour géocoder les adresses. La fonction geocode_address prend une adresse en entrée et renvoie les coordonnées géographiques correspondantes. Nous itérons ensuite sur chaque ligne du DataFrame pour prédire le code postal en utilisant la fonction geocode_address"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ec41fe2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import geopy\n",
    "from geopy.geocoders import Nominatim\n",
    "import re\n",
    "\n",
    "def geocode_address(address):\n",
    "    geolocator = Nominatim(user_agent=\"my-app\") \n",
    "    location = geolocator.geocode(address)\n",
    "    return location\n",
    "\n",
    "# Parcourir les lignes du DataFrame et prédire le code postal pour chaque adresse\n",
    "for index, row in df.iterrows():\n",
    "    address = row['Address']\n",
    "    zip_code = row['ZipCode']\n",
    "\n",
    "    if pd.isnull(zip_code):  # Vérifier si le code postal est NaN\n",
    "        location = geocode_address(address)\n",
    "\n",
    "        if location is not None:\n",
    "            try:\n",
    "                zip_code = location.raw['address']['postcode']\n",
    "            except KeyError:\n",
    "                display_name = location.raw['display_name']\n",
    "                zip_code_match = re.search(r'Washington, (\\d{5})', display_name)\n",
    "                if zip_code_match:\n",
    "                    zip_code = zip_code_match.group(1)\n",
    "                else:\n",
    "                    zip_code = None\n",
    "\n",
    "            df.loc[index, 'ZipCode'] = zip_code\n",
    "        else:\n",
    "            df.loc[index, 'ZipCode'] = None\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f84a4a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ZipCode'].value_counts()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "338a19e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['ZipCode'].isna()==True]['ZipCode']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d51ec801",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ZipCode'] = pd.to_numeric(df['ZipCode'], errors='coerce')\n",
    "df.loc[3372,'ZipCode'] = 98104.0\n",
    "df.loc[3361,'ZipCode'] = 98144.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8abe6c24",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('Comments', inplace=True,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a4b3b8",
   "metadata": {},
   "source": [
    "### B. Traitement des données aberrantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b5d6c4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Outlier'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e567dd41",
   "metadata": {},
   "source": [
    "Dans la colonne \"Outlier\", j'ai choisi de les supprimer en raison de la faible quantité de données, mais on pourrait également les utiliser en utilisant OneHotEncoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "985b550f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop(df[df['Outlier'].isin(['Low outlier', 'High outlier'])].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01a3408d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('Outlier', inplace=True,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6183c9d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['BuildingType'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f325edbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop(df[df['BuildingType'].isin(['Multifamily LR (1-4)', 'Multifamily MR (5-9)', 'Multifamily HR (10+)', 'Nonresidential WA'])].index)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa984235",
   "metadata": {},
   "source": [
    "Dans ce projet, nous nous concentrons uniquement sur les appartements non-résidentiels. C'est pourquoi nous supprimons les lignes de données qui ne correspondent pas à cette sélection."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f8910b1",
   "metadata": {},
   "source": [
    "Étant donné qu'il n'y a qu'une seule donnée dans la catégorie des bâtiments Nonresidential WA  nous l'avons supprimée. La généralisation de cette donnée n'est pas très importante pour les algorithmes d'apprentissage automatique (ML)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "601fabe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ComplianceStatus'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32e70971",
   "metadata": {},
   "source": [
    "J'ai également supprimé certaines lignes qui contenaient des données incomplètes ou manquantes dans la colonne \"ComplianceStatus\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6714a9bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['ComplianceStatus']=='Compliant']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9adf55f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PrimaryPropertyType'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70f25c43",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['PrimaryPropertyType']=='Low-Rise Multifamily']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c267c828",
   "metadata": {},
   "source": [
    "Même si j'ai supprimé les données contenant la catégorie 'Multifamily Housing', certaines données sont encore présentes dans la variable 'PrimaryPropertyType'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcfc2caa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remplacer les valeurs manquantes par une chaîne de caractères vide\n",
    "df[\"LargestPropertyUseType\"].fillna(\"\", inplace=True)\n",
    "\n",
    "# Filtrer les lignes qui contiennent \"Multifamily Housing\"\n",
    "filtre = df[\"LargestPropertyUseType\"].str.contains(\"Multifamily Housing\")\n",
    "resultats = df[filtre]\n",
    "\n",
    "# Imprimer les résultats\n",
    "display(resultats)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d22e066f",
   "metadata": {},
   "source": [
    "Il reste encore quelques catégories de logements multifamiliaux sous la colonne \"largestPropertyUseType\". Dans le cadre de ce projet, j'ai décidé de négliger ces colonnes afin de me concentrer uniquement sur les bâtiments non résidentiels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50a7e6c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[~filtre]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26def53e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Définition des colonnes d'énergie secondaires\n",
    "sub_energy_cols = ['SteamUse(kBtu)', 'Electricity(kBtu)', 'NaturalGas(kBtu)']\n",
    "\n",
    "# Calcul de la somme des énergies secondaires pour chaque ligne\n",
    "df['SiteEnergyUse(kBtu)_total'] = df[sub_energy_cols].sum(axis='columns')\n",
    "\n",
    "# Définition des limites inférieure et supérieure de la plage [90%-110%]\n",
    "lower_bound = df['SiteEnergyUse(kBtu)'] * 0.9\n",
    "upper_bound = df['SiteEnergyUse(kBtu)'] * 1.1\n",
    "\n",
    "# Vérification si la somme des énergies n'est pas comprise dans la plage [90%-110%] de 'SiteEnergyUse(kBtu)'\n",
    "m_en_out = ~df['SiteEnergyUse(kBtu)_total'].between(lower_bound, upper_bound)\n",
    "\n",
    "# Affichage du nombre de lignes où l'énergie totale n'est pas dans la plage [90%-110%] par rapport à 'SiteEnergyUse(kBtu)'\n",
    "print(\"Nombre de lignes où l'énergie totale n'est pas dans la plage [90%-110%] de 'SiteEnergyUse(kBtu)': \",\n",
    "      df[m_en_out].shape[0], '/', df.shape[0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf3ff1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.scatter(df, x='SiteEnergyUse(kBtu)_total', y='SiteEnergyUse(kBtu)', color='LargestPropertyUseType')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea203f53",
   "metadata": {},
   "source": [
    "Il y a 15 données pour lesquelles les valeurs d'énergie utilisée ne correspondent pas à la somme des valeurs d'utilisation d'électricité, de gaz naturel et de vapeur. Cela signifie que ces bâtiments pourraient utiliser d'autres sources d'énergie qui ne sont pas répertoriées dans l'ensemble de données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16216a65",
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_GFA_cols1 = ['PropertyGFAParking_m2', 'PropertyGFABuilding(s)_m2']\n",
    "\n",
    "# Calculer la surface brute totale (GFA) pour chaque ligne\n",
    "df['PropertyGFATotal_total'] = df[sub_GFA_cols1].sum(axis='columns')\n",
    "\n",
    "# Calculer l'écart en pourcentage par rapport à 'PropertyGFATotal' d'origine\n",
    "df['Écart_GFA_en_pourcentage'] = (df['PropertyGFATotal_total'] / df['PropertyGFATotal_m2']) * 100\n",
    "\n",
    "# Filtrer les lignes où la GFA ne se situe pas dans la plage [99%-101%] par rapport à 'PropertyGFATotal'\n",
    "m_GFA_out = ~df['Écart_GFA_en_pourcentage'].between(99, 101)\n",
    "\n",
    "# Compter le nombre de lignes où la GFA est en dehors de la plage [99%-101%]\n",
    "num_GFA_out = df[m_GFA_out].shape[0]\n",
    "\n",
    "# Afficher le résultat\n",
    "print(f\"Nombre de lignes où la surface brute totale (GFA) n'est pas dans la plage [99%-101%] de 'PropertyGFATotal': {num_GFA_out} / {df.shape[0]}\")\n",
    "\n",
    "# Supprimer les colonnes intermédiaires\n",
    "df = df.drop(columns=['PropertyGFATotal_total', 'Écart_GFA_en_pourcentage'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "452f8c41",
   "metadata": {},
   "source": [
    "La somme de la surface de stationnement et de la surface du bâtiment est corrélée avec le nombre total de mètres carrés."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa77f7f",
   "metadata": {},
   "source": [
    "### Z-score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13d617a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "nom_colonne1 = 'TotalGHGEmissions'\n",
    "nom_colonne2 = 'SiteEnergyUse(kBtu)'\n",
    "\n",
    "# Calculer les scores Z pour les colonnes\n",
    "scores_z1 = np.abs((df[nom_colonne1] - df[nom_colonne1].mean()) / df[nom_colonne1].std())\n",
    "scores_z2 = np.abs((df[nom_colonne2] - df[nom_colonne2].mean()) / df[nom_colonne2].std())\n",
    "\n",
    "# Trouver les valeurs aberrantes en utilisant un seuil\n",
    "seuil = 3\n",
    "valeurs_aberrantes1 = df[scores_z1 > seuil][nom_colonne1]\n",
    "valeurs_aberrantes2 = df[scores_z2 > seuil][nom_colonne2]\n",
    "\n",
    "# Créer un tableau pour afficher les valeurs aberrantes\n",
    "tableau_valeurs_aberrantes = pd.DataFrame({\n",
    "    nom_colonne1: valeurs_aberrantes1,\n",
    "    nom_colonne2: valeurs_aberrantes2\n",
    "})\n",
    "\n",
    "# Afficher le tableau\n",
    "print(\"Tableau des valeurs aberrantes:\")\n",
    "print(tableau_valeurs_aberrantes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5640eace",
   "metadata": {},
   "source": [
    "Nous avons obtenu des valeurs aberrantes pour les colonnes TotalGHGEmissions et SiteEnergyUse(kBtu). Beaucoup d'entre elles partagent également le même indice. Par conséquent, nous avons décidé de supprimer les lignes de données qui pourraient affecter les méthodes de ML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2d938b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Supprimer les données aberrantes du DataFrame\n",
    "df = df.drop(tableau_valeurs_aberrantes.index)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c327d2aa",
   "metadata": {},
   "source": [
    "<a id='sec:selection-variables'></a>\n",
    "# IV. Sélection de variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b9799ae",
   "metadata": {},
   "source": [
    "### A. Analyse de la corrélation entre les variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efc60ddb",
   "metadata": {},
   "source": [
    "J'ai supprimé les colonnes qui ne contiennent qu'une seule valeur unique, puis j'ai supprimé certaines colonnes qui sont sans pertinence dans le cadre de ce projet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4103ce75",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in df.columns:\n",
    "    if df[col].nunique() == 1:\n",
    "        df.drop(col, axis=1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb1572a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PropertyName'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc3a88a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('PropertyName',axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b1548e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('TaxParcelIdentificationNumber',axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81ff36d2",
   "metadata": {},
   "source": [
    "La plupart des données de consommation énergétique élevée proviennent de l'hôtel qui possède la plus grande superficie brute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5a0b556",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.scatter(df, x='PropertyGFATotal_m2', y='SiteEnergyUse(kBtu)', color='LargestPropertyUseType')\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddcef086",
   "metadata": {},
   "source": [
    "Les laboratoires, les supermarchés et les hôpitaux utilisent généralement plus d'énergie, mais la catégorie \"Other\" regroupe des valeurs énergétiques très diverses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f82ef73",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.box(df, x='PrimaryPropertyType', y='SiteEUI(kBtu/sf)',\n",
    "             width=800, height=600, color_discrete_sequence=px.colors.qualitative.Pastel)\n",
    "\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f249a58f",
   "metadata": {},
   "source": [
    "### B. Analyse de la variance des variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06186e26",
   "metadata": {},
   "source": [
    "J'ai réalisé une visualisation de la distribution du score ENERGY STAR en utilisant une estimation de la densité par noyau (KDE). Cette visualisation nous permet d'observer la répartition des scores ENERGY STAR dans notre jeu de données. La méthode KDE nous donne une estimation de la densité de probabilité, ce qui nous permet de mieux comprendre la variabilité des scores. Cette analyse de la variance des variables nous aide à identifier les variations et les tendances dans les scores ENERGY STAR,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb27aa83",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ENERGYSTARScore'].iplot(kind='hist')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09b1849b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop rows with NaN values in 'ENERGYSTARScore' column\n",
    "score_data = df['ENERGYSTARScore'].dropna()\n",
    "\n",
    "# create the histogram with KDE\n",
    "fig = ff.create_distplot([score_data], ['Energy Star Score'], bin_size=1, curve_type='kde')\n",
    "\n",
    "# update layout and display the figure\n",
    "fig.update_layout(title='Energy Star Score Distribution with KDE', xaxis_title='Score', yaxis_title='Density')\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da132a52",
   "metadata": {},
   "source": [
    "J'ai effectué plusieurs visualisations pour analyser la variance des variables . Tout d'abord, j'ai tracé un histogramme de la variable 'SiteEUI(kBtu/sf)' afin d'examiner la distribution de la consommation d'énergie par pied carré. Cela nous permet de mieux comprendre la répartition des valeurs et de détecter d'éventuelles tendances ou valeurs aberrantes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0eda4497",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['SiteEUI(kBtu/sf)'].iplot(kind='hist')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbfeb974",
   "metadata": {},
   "source": [
    "Ensuite, j'ai regroupé les années de construction des bâtiments en catégories de 10 ans (par exemple, 1900-1909, 1910-1919, etc.) et calculé la moyenne du score ENERGY STAR pour chaque groupe d'années. J'ai représenté ces moyennes sous forme de diagramme à barres afin de visualiser les variations du score d'efficacité énergétique au fil des années. Cela nous permet d'analyser l'évolution de l'efficacité énergétique des bâtiments en fonction de leur année de construction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7a4d67d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new column for year groups\n",
    "df['YearGroup'] = pd.cut(df['YearBuilt'], bins=range(df['YearBuilt'].min(), df['YearBuilt'].max()+11, 10), labels=range(df['YearBuilt'].min(), df['YearBuilt'].max(), 10))\n",
    "\n",
    "# calculate the average energy score for each year group\n",
    "df_grouped = df.groupby('YearGroup')['ENERGYSTARScore'].mean().reset_index()\n",
    "\n",
    "# create the bar plot\n",
    "fig = px.bar(df_grouped, x='YearGroup', y='ENERGYSTARScore', labels={'YearGroup': 'Year Group', 'ENERGYSTARScore': 'Average Energy Score'})\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f031bcf5",
   "metadata": {},
   "source": [
    "Enfin, j'ai créé des diagrammes de dispersion pour examiner les relations entre différentes variables liées à la consommation et à l'utilisation d'énergie. J'ai tracé des diagrammes de dispersion pour les paires de variables suivantes : 'SourceEUI' vs 'SourceEUIWN', 'SiteEUI' vs 'SiteEUIWN', et 'SiteEnergyUse' vs 'SiteEnergyUseWN'. Ces diagrammes de dispersion nous permettent d'analyser les corrélations et les tendances entre ces variables, ce qui est important pour comprendre les facteurs influençant la consommation d'énergie des bâtiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e7e5557",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a scatter plot for SourceEUI and SourceEUIWN\n",
    "fig1 = px.scatter(df, x='SourceEUI(kBtu/sf)', y='SourceEUIWN(kBtu/sf)')\n",
    "\n",
    "# Create a scatter plot for SiteEUI and SiteEUIWN\n",
    "fig2 = px.scatter(df, x='SiteEUI(kBtu/sf)', y='SiteEUIWN(kBtu/sf)')\n",
    "\n",
    "# Create a scatter plot for SiteEnergyUse and SiteEnergyUseWN\n",
    "fig3 = px.scatter(df, x='SiteEnergyUse(kBtu)', y='SiteEnergyUseWN(kBtu)')\n",
    "\n",
    "# Show the plots\n",
    "fig1.show()\n",
    "fig2.show()\n",
    "fig3.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331c6a30",
   "metadata": {},
   "source": [
    "Les valeurs d'énergie ajustées pour représenter ce que la propriété aurait consommé lors de conditions météorologiques moyennes sur 30 ans montrent une tendance similaire aux valeurs non ajustées. Cependant, certaines valeurs ajustées sont manquantes et donnent des valeurs nulles. Le reste des valeurs ajustées est similaire aux valeurs non ajustées."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c5a772f",
   "metadata": {},
   "outputs": [],
   "source": [
    "color_var = pd.factorize(df['SiteEUI(kBtu/sf)'])[0]\n",
    "\n",
    "fig = px.scatter_mapbox(df, lat=\"Latitude\", lon=\"Longitude\", hover_name=\"SiteEUI(kBtu/sf)\",\n",
    "                        color=color_var, zoom=9, height=300)\n",
    "fig.update_layout(mapbox_style=\"open-street-map\")\n",
    "fig.update_layout(margin={\"r\":0,\"t\":0,\"l\":0,\"b\":0})\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f2934cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 6))\n",
    "ax = sns.boxplot(data=df, x='Neighborhood', y='SiteEnergyUse(kBtu)')\n",
    "\n",
    "# Increase font size of axis labels\n",
    "ax.set_xlabel('Quartier', fontsize=20)\n",
    "ax.set_ylabel('Consommation d\\'énergie (kBtu)', fontsize=20)\n",
    "\n",
    "plt.xticks(fontsize=12, rotation=45)  # Increase font size of x-axis labels and rotate them for better visibility\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f3abafb",
   "metadata": {},
   "source": [
    "La consommation d'énergie n'est pas directement proportionnelle à la latitude et à la longitude. Cependant, dans le centre-ville, le taux de consommation d'énergie est plus élevé que dans le reste de la ville."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b70fa613",
   "metadata": {},
   "source": [
    "### C. Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1591f75d",
   "metadata": {},
   "source": [
    "Dans cette section du notebook, nous aborderons l'ingénierie des caractéristiques (feature engineering). Nous explorerons différentes techniques pour créer de nouvelles variables à partir des données existantes, telles que la transformation des variables, la création de variables d'interaction et la sélection de variables pertinentes. L'objectif est d'améliorer la performance de nos modèles de prédiction en fournissant des informations supplémentaires et en réduisant le bruit dans les données."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f081fc7c",
   "metadata": {},
   "source": [
    "### Energy Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60c53259",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['SteamRatio'] = df['SteamUse(kBtu)'] / df['SiteEnergyUse(kBtu)']\n",
    "df['ElectricityRatio'] = df['Electricity(kBtu)'] / df['SiteEnergyUse(kBtu)']\n",
    "df['NaturalGasRatio'] = df['NaturalGas(kBtu)'] / df['SiteEnergyUse(kBtu)']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7db54fe",
   "metadata": {},
   "source": [
    " Cette étape d'ingénierie des caractéristiques vise à capturer les contributions relatives de différentes sources d'énergie dans la consommation d'énergie globale des bâtiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b347406",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[['SteamUse(kBtu)', 'Electricity(kBtu)', 'NaturalGas(kBtu)']].idxmax(axis=1).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a15f1149",
   "metadata": {},
   "source": [
    "Parmi les trois types d'énergie, l'électricité est le plus utilisé, suivi du gaz naturel. Le type d'énergie le moins utilisé est la vapeur. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb845d32",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Création de nouvelles colonnes avec des valeurs initiales définies à 0\n",
    "df['most_steam'] = 0\n",
    "df['most_electricity'] = 0\n",
    "df['most_gas'] = 0\n",
    "\n",
    "# Recherche de la colonne avec la valeur la plus élevée dans chaque ligne et configuration de la nouvelle colonne correspondante à 1\n",
    "max_column = df[['SteamUse(kBtu)', 'Electricity(kBtu)', 'NaturalGas(kBtu)']].idxmax(axis=1)\n",
    "df.loc[max_column == 'SteamUse(kBtu)', 'most_steam'] = 1\n",
    "df.loc[max_column == 'Electricity(kBtu)', 'most_electricity'] = 1\n",
    "df.loc[max_column == 'NaturalGas(kBtu)', 'most_gas'] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "560f324f",
   "metadata": {},
   "source": [
    "Dans ce code, nous créons de nouvelles colonnes avec des valeurs initiales définies à 0. Ensuite, nous recherchons la colonne avec la valeur la plus élevée dans chaque ligne et nous configurons la nouvelle colonne correspondante à 1. Ainsi, les colonnes \"plus_de_vapeur\", \"plus_d_electricite\" et \"plus_de_gaz\" auront une valeur de 1 si la vapeur, l'électricité ou le gaz ont la plus grande consommation respective dans chaque bâtiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae805742",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prendre le logarithme des valeurs d'énergie\n",
    "df['log_energie'] = np.log(df['SiteEnergyUse(kBtu)'])\n",
    "\n",
    "# Regrouper les données par quartier et calculer la moyenne de SiteEnergyUse(kBtu) pour chaque groupe\n",
    "moyenne_par_quartier = df.groupby('Neighborhood')['log_energie'].mean()\n",
    "\n",
    "# Créer un DataFrame trié\n",
    "ordre_quartier = moyenne_par_quartier.sort_values(ascending=False).index\n",
    "\n",
    "# Créer le diagramme en boîte à moustaches avec Plotly\n",
    "fig = px.violin(df, x='Neighborhood', y='log_energie', category_orders={'Neighborhood': ordre_quartier})\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6269b081",
   "metadata": {},
   "source": [
    "Le logarithme est souvent utilisé pour transformer les variables cibles dans le contexte de l'apprentissage automatique. L'utilisation du logarithme permet de réduire l'échelle des valeurs et de rendre la distribution des données plus symétrique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0325a7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prendre le logarithme des valeurs d'émissions\n",
    "df['log_emission'] = np.log(df['TotalGHGEmissions'].replace(0, np.nan))\n",
    "\n",
    "# Regrouper les données par quartier et calculer la moyenne de log_emission pour chaque groupe\n",
    "moyenne_par_quartier = df.groupby('Neighborhood')['log_emission'].mean()\n",
    "\n",
    "# Créer un DataFrame trié\n",
    "ordre_quartier = moyenne_par_quartier.sort_values(ascending=False).index\n",
    "\n",
    "# Créer le diagramme en boîte avec Plotly\n",
    "fig = px.box(df, x='Neighborhood', y='log_emission', category_orders={'Neighborhood': ordre_quartier})\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45207007",
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(df['log_emission'].isna())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908230ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(subset=['log_emission'], inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "629478a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd832c1d",
   "metadata": {},
   "source": [
    "De plus, la transformation logarithmique peut aider à atténuer l'effet des valeurs aberrantes et à rendre les relations entre les variables plus linéaires, ce qui peut améliorer les performances des modèles d'apprentissage automatique"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae62d350",
   "metadata": {},
   "source": [
    "### Mean GFA per floor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd7bc44f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"PropertyGFABuilding(s)_m2\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ff3b26",
   "metadata": {},
   "source": [
    "Nous créons une nouvelle caractéristique appelée 'MeanGFAperFloor', qui représente la surface brute moyenne par étage d'un bâtiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8102eb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['MeanGFAperFloor'] = df['PropertyGFABuilding(s)_m2']/(df['NumberofFloors']+1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ba0a715",
   "metadata": {},
   "source": [
    "Nous ajoutons 1 au dénominateur pour éviter une division par zéro au cas où un bâtiment n'aurait aucun étage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da63373",
   "metadata": {},
   "source": [
    "### Parking GFA ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a01e2c4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PropertyGFAParking_m2'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aae8ff2e",
   "metadata": {},
   "source": [
    "nous créons une nouvelle caractéristique appelée 'ParkingGFARatio', qui représente le ratio de la surface dédiée au stationnement par rapport à la surface totale du bâtiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb5d7dfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ParkingGFARatio'] = df['PropertyGFAParking_m2']/df['PropertyGFATotal_m2']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fae96cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ParkingGFARatio'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "152d5b0c",
   "metadata": {},
   "source": [
    "## Largest Property Use Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb86f254",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['LargestPropertyUseType'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "052accf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['LargestPropertyUseType'] != '']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e5ab055",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['LargestPropertyUseType'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb31eb69",
   "metadata": {},
   "source": [
    "Nous avons créé un mapping des types de bâtiments aux noms de catégories. Chaque type de bâtiment est associé à une catégorie correspondante."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7748ca91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mapping of building types to category names\n",
    "building_category_mapping = {\n",
    "    'Office': 'Commercial Facilities',\n",
    "    'Non-Refrigerated Warehouse': 'Industrial Facilities',\n",
    "    'Retail Store': 'Commercial Facilities',\n",
    "    'Other': 'Miscellaneous Facilities',\n",
    "    'Hotel': 'Miscellaneous Facilities',\n",
    "    'Worship Facility': 'Miscellaneous Facilities',\n",
    "    'Distribution Center': 'Industrial Facilities',\n",
    "    'K-12 School': 'Educational Facilities',\n",
    "    'Medical Office': 'Healthcare Facilities',\n",
    "    'Supermarket/Grocery Store': 'Commercial Facilities',\n",
    "    'Other - Recreation': 'Entertainment and Recreation Facilities',\n",
    "    'Parking': 'Miscellaneous Facilities',\n",
    "    'Self-Storage Facility': 'Miscellaneous Facilities',\n",
    "    'Residence Hall/Dormitory': 'Residential Facilities',\n",
    "    'Other - Entertainment/Public Assembly': 'Entertainment and Recreation Facilities',\n",
    "    'College/University': 'Educational Facilities',\n",
    "    'Senior Care Community': 'Healthcare Facilities',\n",
    "    'Laboratory': 'Educational Facilities',\n",
    "    'Restaurant': 'Entertainment and Recreation Facilities',\n",
    "    'Refrigerated Warehouse': 'Miscellaneous Facilities',\n",
    "    'Social/Meeting Hall': 'Miscellaneous Facilities',\n",
    "    'Hospital (General Medical & Surgical)': 'Healthcare Facilities',\n",
    "    'Manufacturing/Industrial Plant': 'Industrial Facilities',\n",
    "    'Strip Mall': 'Commercial Facilities',\n",
    "    'Repair Services (Vehicle, Shoe, Locksmith, etc)': 'Industrial Facilities',\n",
    "    'Fitness Center/Health Club/Gym': 'Entertainment and Recreation Facilities',\n",
    "    'Museum': 'Entertainment and Recreation Facilities',\n",
    "    'Other - Lodging/Residential': 'Residential Facilities',\n",
    "    'Automobile Dealership': 'Miscellaneous Facilities',\n",
    "    'Other - Services': 'Miscellaneous Facilities',\n",
    "    'Other - Mall': 'Miscellaneous Facilities',\n",
    "    'Other/Specialty Hospital': 'Healthcare Facilities',\n",
    "    'Financial Office': 'Commercial Facilities',\n",
    "    'Library': 'Miscellaneous Facilities',\n",
    "    'Bank Branch': 'Commercial Facilities',\n",
    "    'Urgent Care/Clinic/Other Outpatient': 'Healthcare Facilities',\n",
    "    'Other - Education': 'Educational Facilities',\n",
    "    'Performing Arts': 'Entertainment and Recreation Facilities',\n",
    "    'Prison/Incarceration': 'Miscellaneous Facilities',\n",
    "    'Data Center': 'Industrial Facilities',\n",
    "    'Lifestyle Center': 'Miscellaneous Facilities',\n",
    "    'Other - Public Services': 'Miscellaneous Facilities',\n",
    "    'Other - Restaurant/Bar': 'Miscellaneous Facilities',\n",
    "    'Adult Education': 'Educational Facilities',\n",
    "    'Other - Utility': 'Miscellaneous Facilities',\n",
    "    'Pre-school/Daycare': 'Residential Facilities',\n",
    "    'Fire Station': 'Miscellaneous Facilities',\n",
    "    'Wholesale Club/Supercenter': 'Miscellaneous Facilities',\n",
    "    'Residential Care Facility': 'Healthcare Facilities',\n",
    "    'Police Station': 'Miscellaneous Facilities',\n",
    "    'Food Service': 'Miscellaneous Facilities',\n",
    "    'Movie Theater': 'Entertainment and Recreation Facilities',\n",
    "    'Personal Services (Health/Beauty, Dry Cleaning, etc)': 'Miscellaneous Facilities',\n",
    "    'Courthouse': 'Miscellaneous Facilities'\n",
    "}\n",
    "\n",
    "# Create the category_building column based on the mapping\n",
    "df['category_building'] = df['LargestPropertyUseType'].map(building_category_mapping).fillna('Other')\n",
    "\n",
    "# Display the updated DataFrame\n",
    "print(df[['LargestPropertyUseType', 'category_building']])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96393fc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['category_building'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b86fd04",
   "metadata": {},
   "source": [
    "Pour chaque catégorie, nous avons créé une nouvelle colonne avec le nom de la catégorie suivi de '_ratio'. Cette colonne représente le ratio de la surface 'LargestPropertyUseTypeGFA_m2' par rapport à la surface totale 'PropertyGFATotal_m2' pour les bâtiments de cette catégorie. Les valeurs manquantes sont remplies avec 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31418bbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert 'category_building' column back to string data type\n",
    "df['category_building'] = df['category_building'].astype(str)\n",
    "\n",
    "# Iterate over each unique value in 'category_building'\n",
    "for category in df['category_building'].unique():\n",
    "    # Create a new column with the unique value as the column name\n",
    "    category_str = str(category)\n",
    "    df[category_str + '_ratio'] = df.loc[df['category_building'] == category, 'LargestPropertyUseTypeGFA_m2'] / df.loc[df['category_building'] == category, 'PropertyGFATotal_m2']\n",
    "    df[category_str + '_ratio'] = df[category_str + '_ratio'].fillna(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d4bcfe9",
   "metadata": {},
   "source": [
    "### Ratio Second and third largest "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab741468",
   "metadata": {},
   "source": [
    "Ces étapes permettent de calculer les ratios de surfaces pour les deuxièmes et troisièmes plus grandes utilisations de propriétés, en les rapportant à la surface totale du bâtiment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29665bbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['SecondLargestPropertyUseTypeGFA_m2'].fillna(0, inplace=True)\n",
    "df['ThirdLargestPropertyUseTypeGFA_m2'].fillna(0, inplace=True)\n",
    "df['SLP_Ratio'] = df['SecondLargestPropertyUseTypeGFA_m2'] / df['PropertyGFATotal_m2']\n",
    "df['TLP_Ratio'] = df['ThirdLargestPropertyUseTypeGFA_m2'] / df['PropertyGFATotal_m2']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba340430",
   "metadata": {},
   "source": [
    "### Years EnergySTAR Certified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b636c62f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['YearsENERGYSTARCertified'].value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045b7b30",
   "metadata": {},
   "source": [
    "Nous avons défini une fonction appelée 'split_dates' pour diviser les dates en années individuelles. Cette fonction gère les cas où les dates sont fournies sous forme d'une chaîne séparée par des virgules ou sous forme d'une chaîne continue sans virgules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9651a66f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dates(dates):\n",
    "    dates_str = str(dates).replace('nan', '')\n",
    "    if ',' in dates_str:\n",
    "        ls_date = [s.strip() for s in str(dates_str).split(',')]\n",
    "    else:\n",
    "        if len(dates_str)%4==0:\n",
    "            ls_date = [dates_str[4*(i):4*(i+1)] for i in range(int(len(dates_str)/4))]\n",
    "        else:\n",
    "            print(\"ERROR: \", dates_str)\n",
    "    return tuple(ls_date)\n",
    "\n",
    "\n",
    "df['YearsENERGYSTARCertified'] = df['YearsENERGYSTARCertified']\\\n",
    "                                             .apply(lambda x: split_dates(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77587d3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['YearsENERGYSTARCertified'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddafdb99",
   "metadata": {},
   "source": [
    "Nous avons créé une nouvelle colonne appelée 'NumYearsENERGYSTARCertified' qui représente le nombre d'années pendant lesquelles une propriété a été certifiée ENERGY STAR. Cela a été réalisé en appliquant la fonction 'len' à chaque valeur de tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42f65014",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['NumYearsENERGYSTARCertified'] = df['YearsENERGYSTARCertified'].apply(lambda x: len(x))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe8c61a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "# Get the current year\n",
    "current_year = datetime.datetime.now().year\n",
    "\n",
    "# Function to extract the first year from the tuple or return None if it's empty\n",
    "def get_first_year(years):\n",
    "    if isinstance(years, tuple) and len(years) > 0:\n",
    "        return years[-1]\n",
    "    return None\n",
    "\n",
    "\n",
    "# Extract the first year from the tuple\n",
    "df['FirstCertifiedYear'] = df['YearsENERGYSTARCertified'].apply(get_first_year)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a4510b",
   "metadata": {},
   "source": [
    "Nous avons défini une fonction appelée 'get_first_year' pour extraire la première année (la plus récente) du tuple ou retourner None si le tuple est vide. Nous avons appliqué la fonction 'get_first_year' à la colonne 'YearsENERGYSTARCertified' et assigné l'année extraite à la colonne 'FirstCertifiedYear'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "972fca0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Convert 'LastCertifiedYear' column to numeric type\n",
    "df['FirstCertifiedYear'] = pd.to_numeric(df['FirstCertifiedYear'], errors='coerce')\n",
    "\n",
    "# Perform the calculation (2023 - df['LastCertifiedYear'])\n",
    "df['YearsSinceCertified'] = 2023 - df['FirstCertifiedYear']\n",
    "\n",
    "# Replace NaN values in 'YearsSinceCertified' column with 0\n",
    "df['YearsSinceCertified'] = df['YearsSinceCertified'].fillna(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef37eb8f",
   "metadata": {},
   "source": [
    "Nous avons calculé le nombre d'années depuis la première certification en soustrayant 'FirstCertifiedYear' de l'année actuelle (2023) et avons stocké le résultat dans la colonne 'YearsSinceCertified'."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "903b6ce8",
   "metadata": {},
   "source": [
    "<a id='sec:conc'></a>\n",
    "# V. Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4284173e",
   "metadata": {},
   "source": [
    "### Analyse des corrélations entre les variables après Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded7fd33",
   "metadata": {},
   "source": [
    "Nous avons réalisé une ingénierie des caractéristiques sur notre ensemble de données, ce qui nous a permis de créer de nouvelles variables basées sur des calculs et des transformations des variables existantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f625bb20",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = make_subplots(rows=1, cols=2)\n",
    "\n",
    "# Add the first subplot\n",
    "fig.add_trace(\n",
    "    go.Scatter(x=df['ENERGYSTARScore'], y=df['YearsSinceCertified'], mode='markers'),\n",
    "    row=1, col=1\n",
    ")\n",
    "fig.update_xaxes(title_text='ENERGYSTARScore', row=1, col=1)\n",
    "fig.update_yaxes(title_text='YearsSinceCertified', row=1, col=1)\n",
    "\n",
    "# Add the second subplot\n",
    "fig.add_trace(\n",
    "    go.Scatter(x=df['ENERGYSTARScore'], y=df['NumYearsENERGYSTARCertified'], mode='markers'),\n",
    "    row=1, col=2\n",
    ")\n",
    "fig.update_xaxes(title_text='ENERGYSTARScore', row=1, col=2)\n",
    "fig.update_yaxes(title_text='NumYearsENERGYSTARCertified', row=1, col=2)\n",
    "\n",
    "# Update layout and display the subplot\n",
    "fig.update_layout(title='ENERGYSTARScore Subplots')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c9932e",
   "metadata": {},
   "source": [
    "Si un bâtiment est certifié plusieurs fois, cela lui donne un score plus élevé dans le classement ENERGY STAR."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c15c9dc5",
   "metadata": {},
   "source": [
    "Nous avons effectué un test de corrélation pour évaluer les relations entre les nouvelles variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2446cb75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sélectionner les colonnes pour la heatmap\n",
    "col_num = ['ENERGYSTARScore', 'SiteEnergyUse(kBtu)', 'SiteEUI(kBtu/sf)', 'SourceEUI(kBtu/sf)', 'BuildingAge',\n",
    "           'MeanGFAperFloor', 'ParkingGFARatio', 'SteamRatio', 'NumberofBuildings', 'NumberofFloors',\n",
    "           'PropertyGFATotal_m2', 'ElectricityRatio', 'NaturalGasRatio', 'log_energie', 'log_emission',\n",
    "           'TotalGHGEmissions', 'GHGEmissionsIntensity', 'YearsSinceCertified', 'NumYearsENERGYSTARCertified']\n",
    "\n",
    "# Créer une matrice de corrélation en utilisant la corrélation de Pearson\n",
    "correlation_matrix = df[col_num].corr(method='pearson')\n",
    "\n",
    "# Générer un masque pour le triangle supérieur\n",
    "mask = np.triu(np.ones_like(correlation_matrix, dtype=bool))\n",
    "\n",
    "# Définir la taille de la figure\n",
    "plt.figure(figsize=(30, 30))\n",
    "\n",
    "# Générer la heatmap avec les valeurs masquées\n",
    "heatmap = sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', mask=mask)\n",
    "\n",
    "# Augmenter la taille de la police des index sur l'axe x et y\n",
    "heatmap.set_xticklabels(heatmap.get_xticklabels(), fontsize=25)\n",
    "heatmap.set_yticklabels(heatmap.get_yticklabels(), fontsize=25)\n",
    "\n",
    "# Augmenter la taille de la police des nombres à l'intérieur de la heatmap\n",
    "for text in heatmap.texts:\n",
    "    text.set_fontsize(20)\n",
    "    text.set_weight('bold')\n",
    "\n",
    "# Définir le titre\n",
    "plt.title('Heatmap de corrélation de Pearson des colonnes liées à l\\'énergie', fontsize=25)\n",
    "\n",
    "# Afficher le graphique\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b5d5ee",
   "metadata": {},
   "source": [
    "Ce code sélectionne uniquement les colonnes numériques du DataFrame, calcule la matrice de corrélation entre ces variables, filtre la matrice de corrélation en ne conservant que les valeurs absolues supérieures à 0.5 (et différentes de 1), puis supprime les colonnes où toutes les corrélations sont inférieures à 0.5. Enfin, il affiche la matrice de corrélation filtrée."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97cc9b29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sélectionner uniquement les colonnes numériques\n",
    "numerical_df = df.select_dtypes(include=['float64', 'int64'])\n",
    "\n",
    "# Calculer la matrice de corrélation\n",
    "correlation_matrix = numerical_df.corr()\n",
    "\n",
    "# Filtrer la matrice de corrélation en fonction des valeurs absolues supérieures à 0.5\n",
    "filtered_corr_matrix = correlation_matrix[(abs(correlation_matrix) > 0.5) & (correlation_matrix != 1)]\n",
    "\n",
    "# Supprimer les colonnes où toutes les corrélations sont inférieures à 0.5\n",
    "filtered_corr_matrix = filtered_corr_matrix.dropna(how='all', axis=0).dropna(how='all', axis=1)\n",
    "\n",
    "# Afficher la matrice de corrélation filtrée\n",
    "filtered_corr_matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6052c692",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Créer une heatmap triangulaire\n",
    "sns.set(style=\"white\")\n",
    "mask = np.triu(np.ones_like(filtered_corr_matrix, dtype=bool))\n",
    "fig, ax = plt.subplots(figsize=(80, 50))\n",
    "heatmap = sns.heatmap(filtered_corr_matrix, mask=mask, cmap='coolwarm', annot=True, fmt=\".2f\", linewidths=2, ax=ax)\n",
    "\n",
    "# Augmenter la taille de la police de tous les textes\n",
    "heatmap.set_xticklabels(heatmap.get_xticklabels(), fontsize=70)\n",
    "heatmap.set_yticklabels(heatmap.get_yticklabels(), fontsize=70)\n",
    "heatmap.set_title(\"Heatmap de la matrice de corrélation filtrée\", fontsize=150)\n",
    "\n",
    "# Augmenter la taille de la police des nombres à l'intérieur de la heatmap\n",
    "for text in heatmap.texts:\n",
    "    text.set_fontsize(50)\n",
    "    text.set_weight('bold')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2141b3cb",
   "metadata": {},
   "source": [
    "Dans ce notebook, j'ai principalement réalisé des études exploratoires pour découvrir les relations entre les différentes colonnes et extraire les données les plus pertinentes à utiliser dans la section de prédiction. J'ai sélectionné la section des bâtiments non résidentiels, nettoyé les données, identifié et supprimé les valeurs aberrantes. Enfin, j'ai travaillé sur la partie de l'ingénierie des caractéristiques afin d'obtenir des données utiles que les méthodes d'apprentissage automatique peuvent bien comprendre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87662bce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enregistrer le DataFrame nettoyé dans un fichier CSV\n",
    "\n",
    "df.to_csv('data_clean.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b054273e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}