03-formatage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a950b321-88cf-435d-a0b7-fcadc1b60c52",
   "metadata": {},
   "source": [
    "# Analyse et visualisation de données avec Python\n",
    "## Formatage des données\n",
    "Questions\n",
    "* Quels sont les différents types de données dans Pandas?\n",
    "* Quel est l'impact des types sur les statistiques descriptives?\n",
    "* Comment gérer les valeurs non définies ou nulles?\n",
    "* Comment sauvegarder un dataframe?\n",
    "\n",
    "Objectifs\n",
    "* Manipuler les types de données\n",
    "* Créer une copie d'un dataframe.\n",
    "* Transformer ou éliminer les valeurs non définies ou nulles.\n",
    "* Écrire les données manipulées dans un fichier CSV."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c4467ef-81a7-4590-8f65-15f7bbecbdfc",
   "metadata": {},
   "source": [
    "## Charger nos données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efe856c7-f8a1-47a5-9602-e582f0dec6f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Charger le module pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Charger les données\n",
    "surveys_df = pd.read_csv(\"../data/surveys.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7d2ddc2-b7c6-4717-8762-e976288cc446",
   "metadata": {},
   "source": [
    "## Types de données\n",
    "### Vérifier le type de données pour chaque colonne"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "315c4096-65e3-4590-b172-3532e38b4a4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtenir le type de données pour chaque colonne\n",
    "surveys_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "354050a2-7051-4a65-86e7-72b117c28856",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Le type de données pour la colonne des mois\n",
    "surveys_df['month'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6274bed8-3018-4390-92a7-72e571cc0ca5",
   "metadata": {},
   "source": [
    "Types Python | Types Pandas | Description\n",
    ":-----------:|:------------:|:-----------\n",
    "`str`        | `object`     | Type générique, aussi utilisé en cas de multiples types\n",
    "`int`        | `int64`      | Nombres entiers représentés avec 64 bits\n",
    "`float`      | `float64`    | Nombres réels représentés avec 64 bits, ou non-définis (NaN)\n",
    " N/A         | `datetime64` | Dates et heures, avec une précision allant jusqu'à la nanoseconde"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef2a5100-1723-4284-abf7-7dd93a2af0f4",
   "metadata": {},
   "source": [
    "### Impacts sur les statistiques descriptives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f38a443-1268-48e7-9cb5-ab1f0afd9d73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Statistiques descriptives sur les valeurs numériques\n",
    "surveys_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9e8d5e2-0cea-4865-a0be-e5476bc23abb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convertir les numéros de mois en catégories\n",
    "surveys_df['month'] = surveys_df['month'].astype('str')\n",
    "surveys_df['month'].dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c66f5ba-ca6e-4078-ac29-ff8d76cb46de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Statistiques descriptives sur des catégories\n",
    "surveys_df['month'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e05ac7f-9e60-436e-a797-eae69b7a57a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lister les différents mois\n",
    "surveys_df['month'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02888643-2875-4520-88fc-e7356c5a1446",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lister les différentes années\n",
    "surveys_df['year'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dac99fd-78bc-4758-ad8d-8a1fbb244067",
   "metadata": {},
   "source": [
    "### Exercices - Types et statistiques descriptives"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "823aeeb7-60de-493e-be65-75ebb7b01d08",
   "metadata": {},
   "source": [
    "`1`. Convertir les valeurs de `weight` en entiers `int64` produira une erreur. Pourquoi?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "978b68b7-4045-4e37-b4f3-73f25801f9b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    surveys_df['weight'].astype('int64')\n",
    "except BaseException as erreur:\n",
    "    print(f'La raison : {erreur}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d133b88b-0605-4d1e-8505-43655a00d8a6",
   "metadata": {},
   "source": [
    "`2`. Essayez de convertir la colonne `plot_id` au type Python `float`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9369602e-866a-49f9-adeb-f37b23573f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['plot_id']\n",
    "surveys_df['plot_id'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90bf1bf1-01bf-46d5-8ab4-715b50331e3b",
   "metadata": {},
   "source": [
    "## Sélection et nettoyage des valeurs non définies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ef5a350-02f6-42f5-a543-5d3bf7f4c035",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pour chaque valeur, déterminer si non définie\n",
    "surveys_df.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "978a7ee5-d1d3-403e-b1dd-0b159d92c8c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sélectionner les enregistrements ayant au moins une valeur NaN\n",
    "masque_nan = surveys_df.isnull().any(axis=1)\n",
    "surveys_df[masque_nan]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a45916ae-280e-4aec-a011-4185afebdf01",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Qu'est-ce que le code suivant va retourner?\n",
    "masque_nan = surveys_df['weight'].isnull()\n",
    "une_selection = surveys_df[masque_nan]\n",
    "une_selection.groupby('species_id')['record_id'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b64954-6a10-45ac-9719-c5b6f714c477",
   "metadata": {},
   "source": [
    "### Recréer des données manquantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15bd3b23-44b6-4619-bbeb-5f0c49589309",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Le \"avant\" - nombre et moyenne\n",
    "print(surveys_df['weight'].count(), surveys_df['weight'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "661bb40d-fee2-47f5-94e0-37c81dc8f430",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Créer une copie pour ne pas modifier l'objet original\n",
    "copie_surveys = surveys_df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b019da7-f981-4a3f-bc7d-530cbe5a0432",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pour une valeur moyenne stable\n",
    "moyennePoids = copie_surveys['weight'].mean()\n",
    "copie_surveys['weight'] = copie_surveys['weight'].fillna(moyennePoids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1001f7d6-68ef-4c24-92eb-d03947e9ab4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Le \"après\" - nombre et moyenne\n",
    "print(copie_surveys['weight'].count(), copie_surveys['weight'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0bc3799-d670-4570-90bd-508ccd6d6f50",
   "metadata": {},
   "source": [
    "### Exercices - Types et statistiques descriptives\n",
    "\n",
    "À partir de `copie_surveys`,\n",
    "convertissez les valeurs de `weight` en entiers `int64`.\n",
    "Nous verrons ensuite le changement sur la moyenne."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "133cf6a8-03ff-4841-abc3-ac5a2770bcff",
   "metadata": {},
   "outputs": [],
   "source": [
    "copie_surveys['weight'] =\n",
    "copie_surveys['weight'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef805114-f932-49a4-9e2e-cdc2aac20e12",
   "metadata": {},
   "source": [
    "### Sauvegarde après nettoyage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca36b480-3d67-41ea-a9b0-ff9e39f5e902",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ne garder que les lignes sans aucun NA\n",
    "df_sans_na = surveys_df.dropna()\n",
    "df_sans_na"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b08ad6f-ed02-4cf1-b16c-610d63b3d8ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sauvegarder le dataframe filtré\n",
    "fichier_csv = 'surveys_sans_NA.csv'\n",
    "df_sans_na.to_csv(fichier_csv, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44166511-3221-4e97-929d-a8755ea37027",
   "metadata": {},
   "source": [
    "## Résumé technique\n",
    "* **Gestion des types**\n",
    "    * Pour un **DataFrame** :\n",
    "        * Attributs : `dtypes`\n",
    "    * Pour une **série** (colonne) :\n",
    "        * Attributs : `dtype`\n",
    "        * Méthodes : `astype()`\n",
    "* **Nettoyage**\n",
    "    * `df.copy()`\n",
    "    * `isna()`, `isnull()` (le second est un alias du premier)\n",
    "    * `notna()`, `notnull()`  (le second est un alias du premier)\n",
    "    * `colonne.fillna(valeur, inplace=True)`\n",
    "* **Sauvegarde**\n",
    "    * `df.to_csv(nom_csv, index=False)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c170a8b7-ddff-4213-806b-33895f16ae8d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}