notebooks/load_corrupt_and_test_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from jenga.corruptions.generic import CategoricalShift, SwappedValues\n",
    "from jenga.corruptions.numerical import GaussianNoise, Scaling\n",
    "\n",
    "from conformal_data_cleaning.config import error_fractions\n",
    "from conformal_data_cleaning.data import (\n",
    "    _ID_TO_TASK_TYPE,\n",
    "    AVAILABLE_DATASETS,\n",
    "    fetch_and_save_dataset,\n",
    "    get_X_y_paths,\n",
    "    read_dataset,\n",
    ")\n",
    "from conformal_data_cleaning.jenga_extension import get_OpenMLTask"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and split datasets\n",
    "\n",
    "This downloads and splits the datasets in [`data/training`](../data/training/) and [`data/test`](../data/test/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for task_id in AVAILABLE_DATASETS:\n",
    "    if not fetch_and_save_dataset(task_id=task_id):\n",
    "        print(f\"Downloading dataset with ID {task_id:>5} failed.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corrupt the datasets\n",
    "\n",
    "Use `jenga` to create the corruptions. We train on clean data and simulate how corrupted data impacts the downstream task performance. For this, we load the test sets from [`data/test`](../data/test/) again, create errors, and save them to [`data/corrupted/<corruption type>/<fraction of errors>`](../data/corrupted/) accordingly to their corruption types and fraction of errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corruptions = [Scaling, GaussianNoise, SwappedValues, CategoricalShift]\n",
    "\n",
    "errors_list = []\n",
    "\n",
    "for corruption in corruptions:\n",
    "    for fraction in error_fractions:\n",
    "        for dataset_id in AVAILABLE_DATASETS:\n",
    "            X, y = read_dataset(dataset_id, training=False)\n",
    "            X_corrupted = X.copy()\n",
    "\n",
    "            for column in X_corrupted.columns:\n",
    "                corruption_instance = corruption(column=column, fraction=fraction)\n",
    "                X_corrupted = corruption_instance.transform(X_corrupted)\n",
    "\n",
    "            percent_errors = (X_corrupted != X).sum().sum() / X_corrupted.size\n",
    "            errors_list.append(\n",
    "                (\n",
    "                    dataset_id,\n",
    "                    _ID_TO_TASK_TYPE[dataset_id].value,\n",
    "                    fraction,\n",
    "                    type(corruption_instance).__name__,\n",
    "                    percent_errors,\n",
    "                ),\n",
    "            )\n",
    "\n",
    "            X_path, y_path = get_X_y_paths(\n",
    "                dataset_id,\n",
    "                False,\n",
    "                corruption=type(corruption_instance).__name__,\n",
    "                fraction=fraction,\n",
    "            )\n",
    "            X_corrupted.to_csv(X_path, index=False)\n",
    "            y.to_csv(y_path, index=False)\n",
    "\n",
    "            assert X_path.exists() and X_path.is_file()\n",
    "            assert y_path.exists() and y_path.is_file()\n",
    "\n",
    "errors = pd.DataFrame(errors_list, columns=[\"dataset_id\", \"dataset_type\", \"fraction\", \"error_type\", \"percent_errors\"])\n",
    "errors.to_csv(\"../data/corrupted/error_statistics.csv\", index=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test downloaded datasets\n",
    "\n",
    "Since `jenga` tests whether or not the loaded data correspond to the given type, we once create all `OpenMLTask` objects.\n",
    "\n",
    "This should finish without an error and present 30 available datasets. These are a subset from this benchmark paper: [https://www.frontiersin.org/articles/10.3389/fdata.2021.693674/full](https://www.frontiersin.org/articles/10.3389/fdata.2021.693674/full).\n",
    "\n",
    "- min 50k cells\n",
    "- fewer columns are better\n",
    "\n",
    "### Clean Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_types = {}\n",
    "for task_id in AVAILABLE_DATASETS:\n",
    "    open_ml_task = get_OpenMLTask(task_id=task_id)\n",
    "    number_of_types[type(open_ml_task).__name__] = number_of_types.get(type(open_ml_task).__name__, 0) + 1\n",
    "\n",
    "for type_, number in number_of_types.items():\n",
    "    print(f\"- {number} {type_}\")\n",
    "\n",
    "total_number_of_datasets = sum(number_of_types.values())\n",
    "print(f\"=> in total {total_number_of_datasets} datasets.\")\n",
    "print()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corrupted Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "how_many_corrupted_versions = {}\n",
    "for corruption in corruptions:\n",
    "    for fraction in error_fractions:\n",
    "        for task_id in AVAILABLE_DATASETS:\n",
    "            open_ml_task = get_OpenMLTask(task_id=task_id, corruption=corruption.__name__, fraction=fraction)\n",
    "            how_many_corrupted_versions[task_id] = how_many_corrupted_versions.get(task_id, 0) + 1\n",
    "\n",
    "assert set(how_many_corrupted_versions.values()) == set(\n",
    "    [len(corruptions) * len(error_fractions)],\n",
    "), \"Something went wrong during corruption of the data. Expecting 20 versions (with corruptions) of each dataset.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "equal_dtypes = []\n",
    "for task_id in AVAILABLE_DATASETS:\n",
    "    for corruption in corruptions:\n",
    "        for fraction in error_fractions:\n",
    "            open_ml_task = get_OpenMLTask(task_id=task_id, corruption=corruption.__name__, fraction=fraction)\n",
    "            open_ml_task_orig = get_OpenMLTask(task_id=task_id)\n",
    "\n",
    "            if open_ml_task.train_data.dtypes.to_dict() != open_ml_task_orig.train_data.dtypes.to_dict():\n",
    "                equal_dtypes.append(False)\n",
    "\n",
    "            elif open_ml_task.test_data.dtypes.to_dict() != open_ml_task_orig.test_data.dtypes.to_dict():\n",
    "                equal_dtypes.append(False)\n",
    "\n",
    "            else:\n",
    "                equal_dtypes.append(True)\n",
    "\n",
    "assert all(equal_dtypes), \"Corruptions should not change the data types as this can cause downstream issues!\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_statistics = pd.read_csv(\"../data/corrupted/error_statistics.csv\")\n",
    "\n",
    "error_statistics[\"delta_error\"] = error_statistics[\"percent_errors\"] - error_statistics[\"fraction\"]\n",
    "\n",
    "error_statistics_grouped = error_statistics.groupby([\"fraction\", \"error_type\"])\n",
    "mean_errors = error_statistics_grouped.mean()[[\"percent_errors\", \"delta_error\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_errors"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating 30% `GaussianNoise` only leads to 30% changed values if all columns are numerical. If this is not the case, we will see less `percent_errors`. This is why we `GaussianNoise` to each column if possible. However, it only can be applied to numerical columns.\n",
    "Others, e.g., `SwappedValues`, need at least two columns of the same `dtype` or categorical columns, e.g., `CategoricalShift`."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "d1e079ba1052bcc85be9b78b3c72bf03a4d015549529e67f43de37f743ce1597"
  },
  "kernelspec": {
   "display_name": "Python 3.9.12 ('conformal-data-cleaning-z_iY5k30-py3.9')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}