1-Data_Exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7fbd1e73-f67a-4c1e-b93e-a62ed16d936d",
   "metadata": {},
   "source": [
    "# Notebook 1: Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b627147d-20ff-4980-8573-976f9d53be25",
   "metadata": {},
   "source": [
    "Welcome!, The main goal for this notebook is to provide a deep understanding of the dataset, its characteristics, and any hidden patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afbe996e-90bc-45c1-b336-29cff5865da4",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69ccfaad-bbdf-438c-a705-2a51532e00ee",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b136bff0-b85f-42c1-b51b-9cd936a2056a",
   "metadata": {},
   "source": [
    "#### Data Description"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c78a0cad-ef3d-42f9-ace9-4e990769345e",
   "metadata": {},
   "source": [
    "This project analyzes radio continuum data from the MeerKAT International GHz Tiered Extragalactic Exploration ([MIGHTEE](https://idia.ac.za/mightee/)) survey, focusing on extragalactic radio emissions in the COSMOS field. MIGHTEE offers insights into galaxy evolution, given its depth and multiwavelength coverage. The COSMOS field, devoid of bright radio, UV, and X-ray sources, is ideal for deep astronomical surveys."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fabf8c96-5a70-4462-a600-73d68200b3b6",
   "metadata": {},
   "source": [
    "The MIGHTEE data includes deep fields like COSMOS, XMM-LSS, E-CDFS and ELAIS-S1, covering a 20 deg^2 total area at a central frequency of 1284 MHz. These fields were observed using MeerKAT's L-band receivers for about 1000 hours. They were chosen due to their extensive multiwavelength data from prior and potential future surveys."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e71d2905-2936-4311-bd92-3e4f03ffc5f5",
   "metadata": {},
   "source": [
    "The **MIGHTEE-COSMOS multiwavelength catalogue** we will use here combines radio data with optical, near-to-far infrared, and X-ray measurements from various sources in the COSMOS field. The main radio catalogue was cross-matched with optical and near-infrared data from other studies, resulting in a matched set of 5,224 radio sources. Additionally, 572 of these sources were detected in X-ray observations. This dataset also incorporated Mid-Infrared (MIR) data, referencing specific IRAC flux densities. The catalogue further integrated far-infrared data from the Herschel Extra-galactic Legacy Project, identifying thousands of radio sources across different instruments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efcdfa57-113d-4d3e-9b85-1db481eaaca4",
   "metadata": {},
   "source": [
    "#### Why explore the data?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b9bd7fe-cf9e-42d3-94f0-c67a8caea716",
   "metadata": {},
   "source": [
    "Data exploration serves as the foundational step in data analysis, aimed at understanding inherent structures, patterns, and potential anomalies within a dataset. It helps ascertain the quality of data, understand features' distributions, formulate hypotheses, guide preprocessing decisions, and select appropriate analytical tools and models. Efficient data exploration also aids in communicating initial insights to stakeholders. In this notebook, we will perform some of these critical tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69322266-5441-47f9-ad47-301bc8366659",
   "metadata": {
    "tags": []
   },
   "source": [
    "<div align=\"center\">\n",
    "  <img src=\"images/cweb.jpg\" width=\"400\" height=\"400\">\n",
    "    <p><center>Cosmic Web</center></p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "144a1eea-7557-4300-87f5-72e7f35452fa",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e6fd8aa-525f-49dc-8879-3433dd20fefa",
   "metadata": {},
   "source": [
    "### First we import some libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7625606-747f-49e5-8da1-959ddd575855",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import matplotlib.pyplot as plt \n",
    "import pandas as pd \n",
    "import astropy.io.fits as fits \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "973406ae-3a62-4152-90e2-91bb5e1a360e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "837bb10d-6806-4a95-bc39-3b99235aa961",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Reading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a06531c-a09b-4b22-b206-4fddebc42ee1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "mightee_data = pd.read_pickle(\"data/mightee.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d78053f3-8d9d-4e1d-9ef6-b90e49922dae",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mightee_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19462e37-b36d-47d6-a57b-669b6ec79ebe",
   "metadata": {},
   "source": [
    "From what we can observe, we have 144 features (columns) and 2824 rows. The following are the feature names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97cb4be2-ab5a-4764-9de2-f76e76689b2f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "list(mightee_data.columns[:10]) # printing the first 10 columns\n",
    "# For feature description please check the following file @@ data/README_MIGHTEE.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb74fbca-5f6b-4228-9429-7ef851dde077",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(mightee_data.isna().any().any()) # checking if the dataframe has any missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a07a42b-1887-41c7-bb35-d35b3f63be9e",
   "metadata": {},
   "source": [
    "The dataframe seems to be clean, because someone did the cleaning for you, You guys are more than welcome :D"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e1ba206-0519-4cc6-8a49-628cd492c966",
   "metadata": {},
   "source": [
    "### Exercise \n",
    "\n",
    "- remove the object coordinates from the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76b8b82d-ae43-4402-a722-4722e992371c",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Answer here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45629566-b98e-48b7-9370-a4c7df14d021",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3a17bd-7350-4a52-a7b3-7215756b176b",
   "metadata": {},
   "source": [
    "### Descriptive Statistics:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b54caa9-140c-458b-ba24-8ad60e2faf07",
   "metadata": {},
   "source": [
    "**Descriptive statistics** provide a concise overview of the central tendencies, spread, and shape of the data distribution. By observing the mean, median, mode, and standard deviation, we can quickly grasp the characteristics of our dataset and identify potential anomalies or patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2111bad9-663c-40ee-bbfb-219ebc4f6ea8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "desc_stats = mightee_data.describe()\n",
    "\n",
    "# Display the statistics\n",
    "desc_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc968479-94e3-4b5b-aad0-a8f01a1f8337",
   "metadata": {},
   "source": [
    "### Data Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4c96a96-1b15-4148-a8b3-8d436101600d",
   "metadata": {},
   "source": [
    "From our descriptive statistics and visual data exploration, we notice discrepancies in the scale of our features. Different scales can skew our data visualization and mask meaningful patterns. By normalizing, we rescale features to the range `[0,1]`, ensuring uniformity for better data visualization and insights.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0722f58-17ef-45d0-adde-cf15ec5578d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "normalized_mightee = scaler.fit_transform(mightee_data)\n",
    "normalized_mightee = pd.DataFrame(normalized_mightee, columns=mightee_data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af600115-8dd1-462e-b3c8-270f89dad766",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "normalized_mightee.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08de3736-10e5-4704-8b16-8a4216b7f18d",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f63c85fd-976e-4281-b501-d8f4811fa9ee",
   "metadata": {},
   "source": [
    "### Freature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b4e280f-da12-4c3c-bb3a-c198e33f3762",
   "metadata": {},
   "source": [
    "Having 147 features can be overwhelming for visualization and clustering. To simplify our dataset and remove redundant or highly correlated features, we can utilize correlation analysis. By examining how features correlate with one another, we can identify and remove those that offer similar information, thus streamlining our dataset for more effective analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "108aae15-79ab-4e49-9092-d3521bd19d94",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "correlation_matrix = mightee_data.corr()\n",
    "upper_tri = correlation_matrix.where(np.triu(np.ones(correlation_matrix.shape), k=1).astype(bool))\n",
    "\n",
    "# Find the index of feature columns with a correlation greater than 0.80\n",
    "to_drop = [column for column in upper_tri.columns if any(upper_tri[column] > 0.80)]\n",
    "\n",
    "mightee_data.drop(columns=to_drop, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "712b9bde-7660-4fa9-8974-070f1a2dddfb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mightee_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e31d8e72-68f1-4009-bc27-86ccb3d06ce1",
   "metadata": {
    "tags": []
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8d3f816-48d1-4831-aa65-ab31b60e7142",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Data Visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d35e645-c566-4e94-84d0-08af248931d9",
   "metadata": {},
   "source": [
    "Well, I will let you explore the all 46 features, but I am going to explore only 3:\n",
    "\n",
    "- qir\n",
    "- Mstar\n",
    "- COS_best_z_v5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05acbe21-15a1-4bc3-810b-759b62fd187f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "# Settings for better visualization\n",
    "sns.set(font_scale=1)\n",
    "sns.set_style('ticks')\n",
    "\n",
    "# Using pairplot with hue as 'class_star'\n",
    "sns.pairplot(mightee_data[['Mstar', 'qir', 'COS_best_z_v5']], hue='COS_best_z_v5')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0f1088b-c702-41f7-a12f-4311212509b2",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d1a7dfdb-b719-445a-9cc3-38432b4a49ee",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6bf59b6-ab8c-4c00-bbfa-85faf8e2c29e",
   "metadata": {},
   "source": [
    "### Saving the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "082ff8c1-9b67-40bf-832d-a8112c0cbbe6",
   "metadata": {},
   "source": [
    "Now, we will be using the above three features to perform some clustering in the next notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77b88a3f-ab0b-491d-a22d-696e2eae1ef3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open(\"mightee3Feat.pkl\", \"wb\") as file:\n",
    "    pickle.dump(mightee_data[['Mstar', 'qir', 'COS_best_z_v5']], file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mightee_kerV",
   "language": "python",
   "name": "mightee_kerv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}