how-to-make-spectrogram-from-eeg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4228ba26",
   "metadata": {
    "papermill": {
     "duration": 0.005263,
     "end_time": "2024-02-05T20:28:01.115221",
     "exception": false,
     "start_time": "2024-02-05T20:28:01.109958",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# How To Make Spectrograms from EEG\n",
    "In this notebook, we learn how to make spectrograms from EEG. The EEGs are waveforms and the Spectrograms are images. There is a discussion about this notebook [here][1].\n",
    "\n",
    "In version 1-3, we also train a simple model using our EEG spectrograms to confirm that they work well. We observe that a model trained with EEG spectrograms performs better than baseline models using only train means.\n",
    "\n",
    "# Exciting UPDATE!\n",
    "Version 4 of this notebook uses a different formula to make spectrograms than earlier versions. I trained an EfficientNet model using the old version eeg spectrograms, new version spectrograms, and Kaggle spectrograms. We can see that the new version eeg spectrograms are **powerful**!\n",
    "\n",
    "| Spectrogram | 5-Fold CV | LB |\n",
    "| --- | --- | --- |\n",
    "| Kaggle spectrogram | 0.73 | 0.57 |\n",
    "| Old EEG formula | 0.84 on fold 1 | ?? |\n",
    "| New EEG formula | 0.70 on fold 1| ?? |\n",
    "| Use both Kaggle and New | 0.64 | 0.44 |\n",
    "\n",
    "From the results above, we conclude that our new formula is probably similar or better than the true formula used to create the Kaggle spectrograms. Details about the old and new formula are in the next notebook section. \n",
    "\n",
    "# How To Use EEG Spectrograms\n",
    "Examples of how to use new EEG spectrograms to boost CV score and LB score will be (or already are) published in recent versions of my EfficientNet starter notebook [here][2] and CatBoost starter notebook [here][3]\n",
    "\n",
    "# Kaggle Dataset\n",
    "The new EEG spectrograms from version 4 of this notebook have been uploaded to a Kaggle dataset [here][4]. We can attach this Kaggle dataset to our future notebooks to boost our CV scores and LB scores! Thank you everyone for upvoting my new EEG spectrogram Kaggle dataset!\n",
    "\n",
    "[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\n",
    "[2]: https://www.kaggle.com/code/cdeotte/efficientnetb2-starter-lb-0-57\n",
    "[3]: https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-67\n",
    "[4]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2819d640",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "papermill": {
     "duration": 1.585232,
     "end_time": "2024-02-05T20:28:02.705515",
     "exception": false,
     "start_time": "2024-02-05T20:28:01.120283",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train shape (106800, 15)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eeg_id</th>\n",
       "      <th>eeg_sub_id</th>\n",
       "      <th>eeg_label_offset_seconds</th>\n",
       "      <th>spectrogram_id</th>\n",
       "      <th>spectrogram_sub_id</th>\n",
       "      <th>spectrogram_label_offset_seconds</th>\n",
       "      <th>label_id</th>\n",
       "      <th>patient_id</th>\n",
       "      <th>expert_consensus</th>\n",
       "      <th>seizure_vote</th>\n",
       "      <th>lpd_vote</th>\n",
       "      <th>gpd_vote</th>\n",
       "      <th>lrda_vote</th>\n",
       "      <th>grda_vote</th>\n",
       "      <th>other_vote</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1628180742</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>353733</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>127492639</td>\n",
       "      <td>42516</td>\n",
       "      <td>Seizure</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1628180742</td>\n",
       "      <td>1</td>\n",
       "      <td>6.0</td>\n",
       "      <td>353733</td>\n",
       "      <td>1</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3887563113</td>\n",
       "      <td>42516</td>\n",
       "      <td>Seizure</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1628180742</td>\n",
       "      <td>2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>353733</td>\n",
       "      <td>2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1142670488</td>\n",
       "      <td>42516</td>\n",
       "      <td>Seizure</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1628180742</td>\n",
       "      <td>3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>353733</td>\n",
       "      <td>3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2718991173</td>\n",
       "      <td>42516</td>\n",
       "      <td>Seizure</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1628180742</td>\n",
       "      <td>4</td>\n",
       "      <td>24.0</td>\n",
       "      <td>353733</td>\n",
       "      <td>4</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3080632009</td>\n",
       "      <td>42516</td>\n",
       "      <td>Seizure</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       eeg_id  eeg_sub_id  eeg_label_offset_seconds  spectrogram_id  \\\n",
       "0  1628180742           0                       0.0          353733   \n",
       "1  1628180742           1                       6.0          353733   \n",
       "2  1628180742           2                       8.0          353733   \n",
       "3  1628180742           3                      18.0          353733   \n",
       "4  1628180742           4                      24.0          353733   \n",
       "\n",
       "   spectrogram_sub_id  spectrogram_label_offset_seconds    label_id  \\\n",
       "0                   0                               0.0   127492639   \n",
       "1                   1                               6.0  3887563113   \n",
       "2                   2                               8.0  1142670488   \n",
       "3                   3                              18.0  2718991173   \n",
       "4                   4                              24.0  3080632009   \n",
       "\n",
       "   patient_id expert_consensus  seizure_vote  lpd_vote  gpd_vote  lrda_vote  \\\n",
       "0       42516          Seizure             3         0         0          0   \n",
       "1       42516          Seizure             3         0         0          0   \n",
       "2       42516          Seizure             3         0         0          0   \n",
       "3       42516          Seizure             3         0         0          0   \n",
       "4       42516          Seizure             3         0         0          0   \n",
       "\n",
       "   grda_vote  other_vote  \n",
       "0          0           0  \n",
       "1          0           0  \n",
       "2          0           0  \n",
       "3          0           0  \n",
       "4          0           0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd, numpy as np, os\n",
    "import matplotlib.pyplot as plt, gc\n",
    "\n",
    "train = pd.read_csv('/root/autodl-tmp/hms-harmful-brain-activity-classification/train.csv')\n",
    "print('Train shape', train.shape )\n",
    "display( train.head() )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f646462",
   "metadata": {
    "papermill": {
     "duration": 0.005011,
     "end_time": "2024-02-05T20:28:02.715899",
     "exception": false,
     "start_time": "2024-02-05T20:28:02.710888",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# The Bipolar Double Banana Montage\n",
    "In the Kaggle discussion [here][1], we learn what information we need to make spectrograms from eegs. The following website [here][2] is helpful also. To build 1 spectrogram, we need 1 time series signal. Kaggle provides us with 19 eeg time signals, so we must combine them into 4 time signals to make 4 spectrograms.\n",
    "\n",
    "In the diagram below, we see which electrode signals are needed to make the `LL, LP, RP, RR` spectrograms. Furthermore Kaggle discussions imply that most likely we create differences between consecutive electrodes and average the differences. For example, we create `LL spectrogram` with the formula: \n",
    "\n",
    "    LL = ( (Fp1 - F7) + (F7 - T3) + (T3 - T5) + (T5 - O1) )/4.\n",
    "    \n",
    "I am not positive that this is the correct formula. I also tried the formula below but it produced a worse CV score than the above formula, so perhaps the above is correct. I am confident that we only use these 5 electrodes to create `LL spectrogram`. I'm just a little unsure about the formula:\n",
    "\n",
    "    LL = ( Fp1 + F7 + T3 + T5 + O1 )/5.\n",
    "    \n",
    "# Exciting UPDATE!\n",
    "I believe the above two formulas are wrong. Many Kagglers have pointed out that the above formula reduces to `LL = ( Fp1 - O1 )/4` which means that it does not use all the EEG signals. The new formula below utilizes all the EEG signals and produces EEG spectrograms that achieve better CV score and LB score than the Kaggle spectrograms. Therefore I think the following formula is the correct one:\n",
    "\n",
    "    LL Spec = ( spec(Fp1 - F7) + spec(F7 - T3) + spec(T3 - T5) + spec(T5 - O1) )/4.\n",
    "    \n",
    "Since creating a spectrogram is a non-linear operation, the above formula which computes 4 spectrograms and then takes the average is different than the formula below which computes 1 spectrogam. And the above formula does utilize all EEG signals and cannot be reduced to a shorter formula (like the one below).\n",
    "\n",
    "    LL Spec = spec( ( (Fp1 - F7) + (F7 - T3) + (T3 - T5) + (T5 - O1) )/4. )\n",
    "\n",
    "![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/montage.png)\n",
    "\n",
    "[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\n",
    "[2]: https://www.learningeeg.com/montages-and-technical-components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ab451e49",
   "metadata": {
    "papermill": {
     "duration": 0.018126,
     "end_time": "2024-02-05T20:28:02.739374",
     "exception": false,
     "start_time": "2024-02-05T20:28:02.721248",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "NAMES = ['LL','LP','RP','RR']\n",
    "\n",
    "FEATS = [['Fp1','F7','T3','T5','O1'],\n",
    "         ['Fp1','F3','C3','P3','O1'],\n",
    "         ['Fp2','F8','T4','T6','O2'],\n",
    "         ['Fp2','F4','C4','P4','O2']]\n",
    "\n",
    "directory_path = 'EEG_Spectrograms/'\n",
    "if not os.path.exists(directory_path):\n",
    "    os.makedirs(directory_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe4d7061",
   "metadata": {
    "papermill": {
     "duration": 0.004974,
     "end_time": "2024-02-05T20:28:02.749681",
     "exception": false,
     "start_time": "2024-02-05T20:28:02.744707",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Optional Signal Denoising with Wavelet transform\n",
    "We can optionally denoise the signal before creating the spectrogram. I'm not sure yet if this creates better or worse spectrograms. We can experiment with this. This code comes from Yusaku5738 notebook [here][1] and was suggested by SeshuRajuP in the comments. We have many parent functions to use for denoising. Yusaku5738 suggests using `wavelet = db8`.\n",
    "\n",
    "[1]: https://www.kaggle.com/code/yusaku5739/eeg-signal-denosing-using-wavelet-transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2ad51366",
   "metadata": {
    "papermill": {
     "duration": 0.596149,
     "end_time": "2024-02-05T20:28:03.351254",
     "exception": false,
     "start_time": "2024-02-05T20:28:02.755105",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The wavelet functions we can use:\n",
      "['bior1.1', 'bior1.3', 'bior1.5', 'bior2.2', 'bior2.4', 'bior2.6', 'bior2.8', 'bior3.1', 'bior3.3', 'bior3.5', 'bior3.7', 'bior3.9', 'bior4.4', 'bior5.5', 'bior6.8', 'cgau1', 'cgau2', 'cgau3', 'cgau4', 'cgau5', 'cgau6', 'cgau7', 'cgau8', 'cmor', 'coif1', 'coif2', 'coif3', 'coif4', 'coif5', 'coif6', 'coif7', 'coif8', 'coif9', 'coif10', 'coif11', 'coif12', 'coif13', 'coif14', 'coif15', 'coif16', 'coif17', 'db1', 'db2', 'db3', 'db4', 'db5', 'db6', 'db7', 'db8', 'db9', 'db10', 'db11', 'db12', 'db13', 'db14', 'db15', 'db16', 'db17', 'db18', 'db19', 'db20', 'db21', 'db22', 'db23', 'db24', 'db25', 'db26', 'db27', 'db28', 'db29', 'db30', 'db31', 'db32', 'db33', 'db34', 'db35', 'db36', 'db37', 'db38', 'dmey', 'fbsp', 'gaus1', 'gaus2', 'gaus3', 'gaus4', 'gaus5', 'gaus6', 'gaus7', 'gaus8', 'haar', 'mexh', 'morl', 'rbio1.1', 'rbio1.3', 'rbio1.5', 'rbio2.2', 'rbio2.4', 'rbio2.6', 'rbio2.8', 'rbio3.1', 'rbio3.3', 'rbio3.5', 'rbio3.7', 'rbio3.9', 'rbio4.4', 'rbio5.5', 'rbio6.8', 'shan', 'sym2', 'sym3', 'sym4', 'sym5', 'sym6', 'sym7', 'sym8', 'sym9', 'sym10', 'sym11', 'sym12', 'sym13', 'sym14', 'sym15', 'sym16', 'sym17', 'sym18', 'sym19', 'sym20']\n"
     ]
    }
   ],
   "source": [
    "import pywt\n",
    "print(\"The wavelet functions we can use:\")\n",
    "print(pywt.wavelist())\n",
    "\n",
    "USE_WAVELET = None #or \"db8\" or anything below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a6862054",
   "metadata": {
    "papermill": {
     "duration": 0.020506,
     "end_time": "2024-02-05T20:28:03.377896",
     "exception": false,
     "start_time": "2024-02-05T20:28:03.357390",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# DENOISE FUNCTION\n",
    "def maddest(d, axis=None):\n",
    "    return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n",
    "\n",
    "def denoise(x, wavelet='haar', level=1):    \n",
    "    coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n",
    "    sigma = (1/0.6745) * maddest(coeff[-level])\n",
    "\n",
    "    uthresh = sigma * np.sqrt(2*np.log(len(x)))\n",
    "    coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n",
    "\n",
    "    ret=pywt.waverec(coeff, wavelet, mode='per')\n",
    "    \n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26a0411d",
   "metadata": {
    "papermill": {
     "duration": 0.005123,
     "end_time": "2024-02-05T20:28:03.388598",
     "exception": false,
     "start_time": "2024-02-05T20:28:03.383475",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Create Spectrograms with Librosa\n",
    "We can use library librosa to create spectrograms. We will save them to disk. For each `eeg_id` we will make 1 spectrogram from the middle 50 seconds. We don't want to use more information than 50 seconds at a time because during test inference, we only have access to 50 seconds of EEG for each test `eeg_id`. We will create spectrograms of `size = 128x256 (freq x time)`.\n",
    "\n",
    "The main function is \n",
    "\n",
    "    mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//256, \n",
    "              n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n",
    "              \n",
    "Let's explain these variables.\n",
    "* `y` is the input time series signal\n",
    "* `sr` is the sampling frequency. In this competition EEG is sample 200 times per sec\n",
    "* `hop_length` produces image with `width = len(x)/hop_length`\n",
    "* `n_fft` controls vertical resolution and quality of spectrogram\n",
    "* `n_mels` produces image with `height = n_mels`\n",
    "* `fmin` is smallest frequency in our spectrogram\n",
    "* `fmax` is largest frequency in our spectrogram\n",
    "* `win_length` controls hortizonal resolution and quality of spectrogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7d02d640",
   "metadata": {
    "papermill": {
     "duration": 0.041649,
     "end_time": "2024-02-05T20:28:03.435593",
     "exception": false,
     "start_time": "2024-02-05T20:28:03.393944",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import librosa\n",
    "from scipy.signal import butter, lfilter\n",
    "def butter_bandpass(lowcut, highcut, fs, order=5):\n",
    "    return butter(order, [lowcut, highcut], fs=fs, btype=\"band\")\n",
    "\n",
    "def butter_bandpass_filter(data, lowcut, highcut, fs, order=5):\n",
    "    b, a = butter_bandpass(lowcut, highcut, fs, order=order)\n",
    "    y = lfilter(b, a, data)\n",
    "    return y\n",
    "\n",
    "\n",
    "def spectrogram_from_eeg(parquet_path, display=False):\n",
    "    \n",
    "    # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n",
    "    eeg = pd.read_parquet(parquet_path)\n",
    "    middle = (len(eeg)-10_000)//2\n",
    "    eeg = eeg.iloc[middle:middle+10_000] # (10000, 20)\n",
    "\n",
    "    \n",
    "    # VARIABLE TO HOLD SPECTROGRAM\n",
    "    img = np.zeros((100,300,4),dtype='float32')\n",
    "    \n",
    "    if display: plt.figure(figsize=(10,7))\n",
    "    signals = []\n",
    "    for k in range(4):\n",
    "        COLS = FEATS[k]\n",
    "        \n",
    "        for kk in range(4):\n",
    "            # FILL NANS\n",
    "            x1 = eeg[COLS[kk]].values\n",
    "            x2 = eeg[COLS[kk+1]].values\n",
    "            m = np.nanmean(x1)\n",
    "            if np.isnan(x1).mean()<1: x1 = np.nan_to_num(x1,nan=m)\n",
    "            else: x1[:] = 0\n",
    "            m = np.nanmean(x2)\n",
    "            if np.isnan(x2).mean()<1: x2 = np.nan_to_num(x2,nan=m)\n",
    "            else: x2[:] = 0\n",
    "                \n",
    "            # COMPUTE PAIR DIFFERENCES\n",
    "            x = x1 - x2\n",
    "\n",
    "            x = butter_bandpass_filter(\n",
    "                x,\n",
    "                0.5,\n",
    "                20,\n",
    "                200,\n",
    "                order=2,\n",
    "            )\n",
    "            # DENOISE\n",
    "            if USE_WAVELET:\n",
    "                x = denoise(x, wavelet=USE_WAVELET)\n",
    "            signals.append(x)\n",
    "\n",
    "            # RAW SPECTROGRAM\n",
    "            mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//300, \n",
    "                  n_fft=1024, n_mels=100, fmin=0, fmax=20, win_length=128)\n",
    "            \n",
    "            # LOG TRANSFORM\n",
    "            width = (mel_spec.shape[1]//30)*30\n",
    "            mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n",
    "            img[:,:,k] += mel_spec_db\n",
    "                \n",
    "        # AVERAGE THE 4 MONTAGE DIFFERENCES\n",
    "        img[:,:,k] /= 4.0\n",
    "        \n",
    "        if display:\n",
    "            plt.subplot(2,2,k+1)\n",
    "            plt.imshow(img[:,:,k],aspect='auto',origin='lower')\n",
    "#             plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n",
    "            \n",
    "    if display: \n",
    "        plt.show()\n",
    "        plt.figure(figsize=(10,5))\n",
    "        offset = 0\n",
    "        for k in range(4):\n",
    "            if k>0: offset -= signals[3-k].min()\n",
    "            plt.plot(range(10_000),signals[k]+offset,label=NAMES[3-k])\n",
    "            offset += signals[3-k].max()\n",
    "        plt.legend()\n",
    "#         plt.title(f'EEG {eeg_id} Signals')\n",
    "        plt.show()\n",
    "#         print(); print('#'*25); print()\n",
    "        \n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2abd39ec",
   "metadata": {
    "papermill": {
     "duration": 563.483463,
     "end_time": "2024-02-05T20:37:26.924709",
     "exception": false,
     "start_time": "2024-02-05T20:28:03.441246",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Get all eegs with nans\n",
    "# PATH = '/root/autodl-tmp/hms-harmful-brain-activity-classification/train_eegs/'\n",
    "# files = os.listdir(PATH)\n",
    "# eeg_nans = {}\n",
    "# for file in files:\n",
    "#     eeg = pd.read_parquet(f'{PATH}{file}').values\n",
    "#     if np.isnan(eeg).sum():\n",
    "#         eeg_nans[file] = np.isnan(eeg).sum()\n",
    "# np.save('eeg_nans',eeg_nans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "33a977c5",
   "metadata": {
    "papermill": {
     "duration": 7187.737815,
     "end_time": "2024-02-05T22:37:14.671382",
     "exception": false,
     "start_time": "2024-02-05T20:37:26.933567",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating and writing 17089 spectrograms to disk... 100 , 200 , 300 , 400 , 500 , 600 , 700 , 800 , 900 , 1000 , 1100 , 1200 , 1300 , 1400 , 1500 , 1600 , 1700 , 1800 , 1900 , 2000 , 2100 , 2200 , 2300 , 2400 , 2500 , 2600 , 2700 , 2800 , 2900 , 3000 , 3100 , 3200 , 3300 , 3400 , 3500 , 3600 , 3700 , 3800 , 3900 , 4000 , 4100 , 4200 , 4300 , 4400 , 4500 , 4600 , 4700 , 4800 , 4900 , 5000 , 5100 , 5200 , 5300 , 5400 , 5500 , 5600 , 5700 , 5800 , 5900 , 6000 , 6100 , 6200 , 6300 , 6400 , 6500 , 6600 , 6700 , 6800 , 6900 , 7000 , 7100 , 7200 , 7300 , 7400 , 7500 , 7600 , 7700 , 7800 , 7900 , 8000 , 8100 , 8200 , 8300 , 8400 , 8500 , 8600 , 8700 , 8800 , 8900 , 9000 , 9100 , 9200 , 9300 , 9400 , 9500 , 9600 , 9700 , 9800 , 9900 , 10000 , 10100 , 10200 , 10300 , 10400 , 10500 , 10600 , 10700 , 10800 , 10900 , 11000 , 11100 , 11200 , 11300 , 11400 , 11500 , 11600 , 11700 , 11800 , 11900 , 12000 , 12100 , 12200 , 12300 , 12400 , 12500 , 12600 , 12700 , 12800 , 12900 , 13000 , 13100 , 13200 , 13300 , 13400 , 13500 , 13600 , 13700 , 13800 , 13900 , 14000 , 14100 , 14200 , 14300 , 14400 , 14500 , 14600 , 14700 , 14800 , 14900 , 15000 , 15100 , 15200 , 15300 , 15400 , 15500 , 15600 , 15700 , 15800 , 15900 , 16000 , 16100 , 16200 , 16300 , 16400 , 16500 , 16600 , 16700 , 16800 , 16900 , 17000 , CPU times: user 1h 32min 46s, sys: 5h 49min 16s, total: 7h 22min 3s\n",
      "Wall time: 24min 44s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "PATH = '/root/autodl-tmp/hms-harmful-brain-activity-classification/train_eegs/'\n",
    "DISPLAY = 4\n",
    "EEG_IDS = train.eeg_id.unique()\n",
    "all_eegs = {}\n",
    "\n",
    "for i,eeg_id in enumerate(EEG_IDS):\n",
    "    if (i%100==0)&(i!=0): print(i,', ',end='')\n",
    "        \n",
    "    # CREATE SPECTROGRAM FROM EEG PARQUET\n",
    "    img = spectrogram_from_eeg(f'{PATH}{eeg_id}.parquet', False)\n",
    "    \n",
    "    # SAVE TO DISK\n",
    "    if i==DISPLAY:\n",
    "        print(f'Creating and writing {len(EEG_IDS)} spectrograms to disk... ',end='')\n",
    "    np.save(f'{directory_path}{eeg_id}',img)\n",
    "    all_eegs[eeg_id] = img\n",
    "   \n",
    "# SAVE EEG SPECTROGRAM DICTIONARY\n",
    "np.save('eeg_specs_butter',all_eegs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f60524",
   "metadata": {
    "papermill": {
     "duration": 0.026282,
     "end_time": "2024-02-05T22:37:14.724044",
     "exception": false,
     "start_time": "2024-02-05T22:37:14.697762",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Kaggle Dataset\n",
    "The new EEG spectrograms from version 4 of this notebook have been uploaded to a Kaggle dataset [here][4]. We can attach this Kaggle dataset to our future notebooks to boost our CV scores and LB scores! Thank you everyone for upvoting my new EEG spectrogram Kaggle dataset!\n",
    "\n",
    "Examples of how to use EEG spectrograms to boost CV score and LB score will be (or already are) published in recent versions of my EfficientNet starter notebook [here][2] and CatBoost starter notebook [here][3]\n",
    "\n",
    "Enjoy! Happy Kaggling!\n",
    "\n",
    "[2]: https://www.kaggle.com/code/cdeotte/efficientnetb2-starter-lb-0-57\n",
    "[3]: https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-67\n",
    "[4]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms"
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "none",
   "dataSources": [
    {
     "databundleVersionId": 7469972,
     "sourceId": 59093,
     "sourceType": "competition"
    }
   ],
   "dockerImageVersionId": 30635,
   "isGpuEnabled": false,
   "isInternetEnabled": false,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 7761.165143,
   "end_time": "2024-02-05T22:37:17.443763",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-02-05T20:27:56.278620",
   "version": "2.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}