Create_Reader_Exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8014a533-29cc-410f-8ea5-00e361258709",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pycroscopy/arems25/blob/main/Create_Reader_Exercise.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b879145c-dc99-4cfa-a10f-0eeac05cf367",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sidpy SciFiReaders pycroscopy ipympl\n",
    "!git clone https://github.com/pycroscopy/arems25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ecb60d59-0444-4ed7-adeb-781e707ad8f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "colab = True #Set to True if in Google Colab, else set to False\n",
    "if colab:\n",
    "    %matplotlib widget\n",
    "    from google.colab import output\n",
    "    output.enable_custom_widget_manager()\n",
    "else:\n",
    "    %matplotlib qt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a16bb934-d16c-4077-8b23-9566658a5cef",
   "metadata": {},
   "source": [
    "# Creating a Reader\n",
    "\n",
    "Here we are going to go through an example of how to create a pycroscopy reader\n",
    "\n",
    "We will start with a stack of images of SrTiO3. The metadata is contained in a text file, and the images are contained in individual text files as well. We will need to read them in and then convert them to the sidpy dataset format. Luckily, this is not actually all that hard! Especially if we can leverage ChatGPT to figure out the tricky bits. \n",
    "\n",
    "The raw data is contained in the github repository here:\n",
    "https://github.com/pycroscopy/arems25/tree/main/Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "38c1dc1f-0826-4062-b139-751bb6ddc1b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#First let's import some packages\n",
    "import json\n",
    "import os\n",
    "import re\n",
    "import h5py\n",
    "import sidpy as sid\n",
    "import SciFiReaders as sr\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f8edfeb-4843-4220-bc44-8d6c134e1751",
   "metadata": {},
   "source": [
    "# Read the metadata\n",
    "\n",
    "First, we need to read the metadata from the text file into a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7265f62b-b831-4890-be7c-7d00e7c47369",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_path = r'arems25/Data/image_stack_metadata.txt'\n",
    "\n",
    "# Convert the metadata text into a Python dictionary.\n",
    "#Hint. Use json.read, and ChatGPT will be your friend.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56e01076-cc94-40a0-9a3b-c3d13e2d807e",
   "metadata": {},
   "source": [
    "# Read and arrange the image stack \n",
    "\n",
    "Here we are going to read the indivdiual image files that make up the stack and put them into a single numpy array.\n",
    "HINT: You can use np.loadtxt(fname,delimiter=\",\") to load individual files.\n",
    "\n",
    "The most challenging part of this exercise is to arrange the files in order. You can either try to do this yourself, or if you get tired, use an AI assistant.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f998386-79ae-4910-8856-0d60b71b4899",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now we have to import all of the text files\n",
    "\n",
    "img_fnames = [fname for fname in os.listdir('arems25/Data') if 'img_' in fname]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78fd79c1-ca15-4740-a43f-f0ae1b6e28d5",
   "metadata": {},
   "source": [
    "# Create the sidpy Dataset\n",
    "\n",
    "Once you have extracted the metadata and extracted the full data and placed it into a numpy arrray, we can begin to compile our sidpy dataset. For this, we need to construct our Dimension vectors, and then plug them into the set_dimension() method of sidpy's dataset class.\n",
    "\n",
    "This page will be most helpful:\n",
    "\n",
    "https://pycroscopy.github.io/sidpy/notebooks/00_basic_usage/create_dataset.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f4a68f64-6eb3-4a53-bdb3-101faff69d9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now convert it to a sidpy dataset\n",
    "\n",
    "#First, construct the x,y,and frame dimensions. And then construct the sidpy dataset. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5da2cc24-d72a-47f7-8be1-2cd57efafd5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Plot the data\n",
    "#fig= data_set.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc9de669-c6f2-450e-ab64-e0ec015c920e",
   "metadata": {},
   "source": [
    "# Bonus: Image Windowing\n",
    "\n",
    "Now that we have the sidpy dataset, we can do some fancy things with it. For example, we can do atom finding, or image windowing, or atom statistics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0726d548-255f-4bd1-b3c8-bdad518fada9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rvv/miniforge3/lib/python3.10/site-packages/pycroscopy/image/image_window.py:393: ComplexWarning: Casting complex values to real discards the imaginary part\n",
      "  img_window = np.array(img_window, dtype = np.float32)\n"
     ]
    }
   ],
   "source": [
    "#Image Windowing \n",
    "\n",
    "from pycroscopy.image import ImageWindowing\n",
    "\n",
    "parms_dict = {}\n",
    "parms_dict['window_step_x'] = 16\n",
    "parms_dict['window_step_y'] = 16\n",
    "parms_dict['window_size_x'] = 128\n",
    "parms_dict['window_size_y'] = 128\n",
    "parms_dict['mode'] = 'fft'\n",
    "parms_dict['filter'] = 'hamming'\n",
    "parms_dict['zoom_factor'] = 2\n",
    "parms_dict['interpol_factor'] = 4\n",
    "iw = ImageWindowing(parms_dict)\n",
    "windows = iw.MakeWindows(data_set, dim_slice=0)\n",
    "windows = np.abs(np.log(np.abs(windows)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "703155ec-2897-4683-b4be-b1dff4f9f1b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "windows.plot();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "575ef511-1c7f-4334-bc1d-e9bf311fa0a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using generic parameters for dimension  1\n",
      "using generic parameters for dimension  0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rvv/miniforge3/lib/python3.10/site-packages/sklearn/decomposition/_nmf.py:1759: ConvergenceWarning: Maximum number of iterations 200 reached. Increase it to improve convergence.\n",
      "  warnings.warn(\n",
      "/Users/rvv/miniforge3/lib/python3.10/site-packages/sidpy/base/num_utils.py:54: RuntimeWarning: invalid value encountered in divide\n",
      "  if var / step_avg < tol:\n"
     ]
    }
   ],
   "source": [
    "from pycroscopy.learn.ml.matrix_factor import MatrixFactor\n",
    "\n",
    "mfactor = MatrixFactor(np.abs(windows), method = 'nmf',n_components = 3)\n",
    "output = mfactor.do_fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2817128d-4307-4491-bf06-7ec6fb0cb2dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "abundances = output[0]\n",
    "components = output[1]\n",
    "abund = np.array(abundances)\n",
    "comps = np.array(components)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7ababa9b-74d4-4519-ae60-abb68494d4e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.axes_grid1 import make_axes_locatable\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=mfactor.ncomp, figsize = (10,3))\n",
    "for ind, ax in enumerate(axes.flat):\n",
    "    im1 = ax.imshow(comps[ind,:,:])\n",
    "    ax.set_title('Component #' + str(ind))\n",
    "    divider = make_axes_locatable(ax)\n",
    "    cax = divider.append_axes('right', size='5%', pad=0.05)\n",
    "    fig.colorbar(im1, cax=cax, orientation='vertical')\n",
    "    ax.axis('off')\n",
    "fig.tight_layout()\n",
    "fig.savefig('Fig3b.png', dpi = 300)\n",
    "\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=mfactor.ncomp, figsize = (10,3))\n",
    "for ind, ax in enumerate(axes.flat):\n",
    "    im1 = ax.imshow(abund[:,:,ind])\n",
    "    divider = make_axes_locatable(ax)\n",
    "    cax = divider.append_axes('right', size='5%', pad=0.05)\n",
    "    fig.colorbar(im1, cax=cax, orientation='vertical')\n",
    "    ax.axis('off')\n",
    "\n",
    "fig.tight_layout()\n",
    "fig.savefig('Fig3c.png', dpi = 300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc527e46-c8e7-458e-aa23-bd16f6fd0ec8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}