parse_simulation_outputs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"toc\"></a>\n",
    "# PARSE SIMULATION OUTPUTS\n",
    "\n",
    "This notebook provides the functions and scripts for parsing simulation files (`.json`) into pickled numpy arrays (`.pkl`). \n",
    "\n",
    "---\n",
    "- [WORKSPACE VARIABLES](#workspace-variables)\n",
    "- **[DEFAULT](#default)**\n",
    "- **[MODULE_COMPLEXITY](#module-complexity)**\n",
    "- **[PARAMETER SENSITIVITY](#parameter-sensitivity)**\n",
    "- **[GROWTH CONTEXT](#growth-context)**\n",
    "- **[CELL COMPETITION](#cell-competition)**\n",
    "- **[POPULATION HETEROGENEITY](#population-heterogeneity)**\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main parsing function (`parse_simulations`) iterates through each file in the data path and parses each simulation instance, extracting fields from the simulation setup, cells, and environment.\n",
    "\n",
    "The parsed arrays are organized as:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"setup\": {\n",
    "        \"radius\": R,\n",
    "        \"height\": H,\n",
    "        \"time\": [],\n",
    "        \"pops\": [],\n",
    "        \"types\": [],\n",
    "        \"coords\": []\n",
    "    },\n",
    "    \"agents\": (N seeds) x (T timepoints) x (H height) x (C coordinates) x (P positions),\n",
    "    \"environments\": {\n",
    "        \"glucose\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "        \"oxygen\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "        \"tgfa\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "where each entry in the agents array is a structured entry of the shape:\n",
    "\n",
    "```\n",
    "\"pop\"       int8    population code\n",
    "\"type\"      int8    cell type code\n",
    "\"volume\"    int16   cell volume (rounded)\n",
    "\"cycle\"     int16   average cell cycle length (rounded)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `parse.py` file contains general parsing functions.\n",
    "\n",
    "Parsing can take some time, so parsed `.pkl` files for all simulations are provided along with the raw simulation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.parse import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"workspace-variables\"></a>\n",
    "\n",
    "### WORKSPACE VARIABLES \n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Set up workspace variables for parsing simulations.\n",
    "\n",
    "- **`DATA_PATH`** is the path to data files (`.tar.xz` files of compressed simulation outputs)\n",
    "- **`RESULT_PATH`** is the path for result files (`.pkl` files generated by parsing)\n",
    "- **`EXCLUDE`** is a list of seeds to exclude from parsing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH = \"/path/to/data/files/\"\n",
    "RESULTS_PATH = \"/path/to/result/files/\"\n",
    "EXCLUDE = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"default\"></a>\n",
    "\n",
    "### DEFAULT\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Parse `DEFAULT` simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_simulations(\"DEFAULT\", DATA_PATH, RESULTS_PATH, EXCLUDE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"module-complexity\"></a>\n",
    "\n",
    "### MODULE COMPLEXITY\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Parse `MODULE_COMPLEXITY` simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "parse_simulations(\"MODULE_COMPLEXITY\", DATA_PATH, RESULTS_PATH, EXCLUDE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"parameter-sensitivity\"></a>\n",
    "\n",
    "### PARAMETER SENSITIVITY\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Parse `PARAMETER_SENSITIVITY` simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_simulations(\"PARAMETER_SENSITIVITY\", DATA_PATH, RESULTS_PATH, EXCLUDE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"growth-context\"></a>\n",
    "\n",
    "### GROWTH CONTEXT\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Parse `GROWTH_CONTEXT` simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_simulations(\"GROWTH_CONTEXT\", DATA_PATH, RESULTS_PATH, EXCLUDE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cell-competition\"></a>\n",
    "\n",
    "### CELL COMPETITION\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Parse `CELL_COMPETITION` simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_simulations(\"CELL_COMPETITION\", DATA_PATH, RESULTS_PATH, EXCLUDE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"population-heterogeneity\"></a>\n",
    "\n",
    "### POPULATION HETEROGENEITY\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Parse `POPULATION_HETEROGENEITY` simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_simulations(\"POPULATION_HETEROGENEITY\", DATA_PATH, RESULTS_PATH, EXCLUDE)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}