new-runner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the CaseOLAP Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Server-Wide Steps\n",
    "Steps 1-5 should be performed once and is used globally, across the whole server. The later steps should be customized for each project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Download the documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python '01_run_download.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Parse the documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '02_run_parsing.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Map MeSH to PMID (for document-category relationships)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '03_run_mesh2pmid.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '03_run_mesh2pmid.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 & 5. Index the documents (for document-entity relationships)\n",
    "Make sure Elasticsearch is properly configured and running first.\n",
    "4. Initialize the index. \n",
    "5. Run the index.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '04_run_index_init.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '05_run_index_populate.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project-Specific Steps\n",
    "These steps should be customized for the project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Categorize the documents of interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '06_run_textcube.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Vary the synonyms' cases\n",
    "Makes case-senstivie variations of the synonyms. This increases discovery of the synonyms within the case-sensitive text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "! python '07_run_vary_synonyms_cases.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Count all synonyms in the indexed text\n",
    "This counts the case-varied synonyms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! python '08_run_count_synonyms.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Screen for ambiguous synonyms\n",
    "Some synonyms will likely be ambiguous, leading to false positives. This step identifies the synonyms presumed be potentially ambiguous (i.e. short synonyms, synonyms that are single English words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! python '09_run_screen_synonyms.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps:\n",
    "- Add/remove synonyms as described in the next block\n",
    "- Run steps 10-13. \n",
    "- Inspect the scores. \n",
    "- Add/remove synonyms again\n",
    "- Repeat the process until you're satisfied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modify the file from step 9 (data/remove_these_synonyms.txt). \n",
    "Add bad synonyms, remove good synonyms. The case-varied versions are in here. The first part has synonyms that are English words. The second part has synonyms that are very short.\n",
    "- If you add, add the case-varied versions (e.g., \"Added Protein\", \"added protein\", \"Added protein\", \"added Protein\"). \n",
    "- If you remove, remove the case-varied versions of the entity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  10. Get the entity counts\n",
    "Using the synonyms that aren't bad synonyms and their synonym counts, this assemble the entity counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python '10_run_make_entity_counts.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Update the metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python '11_run_metadata_update.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 12. Produce CaseOLAP scores for the entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python '12_run_caseolap_score.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13. Inspect entity scores\n",
    "- You may notice that some entities score highly due to false positive synonyms. In that case, go back to the file mentioned just before step 9. Add the bad synonyms to the list, and run steps 10-13 again until you are satisfied with the quality of the results.\n",
    "- Check the files in in *results/ranked_entities* and *results/ranked_synonyms* (unless you changed where they're stored)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! python '13_run_inspect_entity_scores.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: Watch out for making conclusions based on proteins that cluster tightly together. They might just be clustering together because they have the same synonyms (although they may have the same synonyms because they are similar, but that could have been determined \n",
    "without looking at the score)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}