Step3-Putting-it-all-together.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# To recap\n",
    "\n",
    "### Data so far\n",
    "\n",
    "1. We have created a list of people and their ORCID identifiers - [data/pure_person_to_orcid.txt](data/pure_person_to_orcid.txt)\n",
    "2. We have created a list of ORCID to PubMed identifiers - [data/orcid.tsv](data/orcid.tsv)\n",
    "3. We have created a list of PubMed IDs to PubMed info - [data/pubmed.tsv](data/pubmed.tsv)\n",
    "4. We have created a list of the top 100 terms for each person - [data/orcid-tf-idf.txt](data/orcid-tf-idf.txt)\n",
    "\n",
    "### Questions\n",
    "\n",
    "Now we need to use the common identifiers in these data to answer some questions:\n",
    "\n",
    "1. Can we produce a set of potential collaborators for each person, a collaborator being someone they have significant terms in common with but not previously published with.\n",
    "2. Using a piece of text, such as a conference or publication, can we select a set of people that most closely map to it?\n",
    "3. If we were to compare the publications from a person outside of our group, where would they fit?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create some dataframes\n",
    "\n",
    "Normally at this point we would use a database, and the phrase `knowledge graph` in the title of the workshop suggests that would be the aim here (if I were to use a database at this point, I would use Neo4j - https://neo4j.com/). However, to keep it all in a single language and framework, and as the data themselves are relatively small, we can use pandas - https://pandas.pydata.org/. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, following on from the data descriptions above:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         pure_person_id             orcid_id\n",
      "0  4ec18a96-fd2e-4311-a6fa-7ec65696a4e9  0000-0003-3108-5775\n",
      "1  7a04c325-7a39-4560-8308-8cbcaa763747  0000-0002-6772-7111\n",
      "2  b4014828-88e9-4861-ae1d-5c369b6ae35a  0000-0001-7328-4233\n",
      "3  ccedc4d6-6d7e-4a60-8313-f59409ecc6dd  0000-0001-6224-3073\n",
      "4  66e3df0f-6cbe-4cf4-a034-9e1b57684f6b  0000-0003-0300-4990\n",
      "(898, 2)\n",
      "              orcid_id      pmid\n",
      "0  0000-0001-5001-3350  26961927\n",
      "1  0000-0001-5001-3350  30605491\n",
      "2  0000-0001-5008-0705  29118635\n",
      "3  0000-0001-5017-9473  18194108\n",
      "4  0000-0001-5017-9473  18607707\n",
      "(15722, 2)\n",
      "       pmid  year                                              title  \\\n",
      "0  25475436  2015  Sixty-five common genetic variants and predict...   \n",
      "1  25011450  2014  Association between alcohol and cardiovascular...   \n",
      "2  28968714  2018  FATHMM-XF: accurate prediction of pathogenic p...   \n",
      "3  21965548  2012  Four genetic loci influencing electrocardiogra...   \n",
      "4  26930047  2016  Diagnosis of Coronary Heart Diseases Using Gen...   \n",
      "\n",
      "                                            abstract  \n",
      "0  We developed a 65 type 2 diabetes (T2D) varian...  \n",
      "1  To use the rs1229984 variant in the alcohol de...  \n",
      "2  We present FATHMM-XF, a method for predicting ...  \n",
      "3  Presence of left ventricular hypertrophy on an...  \n",
      "4  Cardiovascular disease (including coronary art...  \n",
      "(11160, 4)\n",
      "              orcid_id             term    tf-idf\n",
      "0  0000-0003-0924-3247      methylation  0.201768\n",
      "1  0000-0003-0924-3247         variants  0.139032\n",
      "2  0000-0003-0924-3247             loci  0.138066\n",
      "3  0000-0003-0924-3247          genetic  0.136506\n",
      "4  0000-0003-0924-3247  dna methylation  0.127503\n",
      "(52546, 3)\n",
      "                         pure_person_id         person_name\n",
      "0  f8a64592-741a-4b32-a017-5730498d7462   Stefania Placenti\n",
      "1  5208f654-2f78-4432-b82e-2d68fff58e46   Philip D Harfield\n",
      "2  2eb962d5-5a55-4eca-8136-0dda897f6302     Magnus Feldmann\n",
      "3  eed59a07-5ea6-485b-8e78-5d979a78c553      Steven C Greer\n",
      "4  afffe47a-4eb4-4a84-8393-52701454f60b  Jessica L Fielding\n",
      "(3565, 2)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import config\n",
    "\n",
    "# 1. People and their ORCID identifiers \n",
    "personToOrcid = pd.read_csv(config.demoPureOrcidFile,sep='\\t')\n",
    "print(personToOrcid.head())\n",
    "print(personToOrcid.shape)\n",
    "\n",
    "# 2. ORCID to PubMed identifiers\n",
    "orcidToPubmed = pd.read_csv(config.demoOrcidFile,sep='\\t')\n",
    "print(orcidToPubmed.head())\n",
    "print(orcidToPubmed.shape)\n",
    "\n",
    "# 3. PubMed IDs to PubMed info\n",
    "pubmedToInfo = pd.read_csv(config.demoPubmedFile,sep='\\t')\n",
    "print(pubmedToInfo.head())\n",
    "print(pubmedToInfo.shape)\n",
    "\n",
    "# 4. Top 100 terms for each person\n",
    "topTerms = pd.read_csv(config.demoTfidfFile,sep='\\t')\n",
    "print(topTerms.head())\n",
    "print(topTerms.shape)\n",
    "\n",
    "# 5. PURE to name\n",
    "pureToName = pd.read_csv(config.demoPurePeopleFile,sep='\\t')\n",
    "print(pureToName.head())\n",
    "print(pureToName.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A first pass would be to count the number of matching terms between each person. To start with, let's just compare the first in the list to everyone else:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### Comparing 0 4ec18a96-fd2e-4311-a6fa-7ec65696a4e9 0000-0003-3108-5775\n",
      "0000-0001-7086-8882 ['produced mesospheric hotspot', 'produced trace', 'vortex saturns largest', 'stable vortex configuration', 'solar heating moved', 'titan', 'saturns', 'produced trace gases', 'subsidence produced mesospheric', 'results high infrared', 'titans winter', 'vortex dramatically increases', 'timeframe reach stable', 'southpolar subsidence winter', 'strengthening subsidence', 'years postequinox', 'rapid mesospheric', 'postequinox cooling', 'trace gas enrichment', 'unexpected rapid mesospheric', 'substantial nitrogenmethane', 'titan substantial nitrogenmethane', 'long', 'seasonal effects including', 'relatively long atmospheric', 'radiative time constant', 'postequinox cooling far', 'polar', 'titan substantial', 'radiative', 'vortex saturns', 'titans winter polar', 'substantial nitrogenmethane atmosphere', 'southpolar subsidence', 'saturns largest moon', 'trace gases 2012', 'vortex dramatically', 'winter polar hotspots', 'titans trace gases', 'titans 2009 northern', 'stable vortex', 'vortex configuration', 'extreme', 'timeframe reach', 'spring equinox peak', 'vortex formation 20102011', 'winter polar vortex', 'spring equinox', 'polar vortex saturns', 'vortex configuration results', 'winter', 'vortices following titans', 'winter polar vortices', 'mesospheric', 'subsidence winter polar', 'years postequinox long', 'solar heating', 'unusually cold temperatures', 'trace gases', 'subsidence', 'southpolar', 'subsidence winter', 'postequinox', 'gases', 'rapid mesospheric cooling', 'northern', 'temperatures 26', 'strengthening subsidence produced', 'polar vortex', 'radiative cooling', 'titans 2009', 'unexpected rapid', 'unique titan', 'vortex', 'radiative time', 'postequinox long', 'trace gases relatively', 'formation', 'temperatures 26 years', 'polar vortices', 'trace', 'subsidence produced', 'enrichment', 'cooling', 'strong seasonal effects', 'reach stable vortex', 'polar vortex formation', 'saturns largest', 'polar vortices following', 'seasonal effects', 'time constant winter', 'radiative cooling efficiency', 'postequinox long timeframe', 'produced mesospheric', 'vortex formation', 'unusually cold', 'vortices following', 'titans trace', 'titans', 'winter polar']\n"
     ]
    }
   ],
   "source": [
    "#loop through people, limit to top n in list\n",
    "def compare_people(limit):\n",
    "    for i, iData in personToOrcid.head(n=limit).iterrows():\n",
    "        print('### Comparing',i,iData.pure_person_id,iData.orcid_id)\n",
    "        #get all terms and tf-idf values\n",
    "        iTopTerms=topTerms[topTerms['orcid_id']==iData.orcid_id]\n",
    "        iTerms = iTopTerms['term']\n",
    "\n",
    "        #compare to all other people\n",
    "        for j in range(i+1,personToOrcid.shape[0]):\n",
    "            jData=personToOrcid.iloc[j]\n",
    "            jTopTerms=topTerms[topTerms['orcid_id']==jData.orcid_id]\n",
    "            jTerms = jTopTerms['term']\n",
    "            com = list(set(iTerms) & set(jTerms))\n",
    "            #only show maches with more than 1 matching term\n",
    "            if len(com) > 1:\n",
    "                print(jData.orcid_id,com)\n",
    "    \n",
    "compare_people(1)        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are few matches with many overlapping terms, except one **0000-0001-7086-8882**. Let's look at the research pages of these two people\n",
    "\n",
    "https://research-information.bristol.ac.uk/en/persons/melody-a-s-sylvestre(81e7f06c-77d8-4020-9608-0b30dd001c43)/publications.html https://research-information.bristol.ac.uk/en/persons/nicholas-a-teanby(4ec18a96-fd2e-4311-a6fa-7ec65696a4e9)/publications.html \n",
    "\n",
    "This suggests they are from similar research areas, and have indeed co-published.   \n",
    "\n",
    "Let's try again, this time with the top 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### Comparing 0 4ec18a96-fd2e-4311-a6fa-7ec65696a4e9 0000-0003-3108-5775\n",
      "0000-0001-7086-8882 ['produced mesospheric hotspot', 'produced trace', 'vortex saturns largest', 'stable vortex configuration', 'solar heating moved', 'titan', 'saturns', 'produced trace gases', 'subsidence produced mesospheric', 'results high infrared', 'titans winter', 'vortex dramatically increases', 'timeframe reach stable', 'southpolar subsidence winter', 'strengthening subsidence', 'years postequinox', 'rapid mesospheric', 'postequinox cooling', 'trace gas enrichment', 'unexpected rapid mesospheric', 'substantial nitrogenmethane', 'titan substantial nitrogenmethane', 'long', 'seasonal effects including', 'relatively long atmospheric', 'radiative time constant', 'postequinox cooling far', 'polar', 'titan substantial', 'radiative', 'vortex saturns', 'titans winter polar', 'substantial nitrogenmethane atmosphere', 'southpolar subsidence', 'saturns largest moon', 'trace gases 2012', 'vortex dramatically', 'winter polar hotspots', 'titans trace gases', 'titans 2009 northern', 'stable vortex', 'vortex configuration', 'extreme', 'timeframe reach', 'spring equinox peak', 'vortex formation 20102011', 'winter polar vortex', 'spring equinox', 'polar vortex saturns', 'vortex configuration results', 'winter', 'vortices following titans', 'winter polar vortices', 'mesospheric', 'subsidence winter polar', 'years postequinox long', 'solar heating', 'unusually cold temperatures', 'trace gases', 'subsidence', 'southpolar', 'subsidence winter', 'postequinox', 'gases', 'rapid mesospheric cooling', 'northern', 'temperatures 26', 'strengthening subsidence produced', 'polar vortex', 'radiative cooling', 'titans 2009', 'unexpected rapid', 'unique titan', 'vortex', 'radiative time', 'postequinox long', 'trace gases relatively', 'formation', 'temperatures 26 years', 'polar vortices', 'trace', 'subsidence produced', 'enrichment', 'cooling', 'strong seasonal effects', 'reach stable vortex', 'polar vortex formation', 'saturns largest', 'polar vortices following', 'seasonal effects', 'time constant winter', 'radiative cooling efficiency', 'postequinox long timeframe', 'produced mesospheric', 'vortex formation', 'unusually cold', 'vortices following', 'titans trace', 'titans', 'winter polar']\n",
      "### Comparing 1 7a04c325-7a39-4560-8308-8cbcaa763747 0000-0002-6772-7111\n",
      "### Comparing 2 b4014828-88e9-4861-ae1d-5c369b6ae35a 0000-0001-7328-4233\n",
      "0000-0001-6224-3073 ['using', 'functional', 'molecular']\n",
      "0000-0001-7382-2855 ['using', 'analysis', 'database', 'diversity', 'molecular', 'different', 'data']\n",
      "0000-0003-2881-4180 ['using', 'analysis', 'data', 'cancer']\n",
      "0000-0002-1753-3730 ['cell', 'angiogenesis']\n",
      "0000-0002-1040-7635 ['data', 'cancer']\n",
      "0000-0001-8496-6053 ['using', 'analysis', 'data', 'cancer']\n",
      "0000-0002-9848-1409 ['cell', 'stem cell', 'stem']\n",
      "0000-0002-2618-750X ['using', 'analysis', 'species', 'taxa', 'data', 'functional']\n",
      "0000-0002-5555-433X ['breast cancer', 'breast', 'cancer']\n",
      "0000-0002-2497-1026 ['cell', 'using', 'cellular']\n",
      "0000-0001-7206-3674 ['phenotype', 'data']\n",
      "0000-0001-5990-9161 ['sequence', 'data']\n",
      "0000-0002-3510-4155 ['using', 'molecular']\n",
      "0000-0002-4323-1824 ['diversity', 'species', 'clades', 'molecular']\n",
      "0000-0002-7961-8027 ['development', 'heritability']\n",
      "0000-0003-1561-9933 ['cell', 'ld', 'cellular']\n",
      "0000-0002-5311-6213 ['sequence', 'using', 'analysis', 'diversity', 'genome', 'genomes', 'data']\n",
      "0000-0002-9732-5738 ['using', 'parasitic', 'diversity', 'species', 'data', 'nematode', 'nematode species', 'nematodes']\n",
      "0000-0003-0574-5071 ['using', 'analysis', 'data']\n",
      "0000-0002-1035-9020 ['different', 'molecular']\n",
      "0000-0002-7404-9128 ['analysis', 'data']\n",
      "0000-0001-8186-5708 ['cell', 'cancer']\n",
      "0000-0002-3097-3765 ['cell', 'stem', 'genes']\n",
      "0000-0002-4032-9500 ['potential', 'using', 'ld', 'data']\n",
      "0000-0003-3909-6698 ['mechanisms', 'stem']\n",
      "0000-0002-0125-4841 ['cell', 'cancer', 'development']\n",
      "0000-0001-6012-6527 ['analysis', 'different']\n",
      "0000-0002-4049-993X ['using', 'data']\n",
      "0000-0003-4611-8795 ['sequence', 'analysis', 'cellular', 'genome', 'genes', 'cell', 'data']\n",
      "0000-0001-7793-7326 ['mouth', 'heritability', 'ulcers', 'mouth ulcers', 'data']\n",
      "0000-0002-8608-289X ['development', 'analysis', 'heritability', 'genes']\n",
      "0000-0002-5943-3981 ['resource', 'cellular', 'molecular', 'cell', 'data']\n",
      "0000-0003-1217-4852 ['cell', 'genes', 'development']\n",
      "0000-0003-3116-7463 ['using', 'mirna', 'diversity', 'genome', 'clades', 'molecular', 'development', 'species', 'taxa', 'data']\n",
      "0000-0002-1113-7761 ['sequence', 'cell', 'expressed', 'stem', 'transcripts']\n",
      "0000-0001-7808-6121 ['cell', 'stem cell', 'functional', 'stem']\n",
      "0000-0001-5673-9371 ['cell', 'phenotype', 'mechanisms', 'cellular']\n",
      "0000-0003-2906-4035 ['analysis', 'mechanisms']\n",
      "0000-0002-0394-3202 ['sequence', 'analysis', 'potential', 'functional', 'sequences']\n",
      "0000-0003-2946-0353 ['cell', 'expressed', 'sequences', 'genes']\n",
      "0000-0002-5312-0378 ['using', 'molecular']\n",
      "0000-0002-6177-577X ['genome', 'mechanisms', 'species', 'genes']\n",
      "0000-0002-2393-4522 ['cell', 'stem cell', 'stem']\n",
      "0000-0002-2183-8166 ['breast', 'cancer']\n",
      "0000-0002-1573-7908 ['using', 'species']\n",
      "0000-0003-0933-646X ['analysis', 'cancer', 'genome', 'breast cancer', 'breast', 'data']\n",
      "0000-0003-3631-627X ['using', 'data', 'cancer']\n",
      "0000-0003-3565-0479 ['cell', 'using']\n",
      "0000-0002-6977-312X ['breast cancer', 'breast', 'cancer']\n",
      "0000-0002-1407-8314 ['using', 'analysis', 'cancer', 'genes', 'data']\n",
      "0000-0001-5255-8506 ['potential', 'data']\n",
      "0000-0003-3979-7857 ['potential', 'molecular']\n",
      "0000-0001-8202-1144 ['using', 'data']\n",
      "0000-0003-3113-0345 ['data', 'molecular']\n",
      "0000-0002-9528-1495 ['analysis', 'species', 'cellular']\n",
      "0000-0003-1454-8440 ['sequence', 'genome', 'sequences', 'genes']\n",
      "0000-0002-8152-6017 ['using', 'different', 'data']\n",
      "0000-0003-3719-6361 ['different', 'cancer']\n",
      "0000-0002-3853-883X ['development', 'potential', 'using', 'cancer']\n",
      "0000-0002-0935-2200 ['genome', 'analysis', 'data', 'genes']\n",
      "0000-0002-7400-005X ['using', 'species']\n",
      "0000-0003-1562-891X ['cancer', 'breast cancer', 'cell', 'development', 'breast', 'cancer cell']\n",
      "0000-0002-2861-0578 ['different', 'data', 'database', 'cancer']\n",
      "0000-0003-2052-4840 ['using', 'analysis', 'cancer', 'genome', 'genes', 'development', 'data']\n",
      "0000-0002-4701-8713 ['cell', 'development']\n",
      "0000-0003-3894-0623 ['cell', 'angiogenesis']\n",
      "0000-0002-7708-3110 ['annotated', 'data']\n",
      "0000-0002-2115-3503 ['cell', 'genome', 'cellular', 'molecular']\n",
      "0000-0002-3897-5861 ['cell', 'stem cell', 'phenotype', 'stem']\n",
      "0000-0002-7049-3037 ['analysis', 'data', 'cancer']\n",
      "0000-0003-3049-0939 ['cell', 'using']\n",
      "0000-0003-3756-040X ['mouth', 'cancer', 'ulcers', 'mouth ulcers', 'data']\n",
      "0000-0002-8911-2492 ['data', 'heritability']\n",
      "0000-0002-8917-7384 ['cellular', 'cancer', 'tumor', 'breast cancer', 'cell', 'cancer cell']\n",
      "0000-0001-6648-3007 ['using', 'analysis', 'data']\n",
      "0000-0001-8923-0117 ['using', 'analysis', 'cancer']\n",
      "0000-0002-4680-3517 ['using', 'analysis', 'genes', 'development', 'data']\n",
      "0000-0002-2870-1832 ['development', 'data']\n",
      "0000-0001-8091-6534 ['cell', 'molecular']\n",
      "0000-0001-5297-3240 ['cell', 'functional']\n",
      "0000-0001-5747-2699 ['data', 'cancer']\n",
      "0000-0002-2501-6602 ['sequence', 'molecular']\n",
      "0000-0003-0848-935X ['mechanisms', 'genes']\n",
      "0000-0003-1072-0223 ['sequence', 'analysis', 'cellular', 'diversity', 'genome', 'genes', 'freeliving', 'molecular', 'genomes', 'data', 'functional']\n",
      "0000-0003-0924-3247 ['using', 'analysis', 'cancer', 'genome', 'score', 'genes', 'potential', 'data', 'functional']\n",
      "0000-0002-4540-6683 ['different', 'species']\n",
      "0000-0001-7652-2030 ['development', 'species']\n",
      "0000-0002-4612-7141 ['sequence', 'different', 'sequences']\n",
      "0000-0002-5904-8397 ['using', 'analysis', 'data']\n",
      "0000-0003-2601-7575 ['cell', 'breast', 'breast cancer']\n",
      "0000-0003-4395-9396 ['cell', 'expressed']\n",
      "0000-0002-2601-9258 ['development', 'data', 'cancer']\n",
      "0000-0002-2715-9930 ['mechanisms', 'cancer', 'genes', 'development', 'data']\n",
      "0000-0001-6566-9205 ['mechanisms', 'stem']\n",
      "0000-0002-3548-6245 ['cell', 'analysis']\n",
      "0000-0003-2656-4034 ['mechanisms', 'microrna', 'micrornas', 'mirna', 'stem']\n",
      "0000-0003-0920-1055 ['using', 'analysis', 'heritability', 'genome', 'genes', 'ld', 'data']\n",
      "0000-0002-7141-9189 ['using', 'analysis', 'data', 'genes']\n",
      "0000-0003-3405-4843 ['using', 'analysis', 'genes', 'species', 'sequences']\n",
      "0000-0003-4416-4560 ['species', 'molecular']\n",
      "0000-0001-8539-8959 ['cell', 'mirna']\n",
      "0000-0003-0858-6088 ['cell', 'potential']\n",
      "0000-0002-7335-9001 ['sequence', 'genes']\n",
      "0000-0001-6849-8747 ['diversity', 'resource', 'species']\n",
      "0000-0002-8773-2743 ['using', 'analysis', 'genome', 'molecular', 'species', 'data']\n",
      "0000-0001-9595-3338 ['different', 'molecular']\n",
      "0000-0002-9759-5156 ['cell', 'phenotype', 'functional', 'stem']\n",
      "0000-0001-6293-6662 ['sequence', 'analysis', 'species']\n",
      "0000-0001-9426-686X ['cell', 'breast cancer', 'breast', 'cancer']\n",
      "0000-0002-3354-3330 ['development', 'data', 'breast', 'cancer']\n",
      "0000-0002-9133-2021 ['development', 'data', 'genes']\n",
      "0000-0001-8548-1090 ['sequence', 'using', 'molecular']\n",
      "0000-0002-4217-3060 ['diversity', 'different', 'data']\n",
      "0000-0003-1486-745X ['analysis', 'data']\n",
      "0000-0003-2826-3307 ['development', 'using', 'data', 'breast']\n",
      "0000-0002-6793-2262 ['using', 'analysis', 'data']\n",
      "0000-0002-4935-6724 ['cell', 'genes', 'development']\n",
      "0000-0002-1010-8926 ['score', 'using', 'data', 'cancer']\n",
      "0000-0001-6678-8839 ['data', 'molecular']\n",
      "0000-0003-0609-475X ['analysis', 'functional']\n",
      "0000-0003-4690-8826 ['analysis', 'molecular', 'cell', 'different', 'functional']\n",
      "0000-0002-6009-7137 ['cell', 'stem', 'tumor', 'cancer']\n",
      "0000-0003-3589-8335 ['cell', 'mechanisms', 'functional', 'molecular']\n",
      "0000-0003-2088-8310 ['cell', 'using', 'analysis']\n",
      "0000-0002-6828-5888 ['cell', 'cellular', 'stem']\n",
      "0000-0002-3584-9616 ['molecular', 'stem', 'taxa']\n",
      "0000-0002-5300-3331 ['cell', 'stem cell', 'development', 'potential', 'functional', 'stem']\n",
      "0000-0003-2198-3205 ['using', 'analysis', 'data', 'different']\n",
      "0000-0002-3922-0188 ['analysis', 'genes']\n",
      "0000-0002-0950-8148 ['genome', 'genes', 'phenotype']\n",
      "0000-0003-1702-786X ['using', 'genome']\n",
      "0000-0001-5551-9609 ['using', 'molecular']\n",
      "0000-0002-9644-8998 ['diversity', 'different', 'species', 'genes']\n",
      "0000-0002-3537-5166 ['using', 'analysis', 'resource', 'different', 'data']\n",
      "0000-0003-2070-6679 ['using', 'different', 'sequences']\n",
      "0000-0002-3208-0389 ['heritability', 'genes']\n",
      "0000-0002-1904-3735 ['analysis', 'genes', 'molecular', 'species', 'sequences']\n",
      "0000-0002-0546-2381 ['using', 'data']\n",
      "0000-0003-2514-0889 ['data', 'genes']\n",
      "0000-0002-6391-6601 ['functional', 'molecular']\n",
      "0000-0002-4701-793X ['mechanisms', 'species']\n",
      "0000-0002-7992-7719 ['using', 'data', 'cancer']\n",
      "0000-0001-5677-5401 ['using', 'parasitic', 'diversity', 'species', 'data', 'functional']\n",
      "0000-0001-8595-5153 ['cell', 'stem']\n",
      "0000-0001-7240-4563 ['score', 'using', 'analysis', 'data']\n"
     ]
    }
   ],
   "source": [
    "compare_people(3)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second person matched no-one, the third however matched quite a few. Let's modify the function to sort by number:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### Comparing 0 4ec18a96-fd2e-4311-a6fa-7ec65696a4e9 0000-0003-3108-5775\n",
      "('0000-0001-7086-8882', ['produced mesospheric hotspot', 'produced trace', 'vortex saturns largest', 'stable vortex configuration', 'solar heating moved', 'titan', 'saturns', 'produced trace gases', 'subsidence produced mesospheric', 'results high infrared', 'titans winter', 'vortex dramatically increases', 'timeframe reach stable', 'southpolar subsidence winter', 'strengthening subsidence', 'years postequinox', 'rapid mesospheric', 'postequinox cooling', 'trace gas enrichment', 'unexpected rapid mesospheric', 'substantial nitrogenmethane', 'titan substantial nitrogenmethane', 'long', 'seasonal effects including', 'relatively long atmospheric', 'radiative time constant', 'postequinox cooling far', 'polar', 'titan substantial', 'radiative', 'vortex saturns', 'titans winter polar', 'substantial nitrogenmethane atmosphere', 'southpolar subsidence', 'saturns largest moon', 'trace gases 2012', 'vortex dramatically', 'winter polar hotspots', 'titans trace gases', 'titans 2009 northern', 'stable vortex', 'vortex configuration', 'extreme', 'timeframe reach', 'spring equinox peak', 'vortex formation 20102011', 'winter polar vortex', 'spring equinox', 'polar vortex saturns', 'vortex configuration results', 'winter', 'vortices following titans', 'winter polar vortices', 'mesospheric', 'subsidence winter polar', 'years postequinox long', 'solar heating', 'unusually cold temperatures', 'trace gases', 'subsidence', 'southpolar', 'subsidence winter', 'postequinox', 'gases', 'rapid mesospheric cooling', 'northern', 'temperatures 26', 'strengthening subsidence produced', 'polar vortex', 'radiative cooling', 'titans 2009', 'unexpected rapid', 'unique titan', 'vortex', 'radiative time', 'postequinox long', 'trace gases relatively', 'formation', 'temperatures 26 years', 'polar vortices', 'trace', 'subsidence produced', 'enrichment', 'cooling', 'strong seasonal effects', 'reach stable vortex', 'polar vortex formation', 'saturns largest', 'polar vortices following', 'seasonal effects', 'time constant winter', 'radiative cooling efficiency', 'postequinox long timeframe', 'produced mesospheric', 'vortex formation', 'unusually cold', 'vortices following', 'titans trace', 'titans', 'winter polar'])\n",
      "### Comparing 1 7a04c325-7a39-4560-8308-8cbcaa763747 0000-0002-6772-7111\n",
      "### Comparing 2 b4014828-88e9-4861-ae1d-5c369b6ae35a 0000-0001-7328-4233\n",
      "('0000-0003-1072-0223', ['sequence', 'analysis', 'cellular', 'diversity', 'genome', 'genes', 'freeliving', 'molecular', 'genomes', 'data', 'functional'])\n",
      "('0000-0003-3116-7463', ['using', 'mirna', 'diversity', 'genome', 'clades', 'molecular', 'development', 'species', 'taxa', 'data'])\n",
      "('0000-0003-0924-3247', ['using', 'analysis', 'cancer', 'genome', 'score', 'genes', 'potential', 'data', 'functional'])\n",
      "('0000-0002-9732-5738', ['using', 'parasitic', 'diversity', 'species', 'data', 'nematode', 'nematode species', 'nematodes'])\n",
      "('0000-0001-7382-2855', ['using', 'analysis', 'database', 'diversity', 'molecular', 'different', 'data'])\n",
      "('0000-0002-5311-6213', ['sequence', 'using', 'analysis', 'diversity', 'genome', 'genomes', 'data'])\n",
      "('0000-0003-4611-8795', ['sequence', 'analysis', 'cellular', 'genome', 'genes', 'cell', 'data'])\n",
      "('0000-0003-2052-4840', ['using', 'analysis', 'cancer', 'genome', 'genes', 'development', 'data'])\n",
      "('0000-0003-0920-1055', ['using', 'analysis', 'heritability', 'genome', 'genes', 'ld', 'data'])\n",
      "('0000-0002-2618-750X', ['using', 'analysis', 'species', 'taxa', 'data', 'functional'])\n"
     ]
    }
   ],
   "source": [
    "def compare_people(limit):\n",
    "    for i, iData in personToOrcid.head(n=limit).iterrows():\n",
    "        print('### Comparing',i,iData.pure_person_id,iData.orcid_id)\n",
    "        #get all terms and tf-idf values\n",
    "        iTopTerms=topTerms[topTerms['orcid_id']==iData.orcid_id]\n",
    "        iTerms = iTopTerms['term']\n",
    "        jComp={}\n",
    "        for j in range(i+1,personToOrcid.shape[0]):\n",
    "            jData=personToOrcid.iloc[j]\n",
    "            jTopTerms=topTerms[topTerms['orcid_id']==jData.orcid_id]\n",
    "            jTerms = jTopTerms['term']\n",
    "            com = list(set(iTerms) & set(jTerms))\n",
    "            #only show maches with more than 1 matching term\n",
    "            if len(com) > 1:\n",
    "                jComp[jData.orcid_id]=com\n",
    "        #create sorted dictionary using number of items\n",
    "        jComp = sorted(jComp.items(), key=lambda kv: len(kv[1]), reverse=True)\n",
    "        for p in jComp[0:10]:\n",
    "            print(p)\n",
    "            \n",
    "compare_people(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This only shows us terms that are in common. It could be possible to have lots of terms in common, but these terms are not the 'most' representative of either person, or indeed in order of importance per person. It would be better to compare all terms, and their tf-idf values simultaneously.\n",
    "\n",
    "Back in the TF-IDF section (http://localhost:8888/lab#TF-IDF-using-sklearn) we created a similarity matrix from the TF-IDF model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0000-0001-5001-3350\n",
      "\t 0000-0001-5001-3350 0.9999999999999963\n",
      "\t 0000-0002-1407-8314 0.15780716812089102\n",
      "\t 0000-0003-2052-4840 0.12414499704387866\n",
      "\t 0000-0003-0920-1055 0.10864635054285894\n",
      "\t 0000-0002-7141-9189 0.0969238856366499\n",
      "0000-0001-5008-0705\n",
      "\t 0000-0001-5008-0705 0.9999999999999997\n",
      "\t 0000-0003-4416-4560 0.019256884487874857\n",
      "\t 0000-0002-2960-759X 0.019226598250719203\n",
      "\t 0000-0002-1904-3735 0.01522090025025024\n",
      "\t 0000-0003-3405-4843 0.01378333900366982\n",
      "0000-0001-5017-9473\n",
      "\t 0000-0001-5017-9473 1.00000000000002\n",
      "\t 0000-0002-8337-626X 0.11511608769135045\n",
      "\t 0000-0001-5396-3108 0.10194269665348185\n",
      "\t 0000-0002-2064-3359 0.10060758508049958\n",
      "\t 0000-0002-7706-8105 0.09074606109190216\n"
     ]
    }
   ],
   "source": [
    "#get similarity matrix for all people\n",
    "\n",
    "matrixCom = {}\n",
    "\n",
    "#get tfidf matrix and orcidText dictionary\n",
    "%store -r matrix\n",
    "%store -r token_dict\n",
    "\n",
    "#go through each element in matrix and compare to each other\n",
    "for i in range(0,len(matrix)):\n",
    "    iOrcid=list(token_dict)[i]\n",
    "    matrixCom[iOrcid]={}\n",
    "    for j in range(0,len(matrix)):\n",
    "        jOrcid=list(token_dict)[j]\n",
    "        matrixCom[iOrcid][jOrcid]=matrix[i][j]\n",
    "\n",
    "o = open(config.orcidToOrcid,'w')\n",
    "o.write('orcid_1\\torcid_2\\ttf-idf\\n')\n",
    "counter=0\n",
    "for m in matrixCom:\n",
    "    sorted_res = sorted(matrixCom[m].items(), key=lambda kv: kv[1], reverse=True)\n",
    "    if counter<3:\n",
    "        print(m)\n",
    "        for s in sorted_res[0:5]:\n",
    "            print('\\t',s[0],s[1])\n",
    "        counter+=1\n",
    "    for s in sorted_res:\n",
    "        o.write(m+'\\t'+s[0]+'\\t'+str(s[1])+'\\n')\n",
    "o.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Collaboration recommendation engine\n",
    "\n",
    "This similarity data depicts the similarity between each person's publication text based on tf-idf. Often, similarities arise due to co-publication, and perhaps a more informative recommender would be to idenfity cases where people have signficant overlap in their publication text, but have never previously co-published. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(279841, 3)\n",
      "               orcid_1              orcid_2    tf-idf\n",
      "0  0000-0001-5001-3350  0000-0001-5001-3350  1.000000\n",
      "1  0000-0001-5001-3350  0000-0002-1407-8314  0.157807\n",
      "2  0000-0001-5001-3350  0000-0003-2052-4840  0.124145\n",
      "3  0000-0001-5001-3350  0000-0003-0920-1055  0.108646\n",
      "4  0000-0001-5001-3350  0000-0002-7141-9189  0.096924\n"
     ]
    }
   ],
   "source": [
    "#load the new data into a dataframe\n",
    "\n",
    "orcidToOrcid = pd.read_csv(config.demoOrcidToOrcid,sep='\\t')\n",
    "print(orcidToOrcid.shape)\n",
    "print(orcidToOrcid.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0000-0002-1407-8314 33334    0.10258\n",
      "       term  tf-idf_x  tf-idf_y\n",
      "0    cancer  0.085914  0.079305\n",
      "1     genes  0.061123  0.050338\n",
      "2      data  0.054236  0.064645\n",
      "3     using  0.035485  0.060613\n",
      "4  analysis  0.032270  0.051746 \n",
      "\n",
      "0000-0003-1562-891X 33335    0.098\n",
      "            term  tf-idf_x  tf-idf_y\n",
      "0           cell  0.091965  0.151794\n",
      "1         cancer  0.085914  0.264330\n",
      "2         breast  0.059255  0.247853\n",
      "3  breast cancer  0.038263  0.198393\n",
      "4    cancer cell  0.032882  0.042547\n",
      "5    development  0.031938  0.032471 \n",
      "\n",
      "0000-0001-8186-5708 33340    0.074197\n",
      "     term  tf-idf_x  tf-idf_y\n",
      "0    cell  0.091965  0.274139\n",
      "1  cancer  0.085914  0.062854 \n",
      "\n",
      "0000-0002-2715-9930 33341    0.070891\n",
      "          term  tf-idf_x  tf-idf_y\n",
      "0       cancer  0.085914  0.042130\n",
      "1        genes  0.061123  0.115058\n",
      "2         data  0.054236  0.045467\n",
      "3   mechanisms  0.040592  0.030163\n",
      "4  development  0.031938  0.035598 \n",
      "\n",
      "0000-0002-6793-2262 33342    0.070718\n",
      "       term  tf-idf_x  tf-idf_y\n",
      "0      data  0.054236  0.051510\n",
      "1     using  0.035485  0.044307\n",
      "2  analysis  0.032270  0.040558 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#filter on co-publishing\n",
    "example_orcid='0000-0001-7328-4233'\n",
    "#example_orcid='0000-0003-0924-3247'\n",
    "example_pubs=orcidToPubmed[orcidToPubmed['orcid_id']==example_orcid]['pmid']\n",
    "\n",
    "def find_overlapping_tfidf(orcid1,orcid2):\n",
    "    o1=topTerms[topTerms['orcid_id']==orcid1]\n",
    "    o2=topTerms[topTerms['orcid_id']==orcid2]\n",
    "    m = o1.merge(o2, left_on='term', right_on='term')\n",
    "    return m.iloc[:, [1,2,4]]\n",
    "\n",
    "counter=0\n",
    "oColab=orcidToOrcid[orcidToOrcid['orcid_1']==example_orcid]['orcid_2']\n",
    "for o in oColab:\n",
    "    colabPubs=orcidToPubmed[orcidToPubmed['orcid_id']==o]['pmid']\n",
    "    pubCom=len(set(example_pubs).intersection(set(colabPubs)))\n",
    "    #find cases where there are no common publications\n",
    "    if pubCom==0:\n",
    "        if counter<5:\n",
    "            #get overlapping tf-idf terms\n",
    "            overlapPubs = find_overlapping_tfidf(example_orcid,o)\n",
    "            r = orcidToOrcid[(orcidToOrcid['orcid_1']==example_orcid) & (orcidToOrcid['orcid_2']==o)]\n",
    "            print(o,r['tf-idf'].to_string(header=False))\n",
    "            print(overlapPubs,'\\n')\n",
    "            counter+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Matching text to people\n",
    "\n",
    "Using the same tf-idf model, we can try and match any piece of text to the most relevant people. To do this we will compare the tf-idf vectors for a piece of text to each person using the cosine similarity of the vectors. For more info - https://janav.wordpress.com/2013/10/27/tf-idf-and-cosine-similarity/\n",
    "\n",
    ">The cosine measure similarity is another similarity metric that depends on envisioning user preferences as points in space.  Hold in mind the image of user preferences as points in an n-dimensional space. Now imagine two lines from the origin, or  point (0,0,…,0), to each of these two points. When two users are similar, they’ll have similar ratings, and so will be  relatively close in space—at least, they’ll be in roughly the same direction from the origin. The angle formed between these two lines will be relatively small. In contrast, when the two users are dissimilar, their points will be distant, and likely in different directions from the origin, forming a wide angle. This angle can be used as the basis for a similarity metric in the same way that the Euclidean distance was used to form a similarity metric. In this case, the cosine of the angle leads to a similarity value. <b>If you’re rusty on trigonometry, all you need to remember to understand this is that the cosine value is always between –1 and 1: the cosine of a small angle is near 1, and the cosine of a large angle near 180 degrees is close to –1. This is good, because small angles should map to high similarity, near 1, and large angles should map to near –1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#https://stackoverflow.com/questions/55677314/using-sklearn-how-do-i-calculate-the-tf-idf-cosine-similarity-between-documents\n",
    "%store -r tfs\n",
    "corpus_tfidf = tfs\n",
    "%store -r tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "#merge pure people data  to get names\n",
    "pure_people_orcid = personToOrcid.merge(pureToName, left_on='pure_person_id', right_on='pure_person_id')\n",
    "\n",
    "def compare_text(query,topMatches=5):\n",
    "    print('\\n','# Comparing text:',query[:100],'...')\n",
    "    query_tfidf = tfidf.transform([query])\n",
    "    \n",
    "    #get terms for query\n",
    "    feature_names = tfidf.get_feature_names()\n",
    "    res={}\n",
    "    for col in query_tfidf.nonzero()[1]:\n",
    "        res[feature_names[col]]=query_tfidf[0, col]\n",
    "        #reverse sort the results\n",
    "        sorted_res = sorted(res.items(), key=lambda kv: kv[1], reverse=True)\n",
    "    \n",
    "    #compare tfidf vectors using cosine similarity\n",
    "    cosineSimilarities = cosine_similarity(query_tfidf, corpus_tfidf).flatten()\n",
    "    cosineData={}\n",
    "    \n",
    "    #order cosine similarities \n",
    "    for c in range(0,len(cosineSimilarities)):\n",
    "        cosineData[c]=cosineSimilarities[c]\n",
    "    sortedCosineData = sorted(cosineData.items(), key=lambda kv: kv[1], reverse=True)\n",
    "    \n",
    "    #process the output and get more info for matches\n",
    "    for s in sortedCosineData[0:topMatches]:\n",
    "        #get pureID \n",
    "        orcid=list(token_dict)[s[0]]\n",
    "        p = pure_people_orcid[personToOrcid['orcid_id']==orcid]\n",
    "        if not p.empty:\n",
    "            print('\\n',s[1],orcid,p['person_name'].values[0])\n",
    "            #get overlapping terms\n",
    "            o1=topTerms[topTerms['orcid_id']==orcid]\n",
    "            o2=sorted_res[:100]\n",
    "            #create a dataframe for results\n",
    "            data=[]\n",
    "            for i in o2:\n",
    "                if i[0] in o1.term.values:\n",
    "                    data.append([i[0], i[1], o1[o1['term']==i[0]]['tf-idf'].values[0]])\n",
    "            df=pd.DataFrame(data, columns=[\"term\", \"doc-tfidf\", \"person-tdidf\"])\n",
    "            print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " # Comparing text: \n",
      "Diagnosis of Coronary Heart Diseases Using Gene Expression Profiling; Stable Coronary Artery Diseas ...\n",
      "\n",
      " 0.1498063952666541 0000-0002-2515-0852 Chiara Bucciarelliducci\n",
      "                    term  doc-tfidf  person-tdidf\n",
      "0             myocardial   0.190957      0.219544\n",
      "1               coronary   0.143442      0.122800\n",
      "2  myocardial infarction   0.143007      0.080905\n",
      "3             infarction   0.137624      0.082784\n",
      "4                 artery   0.126051      0.040733\n",
      "5                     mi   0.089625      0.032689\n",
      "6                disease   0.089497      0.071288\n",
      "7               patients   0.084028      0.109595\n",
      "\n",
      " 0.14473219216074815 0000-0003-0924-3247 Tom R Gaunt\n",
      "       term  doc-tfidf  person-tdidf\n",
      "0  coronary   0.143442      0.063617\n",
      "1   disease   0.089497      0.098377\n",
      "2      gene   0.061402      0.096811\n",
      "3     genes   0.049303      0.093182\n",
      "\n",
      " 0.14106423986554967 0000-0002-1753-3730 Gianni D Angelini\n",
      "              term  doc-tfidf  person-tdidf\n",
      "0       myocardial   0.190957      0.058452\n",
      "1         coronary   0.143442      0.180756\n",
      "2  coronary artery   0.136398      0.164692\n",
      "3           artery   0.126051      0.200135\n",
      "4          disease   0.089497      0.040263\n",
      "5         patients   0.084028      0.158127\n",
      "6         ischemia   0.060466      0.054850\n",
      "\n",
      " 0.09992455637447066 0000-0003-4638-601X Thomas W Johnson\n",
      "                    term  doc-tfidf  person-tdidf\n",
      "0             myocardial   0.190957      0.120962\n",
      "1               coronary   0.143442      0.111898\n",
      "2  myocardial infarction   0.143007      0.115848\n",
      "3             infarction   0.137624      0.114452\n",
      "4               patients   0.084028      0.054721\n",
      "\n",
      " 0.08853169646516287 0000-0001-7240-4563 Richard W Morris\n",
      "       term  doc-tfidf  person-tdidf\n",
      "0  coronary   0.143442      0.122728\n",
      "1        mi   0.089625      0.053986\n",
      "2   disease   0.089497      0.142586\n",
      "3  patients   0.084028      0.057852\n"
     ]
    }
   ],
   "source": [
    "#PubMed article 26930047\n",
    "query=\"\"\"\n",
    "Diagnosis of Coronary Heart Diseases Using Gene Expression Profiling; Stable Coronary Artery Disease, Cardiac Ischemia with and without Myocardial Necrosis.\n",
    "Cardiovascular disease (including coronary artery disease and myocardial infarction) is one of the leading causes of death in Europe, and is influenced by both \n",
    "environmental and genetic factors. With the recent advances in genomic tools and technologies there is potential to predict and diagnose heart disease using \n",
    "molecular data from analysis of blood cells. We analyzed gene expression data from blood samples taken from normal people (n = 21), non-significant coronary artery \n",
    "disease (n = 93), patients with unstable angina (n = 16), stable coronary artery disease (n = 14) and myocardial infarction (MI; n = 207). We used a feature \n",
    "selection approach to identify a set of gene expression variables which successfully differentiate different cardiovascular diseases. The initial features \n",
    "were discovered by fitting a linear model for each probe set across all arrays of normal individuals and patients with myocardial infarction. Three different \n",
    "feature optimisation algorithms were devised which identified two discriminating sets of genes, one using MI and normal controls (total genes = 6) and another \n",
    "one using MI and unstable angina patients (total genes = 7). In all our classification approaches we used a non-parametric k-nearest neighbour (KNN) classification \n",
    "method (k = 3). The results proved the diagnostic robustness of the final feature sets in discriminating patients with myocardial infarction from healthy controls. \n",
    "Interestingly it also showed efficacy in discriminating myocardial infarction patients from patients with clinical symptoms of cardiac ischemia but no myocardial \n",
    "necrosis or stable coronary artery disease, despite the influence of batch effects and different microarray gene chips and platforms.\n",
    "\"\"\"\n",
    "compare_text(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " # Comparing text: \n",
      "genome wide association gwas\n",
      " ...\n",
      "\n",
      " 0.32574371740903496 0000-0002-4680-3517 M U B St Pourcain\n",
      "          term  doc-tfidf  person-tdidf\n",
      "0         gwas   0.398830      0.057630\n",
      "1  association   0.204107      0.190989\n",
      "\n",
      " 0.2903208869809313 0000-0002-7141-9189 Nicholas John Timpson\n",
      "          term  doc-tfidf  person-tdidf\n",
      "0         gwas   0.398830      0.049034\n",
      "1  association   0.204107      0.217564\n",
      "\n",
      " 0.2810170481824614 0000-0003-2514-0889 Lavinia Paternoster\n",
      "          term  doc-tfidf  person-tdidf\n",
      "0         gwas   0.398830      0.061581\n",
      "1  association   0.204107      0.165009\n",
      "\n",
      " 0.19976744034069038 0000-0003-0920-1055 Gibran Hemani\n",
      "          term  doc-tfidf  person-tdidf\n",
      "0         gwas   0.398830      0.078694\n",
      "1       genome   0.273079      0.033522\n",
      "2  association   0.204107      0.082512\n",
      "\n",
      " 0.1813558169199878 0000-0002-1407-8314 George Davey Smith\n",
      "          term  doc-tfidf  person-tdidf\n",
      "0  association   0.204107      0.176515\n"
     ]
    }
   ],
   "source": [
    "query=\"\"\"\n",
    "genome wide association gwas\n",
    "\"\"\"\n",
    "compare_text(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " # Comparing text: \n",
      "quantum mechanics\n",
      " ...\n",
      "\n",
      " 0.28945476707203033 0000-0001-6184-1813 Jonathan C F Matthews\n",
      "                term  doc-tfidf  person-tdidf\n",
      "0  quantum mechanics   0.668059      0.042948\n",
      "1          mechanics   0.530568      0.031977\n",
      "2            quantum   0.521724      0.457318\n",
      "\n",
      " 0.24824713212689506 0000-0002-1066-3110 Anthony Laing\n",
      "      term  doc-tfidf  person-tdidf\n",
      "0  quantum   0.521724      0.416979\n",
      "\n",
      " 0.22624172866293998 0000-0002-3429-4890 Joshua W Silverstone\n",
      "      term  doc-tfidf  person-tdidf\n",
      "0  quantum   0.521724      0.384462\n",
      "\n",
      " 0.18502285880883648 0000-0003-2944-1988 Karoline Wiesner\n",
      "      term  doc-tfidf  person-tdidf\n",
      "0  quantum   0.521724      0.265114\n",
      "\n",
      " 0.18145107274095362 0000-0001-9645-0580 Raffaele Santagati\n",
      "      term  doc-tfidf  person-tdidf\n",
      "0  quantum   0.521724      0.351304\n"
     ]
    }
   ],
   "source": [
    "query=\"\"\"\n",
    "quantum mechanics\n",
    "\"\"\"\n",
    "compare_text(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " # Comparing text: the breast and ovarian cancer protein, brca1, in dna replication and repair\n",
      "\n",
      "16 may 2019, 13.00 - 14 ...\n",
      "\n",
      " 0.18563432508821895 0000-0002-4612-7141 Mark S Dillingham\n",
      "                       term  doc-tfidf  person-tdidf\n",
      "0                       dna   0.265248      0.497847\n",
      "1           dna replication   0.234502      0.026613\n",
      "2             recombination   0.214127      0.108374\n",
      "3  homologous recombination   0.208176      0.035161\n",
      "4                    repair   0.174419      0.082862\n",
      "5               replication   0.163561      0.063496\n",
      "6                homologous   0.144597      0.030190\n",
      "7                  proteins   0.130268      0.039437\n",
      "8                 resection   0.121233      0.032990\n",
      "9                   protein   0.081844      0.090686\n",
      "\n",
      " 0.12544075926064913 0000-0002-2501-6602 Mark D Szczelkun\n",
      "      term  doc-tfidf  person-tdidf\n",
      "0      dna   0.265248      0.452884\n",
      "1  protein   0.081844      0.046910\n",
      "\n",
      " 0.10514019174413414 0000-0002-2715-9930 Matthew J Suderman\n",
      "     term  doc-tfidf  person-tdidf\n",
      "0     dna   0.265248      0.328981\n",
      "1  cancer   0.127144      0.042130\n",
      "2   cells   0.027588      0.050363\n",
      "\n",
      " 0.10430409515346778 0000-0003-1562-891X Claire M Perks\n",
      "      term  doc-tfidf  person-tdidf\n",
      "0    brca1   0.312669      0.070731\n",
      "1   breast   0.128414      0.247853\n",
      "2   cancer   0.127144      0.264330\n",
      "3  cancers   0.043883      0.108111\n",
      "4  pathway   0.029157      0.033967\n",
      "5    cells   0.027588      0.182785\n",
      "\n",
      " 0.09764584325657875 0000-0003-2052-4840 Caroline L Relton\n",
      "     term  doc-tfidf  person-tdidf\n",
      "0     dna   0.265248      0.269643\n",
      "1  cancer   0.127144      0.076682\n",
      "2  genome   0.033277      0.052837\n"
     ]
    }
   ],
   "source": [
    "#load from file\n",
    "#https://www.bristol.ac.uk/cancer/events/2019/bioc-16may.html\n",
    "text=open('data/cancer-conf.txt','r').read()\n",
    "compare_text(text.lower(),topMatches=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Compare a person\n",
    "\n",
    "How would a person outside of this group of people compare. If you have an ORCID, or know of one, try it and see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 Getting ORCID data for 0000-0003-2628-8287\n",
      "3 PMIDs\n",
      "3 DOIs\n",
      "['25805555', '25333362', '27090530']\n",
      "Read existing downloaded pubmed data from output/pubmed.tsv\n",
      "219 publication(s) already downloaded\n",
      "Processing ['25805555', '25333362', '27090530']\n",
      "multiple pmid\n",
      "3 article(s) returned\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'config' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-74b7d18a479e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;31m#load publication ids for orcid data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0mpubData\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0morcidFile\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0morcid\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mpmid\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrstrip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\\t'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'config' is not defined"
     ]
    }
   ],
   "source": [
    "from scripts.common_functions import orcid_to_pubmedData\n",
    "\n",
    "#example_orcid='0000-0001-9918-058X'\n",
    "#example_orcid='0000-0002-8132-3885'\n",
    "#example_orcid='0000-0001-7615-8523'\n",
    "example_orcid='0000-0003-2628-8287'\n",
    "\n",
    "#get publication data fron orcid and write to file\n",
    "orcid_to_pubmedData([example_orcid])\n",
    "\n",
    "#load publication ids for orcid data\n",
    "pubData=set()\n",
    "with open(config.orcidFile,'r') as f:\n",
    "    for line in f:\n",
    "        orcid,pmid = line.rstrip().split('\\t')\n",
    "        if orcid==example_orcid:\n",
    "            pubData.add(pmid)\n",
    "#print(pubData)\n",
    "\n",
    "#load publication text matching PubMed IDs\n",
    "text=''\n",
    "with open(config.pubmedFile,'r') as f:\n",
    "    for line in f:\n",
    "        pmid,year,title,abstract = line.split('\\t')\n",
    "        if pmid in pubData:\n",
    "            pubData.add(pmid)\n",
    "            text+=title+' '+abstract.rstrip()\n",
    "if len(text)>10:\n",
    "    compare_text(text.lower(),topMatches=10)\n",
    "else:\n",
    "    print('Not enough text',len(text))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final thoughts\n",
    "\n",
    "This is a very crude demonstration of how you might do this kind of thing. Natural language processing methods are developing at an incredible rate, and there are many alternative methods, e.g. word2vec, doc2vec, BERT. We haven't covered many important areas, such as entity tagging, context aware text processing, stemming/lematizing, building graphs, network analysis, etc. Maybe next time :)\n",
    "\n",
    "There are also lots of alternative tools, e.g. Scopus, Scival, Fingerprints (Pure) but these are mainly closed source and provide no specific information on methods. I plan to develop the above ideas in relation to AXON and welcome collaborations and ideas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}