02-LoadCSVOneToMany-ACogSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4077b4ee-73ea-4155-afbd-1de66dd6b650",
   "metadata": {},
   "source": [
    "# Load CSVs (one-to-many) to Azure Cognitive Search\n",
    "\n",
    "In this Jupyter Notebook, we create and run steps to index a CSV file in which each row is an indivual and independent record/document. Each row then becomes searchable in Azure Cognitive Search. \n",
    "The reference documentation can be found at [Indexing blobs and files to produce multiple search documents](https://learn.microsoft.com/en-us/azure/search/search-howto-index-one-to-many-blobs).\n",
    "\n",
    "By default, an indexer will treat the contents of a blob or file as a single search document. If you want a more granular representation in a search index, you can set parsingMode values to create multiple search documents from one blob or file.\n",
    "\n",
    "We are going to be using a public Blob Storage container that has abstracts of ~52k Medical publications about COVID-19 published in 2020. You can check the website [HERE](https://www.ncbi.nlm.nih.gov/research/coronavirus/)\n",
    "\n",
    "If you want to download the dataset, go [HERE](https://ftp.ncbi.nlm.nih.gov/pub/lu/LitCovid/topic_tagger/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c088c844-1e71-4279-a8fe-a77a007c15c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import requests\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "# Set the Data source connection string. This is the location of the CSV with the COVID articles on each line. \n",
    "# You can change it and use your own data\n",
    "DATASOURCE_CONNECTION_STRING = \"DefaultEndpointsProtocol=https;AccountName=demodatasetsp;AccountKey=IOzmP701+W8yNZ2/TqSjGjYqV7mWxgCDeyDccU/8hACS7cqBBYbPtOjNAoQpbx6V+BKvUuRhesuN+AStY+XI0Q==;EndpointSuffix=core.windows.net\"\n",
    "DATASOURCE_SAS_TOKEN = \"?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\"\n",
    "BLOB_CONTAINER_NAME = \"litcovid\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c4908539-1d17-46a3-b9e0-dcc46a210c4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the names for the data source, index and indexer\n",
    "datasource_name = \"cogsrch-datasource-csv\"\n",
    "skillset_name = \"cogsrch-skillset-csv\"\n",
    "index_name = \"cogsrch-index-csv\"\n",
    "indexer_name = \"cogsrch-indexer-csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f2434379-070e-4110-8f5a-7d5bda9a0b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}\n",
    "params = {'api-version': os.environ['AZURE_SEARCH_API_VERSION']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecad0e75-e3c8-4147-b8c6-b938435bc8f5",
   "metadata": {},
   "source": [
    "## Create Data Source (Blob container with the Litcovid CSV data file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a9fa6c09-a489-4b6d-8c93-5fc26bae63a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create a data source\n",
    "\n",
    "datasource_payload = {\n",
    "    \"name\": datasource_name,\n",
    "    \"description\": \"Demo files to demonstrate cognitive search capabilities of one-to-many.\",\n",
    "    \"type\": \"azureblob\",\n",
    "    \"credentials\": {\n",
    "        \"connectionString\": DATASOURCE_CONNECTION_STRING\n",
    "    },\n",
    "    \"container\": {\n",
    "        \"name\": BLOB_CONTAINER_NAME\n",
    "    }\n",
    "}\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/datasources/\" + datasource_name,\n",
    "                 data=json.dumps(datasource_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0b3935d-8546-4756-95cd-7f4fcecb9836",
   "metadata": {},
   "source": [
    "## Create Skillset - Text Splitter, Language Detection\n",
    "We will use cognitive services enrichment for spliting the text of each content field into chunks (pages) and for language detection. We should always split the text since we don't know how big the content of each row might be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b46cfa90-28b4-4602-b6ff-743a3407fd72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create a skillset\n",
    "skillset_payload = {\n",
    "    \"name\": skillset_name,\n",
    "    \"description\": \"Splits Text and detect language\",\n",
    "    \"skills\":\n",
    "    [\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.LanguageDetectionSkill\",\n",
    "            \"description\": \"If you have multilingual content, adding a language code is useful for filtering\",\n",
    "            \"context\": \"/document\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\": \"text\",\n",
    "                  \"source\": \"/document/abstract\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"languageCode\",\n",
    "                  \"targetName\": \"language\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.SplitSkill\",\n",
    "            \"context\": \"/document\",\n",
    "            \"textSplitMode\": \"pages\",\n",
    "            \"maximumPageLength\": 5000, # 5000 is default\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\",\n",
    "                    \"source\": \"/document/abstract\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"languageCode\",\n",
    "                    \"source\": \"/document/language\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"textItems\",\n",
    "                    \"targetName\": \"pages\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"cognitiveServices\": {\n",
    "        \"@odata.type\": \"#Microsoft.Azure.Search.CognitiveServicesByKey\",\n",
    "        \"description\": os.environ['COG_SERVICES_NAME'],\n",
    "        \"key\": os.environ['COG_SERVICES_KEY']\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/skillsets/\" + skillset_name,\n",
    "                 data=json.dumps(skillset_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b7ff86-19fc-48d3-88d1-b098e8d01302",
   "metadata": {},
   "source": [
    "## Inspect CSV file so we can understand the column types before creating the Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b38d49d5-57ce-4bea-a5d8-d6356be64ee5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Download the csv files to disk and inspect using pandas\n",
    "import pandas as pd\n",
    "remote_file_path = \"https://demodatasetsp.blob.core.windows.net/litcovid/train.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aaac918a-8859-45f5-9519-2cf56bfded88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of lines: 52419\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pmid</th>\n",
       "      <th>journal</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "      <th>keywords</th>\n",
       "      <th>label</th>\n",
       "      <th>pub_type</th>\n",
       "      <th>authors</th>\n",
       "      <th>date1</th>\n",
       "      <th>doi</th>\n",
       "      <th>date2</th>\n",
       "      <th>label_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>32410266</td>\n",
       "      <td>J Med Virol</td>\n",
       "      <td>Immunoregulation with mTOR inhibitors to preve...</td>\n",
       "      <td>Coronavirus disease 2019 (COVID-19) has become...</td>\n",
       "      <td>ade;antibody-dependent enhancement;coronavirus...</td>\n",
       "      <td>Treatment;Mechanism</td>\n",
       "      <td>Journal Article;Systematic Review</td>\n",
       "      <td>Zheng, Yunfeng;Li, Renfeng;Liu, Shunai</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.1002/jmv.26009</td>\n",
       "      <td>2020-05-16</td>\n",
       "      <td>title_abstract_abstract</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>33052950</td>\n",
       "      <td>PLoS One</td>\n",
       "      <td>Measuring the resilience of criminogenic ecosy...</td>\n",
       "      <td>This paper uses resilience as a lens through w...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Journal Article;Research Support, Non-U.S. Gov't</td>\n",
       "      <td>Borrion, Herve;Kurland, Justin;Tilley, Nick;Ch...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.1371/journal.pone.0240077</td>\n",
       "      <td>2020-10-15</td>\n",
       "      <td>abstract_only</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>32589531</td>\n",
       "      <td>Br J Hosp Med (Lond)</td>\n",
       "      <td>Pulmonary embolism in acute medicine: a case-b...</td>\n",
       "      <td>Pulmonary embolism remains an important cause ...</td>\n",
       "      <td>covid-19;catheter-directed thrombolysis;pulmon...</td>\n",
       "      <td>Prevention</td>\n",
       "      <td>Case Reports;Journal Article;Review</td>\n",
       "      <td>Stevenson, Alexander;Davis, Sarah;Murch, Nick</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.12968/hmed.2020.0300</td>\n",
       "      <td>2020-06-27</td>\n",
       "      <td>title_abstract_abstract</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32835070</td>\n",
       "      <td>Groundw Sustain Dev</td>\n",
       "      <td>A positive perspective during COVID-19 related...</td>\n",
       "      <td>The months from March to June refer as water c...</td>\n",
       "      <td>covid-19;groundwater;positive perspective;rain...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Journal Article</td>\n",
       "      <td>Patni, Kiran;Jindal, Manoj Kumar</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.1016/j.gsd.2020.100420</td>\n",
       "      <td>2020-08-25</td>\n",
       "      <td>abstract_only</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32620125</td>\n",
       "      <td>J Transl Med</td>\n",
       "      <td>The timeline and risk factors of clinical prog...</td>\n",
       "      <td>BACKGROUND: The novel coronavirus disease 2019...</td>\n",
       "      <td>covid-19;clinical progression;pneumonia;retros...</td>\n",
       "      <td>Treatment;Diagnosis</td>\n",
       "      <td>Journal Article;Research Support, Non-U.S. Gov't</td>\n",
       "      <td>Wang, Fang;Qu, Mengyuan;Zhou, Xuan;Zhao, Kai;L...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.1186/s12967-020-02423-8</td>\n",
       "      <td>2020-07-06</td>\n",
       "      <td>title_abstract_abstract</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       pmid               journal   \n",
       "0  32410266           J Med Virol  \\\n",
       "1  33052950              PLoS One   \n",
       "2  32589531  Br J Hosp Med (Lond)   \n",
       "3  32835070   Groundw Sustain Dev   \n",
       "4  32620125          J Transl Med   \n",
       "\n",
       "                                               title   \n",
       "0  Immunoregulation with mTOR inhibitors to preve...  \\\n",
       "1  Measuring the resilience of criminogenic ecosy...   \n",
       "2  Pulmonary embolism in acute medicine: a case-b...   \n",
       "3  A positive perspective during COVID-19 related...   \n",
       "4  The timeline and risk factors of clinical prog...   \n",
       "\n",
       "                                            abstract   \n",
       "0  Coronavirus disease 2019 (COVID-19) has become...  \\\n",
       "1  This paper uses resilience as a lens through w...   \n",
       "2  Pulmonary embolism remains an important cause ...   \n",
       "3  The months from March to June refer as water c...   \n",
       "4  BACKGROUND: The novel coronavirus disease 2019...   \n",
       "\n",
       "                                            keywords                label   \n",
       "0  ade;antibody-dependent enhancement;coronavirus...  Treatment;Mechanism  \\\n",
       "1                                                NaN                  NaN   \n",
       "2  covid-19;catheter-directed thrombolysis;pulmon...           Prevention   \n",
       "3  covid-19;groundwater;positive perspective;rain...                  NaN   \n",
       "4  covid-19;clinical progression;pneumonia;retros...  Treatment;Diagnosis   \n",
       "\n",
       "                                           pub_type   \n",
       "0                 Journal Article;Systematic Review  \\\n",
       "1  Journal Article;Research Support, Non-U.S. Gov't   \n",
       "2               Case Reports;Journal Article;Review   \n",
       "3                                   Journal Article   \n",
       "4  Journal Article;Research Support, Non-U.S. Gov't   \n",
       "\n",
       "                                             authors  date1   \n",
       "0             Zheng, Yunfeng;Li, Renfeng;Liu, Shunai    NaN  \\\n",
       "1  Borrion, Herve;Kurland, Justin;Tilley, Nick;Ch...    NaN   \n",
       "2      Stevenson, Alexander;Davis, Sarah;Murch, Nick    NaN   \n",
       "3                   Patni, Kiran;Jindal, Manoj Kumar    NaN   \n",
       "4  Wang, Fang;Qu, Mengyuan;Zhou, Xuan;Zhao, Kai;L...    NaN   \n",
       "\n",
       "                            doi       date2           label_category  \n",
       "0             10.1002/jmv.26009  2020-05-16  title_abstract_abstract  \n",
       "1  10.1371/journal.pone.0240077  2020-10-15            abstract_only  \n",
       "2       10.12968/hmed.2020.0300  2020-06-27  title_abstract_abstract  \n",
       "3     10.1016/j.gsd.2020.100420  2020-08-25            abstract_only  \n",
       "4    10.1186/s12967-020-02423-8  2020-07-06  title_abstract_abstract  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(remote_file_path+DATASOURCE_SAS_TOKEN)\n",
    "print(\"No. of lines:\",df.shape[0])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b1e39922-7023-4a34-9274-3d29a942d2a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pmid                int64\n",
       "journal            object\n",
       "title              object\n",
       "abstract           object\n",
       "keywords           object\n",
       "label              object\n",
       "pub_type           object\n",
       "authors            object\n",
       "date1             float64\n",
       "doi                object\n",
       "date2              object\n",
       "label_category     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a321916-cd14-4d34-837d-1d153edb1221",
   "metadata": {},
   "source": [
    "## Create the Index\n",
    "In Azure Cognitive Search, both blob indexers and file indexers support a delimitedText parsing mode for CSV files that treats each line in the CSV as a separate search document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5284b80c-9ba6-49d6-8109-5bfdbaa6ddc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "index_payload = {\n",
    "    \"name\": index_name,  \n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"searchable\": \"false\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"false\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"title\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"content\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"language\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"true\", \"filterable\": \"true\", \"facetable\": \"true\"},\n",
    "        {\"name\": \"pages\",\"type\": \"Collection(Edm.String)\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"journal\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"keywords\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"label\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"true\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"pub_type\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"authors\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"date1\", \"type\": \"Edm.Double\", \"searchable\": \"false\", \"retrievable\": \"true\", \"facetable\": \"true\", \"filterable\": \"true\", \"sortable\": \"true\"},\n",
    "        {\"name\": \"doi\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"date2\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"label_category\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"true\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"metadata_storage_name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"metadata_storage_path\", \"type\":\"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"filterable\": \"false\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"metadata_storage_last_modified\", \"type\":\"Edm.DateTimeOffset\", \"searchable\": \"false\", \"retrievable\": \"false\", \"filterable\": \"false\", \"sortable\": \"false\"}\n",
    "    ],\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": \n",
    "                        {\n",
    "                            \"fieldName\": \"title\"\n",
    "                        },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        { \n",
    "                            \"fieldName\":\"content\" \n",
    "                        }\n",
    "                    ],\n",
    "                    \"prioritizedKeywordsFields\": [\n",
    "                        {\n",
    "                          \"fieldName\": \"keywords\"\n",
    "                        }\n",
    "                    ]\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + index_name,\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51849738-6f66-452a-b7df-d34afd11f943",
   "metadata": {},
   "source": [
    "## Create and Run the Indexer - (runs the pipeline)\n",
    "To create one-to-many indexers with CSV blobs, create or update an indexer definition with the delimitedText parsing mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b87b8ebd-8091-43b6-9124-cc17021cfb78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "indexer_payload = {\n",
    "    \"name\": indexer_name,\n",
    "    \"dataSourceName\": datasource_name,\n",
    "    \"targetIndexName\": index_name,\n",
    "    \"skillsetName\": skillset_name,\n",
    "    \"schedule\" : { \"interval\" : \"PT2H\"},\n",
    "    \"fieldMappings\": [\n",
    "        {\n",
    "          \"sourceFieldName\" : \"pmid\",\n",
    "          \"targetFieldName\" : \"id\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"abstract\",\n",
    "          \"targetFieldName\" : \"content\"\n",
    "        }\n",
    "    ],\n",
    "    \"outputFieldMappings\":\n",
    "    [\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/language\",\n",
    "            \"targetFieldName\": \"language\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/pages/*\",\n",
    "            \"targetFieldName\": \"pages\"\n",
    "        }\n",
    "    ],\n",
    "    \"parameters\" : { \n",
    "        \"configuration\" : { \n",
    "            \"dataToExtract\": \"contentAndMetadata\",\n",
    "            \"parsingMode\" : \"delimitedText\", \n",
    "            \"firstLineContainsHeaders\" : True,\n",
    "            \"delimitedTextDelimiter\": \",\"\n",
    "        } \n",
    "    }\n",
    "}\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name,\n",
    "                 data=json.dumps(indexer_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6132c041-7213-410e-a206-1a8c7385128e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "Status: inProgress\n",
      "Items Processed: 25000\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Optionally, get indexer status to confirm that it's running\n",
    "r = requests.get(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name +\n",
    "                 \"/status\", headers=headers, params=params)\n",
    "# pprint(json.dumps(r.json(), indent=1))\n",
    "print(r.status_code)\n",
    "print(\"Status:\",r.json().get('lastResult').get('status'))\n",
    "print(\"Items Processed:\",r.json().get('lastResult').get('itemsProcessed'))\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2152806f-245c-45db-93c6-c19c0569d73a",
   "metadata": {},
   "source": [
    "**When the indexer finishes running we will have all 52419 rows indexed properly as separate documents in our Search Engine!.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eed6f22-437f-4a49-9b67-5fa2e7d066bf",
   "metadata": {},
   "source": [
    "# Reference\n",
    "\n",
    "- https://learn.microsoft.com/en-us/azure/search/search-howto-index-csv-blobs\n",
    "- https://learn.microsoft.com/en-us/azure/search/knowledge-store-create-rest\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d9f82a9-cb4c-44b9-b125-bc124ea23aa8",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "Now that we have two separete indexes loaded with two different types of information, In the next notebook 3, we will do a Multi-Index query, sort the results based on the reranker semantic score of Azure Search, and then use OpenAI to understand this results and give the best answer possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d993b09-7095-41b0-bea0-4fb19387f5fe",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}