07_ingest.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp ingest.ingest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\r\n",
    "from nbdev.showdoc import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exporti\r\n",
    "\r\n",
    "import re\r\n",
    "import pymongo\r\n",
    "import json\r\n",
    "import sc2reader\r\n",
    "import errno\r\n",
    "import jsonschema\r\n",
    "import os\r\n",
    "\r\n",
    "import pandas as pd\r\n",
    "import numpy as np\r\n",
    "\r\n",
    "from typing import *\r\n",
    "from pathlib import Path\r\n",
    "from pprint import pprint\r\n",
    "from jsonschema import validate\r\n",
    "from dataclasses import dataclass, astuple, asdict, field\r\n",
    "\r\n",
    "from sc2reader.engine.plugins import APMTracker\r\n",
    "\r\n",
    "from sc_training.ingest import *\r\n",
    "sc2reader.engine.register_plugin(APMTracker())\r\n",
    "sc2reader.engine.register_plugin(CtrlGroupTracker())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide \r\n",
    "\r\n",
    "test_data_path = (Path(Path.cwd()/'test_replays') \r\n",
    "                  if Path('test_replays').exists() \r\n",
    "                  else Path('../../test_replays'))\r\n",
    "\r\n",
    "test_batch_path = (test_data_path / \"TestProfilerBatch\")\r\n",
    "\r\n",
    "assert test_data_path.exists()\r\n",
    "assert test_data_path.is_dir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9 - The main ingest module "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\r\n",
    "\r\n",
    "This module defines the `inventory_replays` function following the design proposed in the Ingestion Sequence Diagram (see <<2 - Data Ingestion and Clustering Process>>). It exports these functions into the ingest module of the ingest sub-package.\r\n",
    "\r\n",
    "### Exportable Members\r\n",
    "- `inventory_replays`\r\n",
    "\r\n",
    "#### Exportable Helper functions\r\n",
    "- `load_configurations`\r\n",
    "- `set_up_db`\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inventoring and Storing `replays` collection\n",
    "\n",
    "To inventory the replays in a replay batch I can use the `get_replay_info` defined in the `summarise_rpl` module (see Section 1.1).\n",
    "\n",
    "In the following code, I load a batch of replays and I print the summary of the first two replays using the `get_replay_info` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File path:                   c:\\Users\\david\\Documents\\phdcode\\sc_training\\test_replays\\TestProfilerBatch\\16-Bit LE (2).SC2Replay \n",
      "File name:                   16-Bit LE (2).SC2Replay \n",
      "Date (datetime.datetime):    2021-06-04 03:49:10 \n",
      "Duration (seconds):          707 \n",
      "Game type:                   1v1 \n",
      "Game release:                5.0.7.84643 \n",
      "Map:                         16-Bit LE \n",
      "Game category:               Private \n",
      "winner:                      1 \n",
      "players:                     [(1, 'HDEspino', 'Terran', 'Win'), (2, 'A.I. 1 (Harder)', 'Zerg', 'Loss')] \n",
      "\n",
      "<class 'sc_training.ingest.summarise_rpl.Replay_data'>\n",
      "File path:                   c:\\Users\\david\\Documents\\phdcode\\sc_training\\test_replays\\TestProfilerBatch\\16-Bit LE (3).SC2Replay \n",
      "File name:                   16-Bit LE (3).SC2Replay \n",
      "Date (datetime.datetime):    2021-06-08 02:04:37 \n",
      "Duration (seconds):          384 \n",
      "Game type:                   1v1 \n",
      "Game release:                5.0.7.84643 \n",
      "Map:                         16-Bit LE \n",
      "Game category:               Private \n",
      "winner:                      1 \n",
      "players:                     [(1, 'HDEspino', 'Protoss', 'Win'), (2, 'A.I. 1 (Easy)', 'Protoss', 'Loss')] \n",
      "\n",
      "<class 'sc_training.ingest.summarise_rpl.Replay_data'>\n"
     ]
    }
   ],
   "source": [
    "replay_batch = sc2reader.load_replays(str(test_batch_path))\r\n",
    "\r\n",
    "for i, rpl in enumerate(replay_batch):\r\n",
    "    print(get_replay_info(rpl))\r\n",
    "    print(type(get_replay_info(rpl)))\r\n",
    "    if i == 1:\r\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Storing inventory in Database\r\n",
    "\r\n",
    "I can store the `Replay_data` objects returned by `get_replay_info` in a document-based database. Storing this inventory in a database means that other processes could access it to navigate the information built by the ingest process.\r\n",
    "\r\n",
    "`sc_training`, uses a `config.json` file, which users should create and store in the working project's data folder, to set up a MongoDB local client. Using this client and the information in the config file the solution also creates the database. The following code loads this file and defines the loading procedure for when this module is imported. \r\n",
    "\r\n",
    "I will divide this loading process into three steps. \r\n",
    "\r\n",
    "#### 1 - Handle Config file\r\n",
    "I locate and load the `config` file. This location process allows users to create a custom `config.json` in a data directory in their projects. That file can be used to customise the name of the database, the port address and number for the MongoDB client, and the location of the replays the user wants to process. Once the file is located, I also define a procedure to ensure it contains the necessary information to connect with the MongoDB client and access the appropriate database.\r\n",
    "\r\n",
    "Apart from this option, the code also defines how the module can default to a `config` file stored in the library's data folder if the user fails to provide this file. This default file allows the system to attempt to function based on the assumption that the users are using a Windows computer and have a traditional installation of StarCraft II and MongoDB. Of course, this default set-up would not be expected to work in all cases, but it provides an option and a sample of the `config` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exporti\r\n",
    "\r\n",
    "# This section defines the helper functions I use in the \r\n",
    "# config file loading and checking procedure\r\n",
    "# To check that the config file contains all necessary information for \r\n",
    "# the solution's proper working I use the following jsonschema   \r\n",
    "Config_schema = {\r\n",
    "    \"type\": \"object\",\r\n",
    "    \"properties\":{\r\n",
    "        \"DB_NAME\": {\"type\":\"string\"},\r\n",
    "        \"PORT_ADDRESS\":  {\"type\":\"string\"},\r\n",
    "        \"PORT_NUMBER\": {\"type\":\"number\"},\r\n",
    "        \"REPLAY_PATH\": {\"type\":\"string\"}\r\n",
    "    }\r\n",
    "}\r\n",
    "\r\n",
    "\r\n",
    "def validate_config_file(file: Path, schema: Dict[str, Any]) -> bool:\r\n",
    "    \"\"\"This helper function uses the json schema defined above to \r\n",
    "    make sure that the config file includes all the information \r\n",
    "    necessary for the solution's proper work\"\"\"\r\n",
    "    try:\r\n",
    "        validate(file, schema)\r\n",
    "    except jsonschema.exceptions.ValidationError as err:\r\n",
    "        print(err)\r\n",
    "        print(\"config.json does not conform to the required specifications\")\r\n",
    "        raise err\r\n",
    "    except jsonschema.exceptions.SchemaError as err:\r\n",
    "        print(err)\r\n",
    "        print(\"The Config_schema is invalid\")\r\n",
    "        raise err\r\n",
    "    \r\n",
    "    return True\r\n",
    "\r\n",
    "# Using the validate_config_file function I can check that the config \r\n",
    "# file exists and has the proper information for sc_training to work.\r\n",
    " \r\n",
    "def open_config_file(config_file: Path) -> dict[str, Any]:\r\n",
    "    \"\"\"This helper function verifies the existence of the config file\r\n",
    "    and that it has the proper data. If so it imports the data into a\r\n",
    "    dict that can be used to access such data in the program\"\"\"\r\n",
    "    try:\r\n",
    "        if not config_file.exists():\r\n",
    "            raise FileNotFoundError\r\n",
    "        \r\n",
    "        validate_config_file(json.load(config_file.open()), Config_schema)\r\n",
    "        \r\n",
    "        with config_file.open('r') as cf:\r\n",
    "            return json.load(cf)\r\n",
    "\r\n",
    "    except FileNotFoundError as err:\r\n",
    "        print('config.json not found')\r\n",
    "        raise err\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `load_configurations` function, uses various internal helper functions to locate, open, verify and load the information from a project's `config.json` file. It stores this information as a `Config_settings` object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\r\n",
    "@dataclass\r\n",
    "class Config_settings:\r\n",
    "    \"\"\"This type of object stores the data extracted from the config file.\r\n",
    "    \r\n",
    "    *Attributes*\r\n",
    "        - port_address: str\r\n",
    "            Address of the MongoDB Client that the program will connect to.\r\n",
    "        - port_number: int\r\n",
    "            Port number of the client located in the address above\r\n",
    "        - db_name: str\r\n",
    "            Name of the project's data base\r\n",
    "        - replay_path: str\r\n",
    "            Path to the replays that must be analysed and stored in the \r\n",
    "            database\r\n",
    "\"\"\"\r\n",
    "    port_address: str\r\n",
    "    port_number: int\r\n",
    "    db_name: str\r\n",
    "    replay_path: str\r\n",
    "\r\n",
    "    def __str__(self):\r\n",
    "        headers = [\"Port Address: \",\"Port Number: \",\r\n",
    "                  \"DB Name: \",\"Replays file: \"]\r\n",
    "        strings = [f'{h:<15}{att:>40}\\n' for h, att\r\n",
    "                   in zip(headers, astuple(self))]\r\n",
    "        return ''.join(strings)\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"Config_settings\" class=\"doc_header\"><code>class</code> <code>Config_settings</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h4>\n",
       "\n",
       "> <code>Config_settings</code>(**`port_address`**:`str`, **`port_number`**:`int`, **`db_name`**:`str`, **`replay_path`**:`str`)\n",
       "\n",
       "This type of object stores the data extracted from the config file.\n",
       "\n",
       "*Attributes*\n",
       "    - port_address: str\n",
       "        Address of the MongoDB Client that the program will connect to.\n",
       "    - port_number: int\n",
       "        Port number of the client located in the address above\n",
       "    - db_name: str\n",
       "        Name of the project's data base\n",
       "    - replay_path: str\n",
       "        Path to the replays that must be analysed and stored in the \n",
       "        database"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(Config_settings, title_level=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\r\n",
    "def load_configurations() -> Config_settings:\r\n",
    "    \"\"\"Loads the project's configuration information.\r\n",
    "\r\n",
    "    This function locates, verifies and extracts the project's configuration\r\n",
    "    data. This data tells sc_training where to find the replays it needs to\r\n",
    "    inventory and process, how to connect to the MongoDB client it will use\r\n",
    "    to store this data in a database, and the name of the database it \r\n",
    "    should use.\r\n",
    "\r\n",
    "    *Args*\r\n",
    "        - None\r\n",
    "\r\n",
    "    *Returns*\r\n",
    "        - Config_settings\r\n",
    "\r\n",
    "    *Errors*\r\n",
    "        - FileNotFound\r\n",
    "            If there is no valid config file\r\n",
    "        - jsonschema.exceptions.ValidationError\r\n",
    "            If the config file does not contain the necessary data or does\r\n",
    "            not conform to the proper schema necessary to work.\r\n",
    "    \"\"\"\r\n",
    "\r\n",
    "    config_file = (Path(Path.cwd()/'data/config.json') \r\n",
    "               if Path(Path.cwd()/'data/config.json').exists()\r\n",
    "               else Path(Path(__file__)/'../../../data/config.json'))\r\n",
    "\r\n",
    "    config_dict = open_config_file(config_file)\r\n",
    "    return Config_settings(\r\n",
    "        config_dict['PORT_ADDRESS'],\r\n",
    "        config_dict['PORT_NUMBER'],\r\n",
    "        config_dict['DB_NAME'],\r\n",
    "        config_dict['REPLAY_PATH']\r\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following sample code shows the use of `load_configurations` to set up a test data base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_settings = load_configurations()\r\n",
    "mongo_client = pymongo.MongoClient(db_settings.port_address, \r\n",
    "                                   db_settings.port_number)\r\n",
    "worcking_bd = mongo_client[db_settings.db_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "config.json content:\n",
      "Port Address:                                 localhost\n",
      "Port Number:                                      27017\n",
      "DB Name:                                   TEST_library\n",
      "Replays file:          .\\test_replays\\TestProfilerBatch\n",
      "\n",
      "Local Mongo DB Client\n",
      "MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True)\n",
      "\n",
      "Database: \n",
      "Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'TEST_library')\n"
     ]
    }
   ],
   "source": [
    "print('config.json content:')\r\n",
    "print(db_settings)\r\n",
    "print('Local Mongo DB Client')\r\n",
    "print(mongo_client, end='\\n\\n')\r\n",
    "print('Database: ')\r\n",
    "print(worcking_bd)\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 - Index replay batch data in the \"replays\" collection\r\n",
    "Once the database is set, I store the main descriptive data of each replay into the `replays` collection of the database. This information is crucial to be able to iterate through the batch and also to have an index that indicates what players played which matches. It also can be used to review other information about the match, for instance, was the match a ranked match? or what races did each player played with.\r\n",
    "\r\n",
    "To extract this data I use the `get_replay_info` function from the `summarise_rpl` module (see <<3 - Summarising Replays>>).\r\n",
    "\r\n",
    "To collect this data I loop through the replays in a file inserting the return values of the functions from the ingest sub-package's different modules into collections within the database. \r\n",
    "\r\n",
    "> Note: in the example, I set up a new database called `sample_db` that I will use in this notebook for illustration purposes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 added to replays\n",
      "153 already existed in replays\n"
     ]
    }
   ],
   "source": [
    "sample_db = mongo_client['sample_db']\r\n",
    "collection = sample_db['replays']\r\n",
    "\r\n",
    "replay_batch = sc2reader.load_replays(db_settings.replay_path)\r\n",
    "\r\n",
    "count_add = 0\r\n",
    "count_existed = 0\r\n",
    "for rpl in replay_batch:\r\n",
    "    if not collection.count_documents({'replay_name': rpl.filename}, \r\n",
    "                                      limit = 1):\r\n",
    "        # print(f'Adding {Path(rpl.filename).name} to replays collection.')\r\n",
    "        collection.insert_one(asdict(get_replay_info(rpl)))\r\n",
    "        count_add += 1\r\n",
    "    else:\r\n",
    "        count_existed += 1\r\n",
    "        # print(rpl.filename, \"already exists in the replay_info collection.\")\r\n",
    "\r\n",
    "print(f\"{count_add} added to replays\")\r\n",
    "print(f\"{count_existed} already existed in replays\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the loop finishes, the collection has been created in the database. The following snippet shows that this collection is now composed of several documents containing the indexing data for each replay. Additionally, I added a conditional statement within the loop to check if the collection already contains a replay. In that case, the loop will print a warning and avoid inserting repeated documents into the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The collection now contains: 153 documents\n"
     ]
    }
   ],
   "source": [
    "print('The collection now contains:', \r\n",
    "      collection.estimated_document_count(), 'documents')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3 - Building the `indicators` collection\r\n",
    "\r\n",
    "Once a replay's descriptive data is stored in the `replays` collections, I need to also extract and store the performance indicators for each player in the match in the `indicators` collection.\r\n",
    "\r\n",
    "In the following code, I illustrate how I can use all the functions defined in the `ingest` sub-package to build the indicators of two players in a sample replay. Each player's indicators are stored in a single flat dictionary, i.e. a dictionary that has no nested data structures as values.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exporti\r\n",
    "# The following helper function helps eliminate multiple levels of \r\n",
    "# nesting in a dictionary\r\n",
    "def flatten_indicators(nested_value: dict) -> dict[str, float]:\r\n",
    "    if isinstance(list(nested_value.values())[0], dict):\r\n",
    "        output_dict = {}\r\n",
    "        for k, nested in nested_value.items():\r\n",
    "            for nested_k, v in nested.items():\r\n",
    "                output_dict[f'{k}_{nested_k}'] = v\r\n",
    "        return output_dict\r\n",
    "    elif isinstance(list(nested_value.values())[0], tuple):\r\n",
    "        output_dict = {}\r\n",
    "        order = ['first', 'second']\r\n",
    "        for k, nested in nested_value.items():\r\n",
    "            for ord, v in zip(order, nested):\r\n",
    "                output_dict[f'{ord}_{k}'] = str(v)\r\n",
    "        return output_dict\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing: test_replays\\Jagannatha LE.SC2Replay\n",
      "Processing player: pid:1 Player 1 - HDEspino (Protoss)\n",
      "Processing player: pid:2 Player 2 - MxChrisxM (Terran)\n"
     ]
    }
   ],
   "source": [
    "#\r\n",
    "# First I locate the sample file.\r\n",
    "sample_replay = [sc2reader.load_replay('test_replays\\Jagannatha LE.SC2Replay')]\r\n",
    "\r\n",
    "# I create two list of all the functions from the ingest sub-package that I use\r\n",
    "# to collect the player's performance indicators.\r\n",
    "# I use two lists, because I need to make sure that all functions in the \r\n",
    "# list have the same caller structure.\r\n",
    "simple_functions = [get_player_macro_econ_stats,\r\n",
    "                    get_expan_times,\r\n",
    "                    get_expan_counts,\r\n",
    "                    calc_attack_ratio,\r\n",
    "                    calc_ctrlg_ratio,\r\n",
    "                    count_max_active_groups,\r\n",
    "                    calc_get_ctrl_grp_ratio,\r\n",
    "                    calc_select_ratio,\r\n",
    "                    list_player_upgrades,\r\n",
    "                    calc_spe_abil_ratios]\r\n",
    "\r\n",
    "double_functions = [count_composition,\r\n",
    "                    count_started]\r\n",
    "\r\n",
    "# I loop through the replays in sample replay, and through the players\r\n",
    "# in each replay. In each  storing the indicators of each player as a \r\n",
    "# dictionary in the indicators_list. \r\n",
    "indicators_list = []\r\n",
    "for rpl in sample_replay:\r\n",
    "    print(f'Processing: {rpl.filename}')\r\n",
    "    for pid, player in rpl.player.items():\r\n",
    "        print(f'Processing player: pid:{pid} {player}')\r\n",
    "        \r\n",
    "        # Declare the dict that will contain all of the players performance\r\n",
    "        # indicators\r\n",
    "        rpl_indicators = {}\r\n",
    "        \r\n",
    "        # Run through all functions that need the caller arguments rpl\r\n",
    "        # (for the replay being analysed) and pid (for the player id of\r\n",
    "        # the player being parsed) and that return a flat dict. \r\n",
    "        for func in simple_functions:\r\n",
    "            rpl_indicators.update(func(rpl, pid))\r\n",
    "\r\n",
    "        # Run through all functions that need the caller arguments rpl\r\n",
    "        # (for the replay being analysed), pid (for the player id of\r\n",
    "        # the player being parsed) and a flag to focus on extracting data \r\n",
    "        # from the player's building or troops, and that return a flat dict.\r\n",
    "        for func in double_functions:\r\n",
    "            for flag in [True, False]:\r\n",
    "                rpl_indicators.update(flatten_indicators(func(rpl, pid, flag)))\r\n",
    "\r\n",
    "        # I run this last function appart, because I need to flatten the\r\n",
    "        # output so that the resulting dictionary has no nested levels. \r\n",
    "        v = get_prefered_spec_abil(rpl, pid)\r\n",
    "        rpl_indicators.update(flatten_indicators(v))\r\n",
    "                \r\n",
    "        indicators_list.append(rpl_indicators)\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result of the code above, `indicators_list` now contains two dictionaries that store the indicators for each player. In the `inventory_replays` function, I store this data in the database's `indicators` collection instead of a list. In the previous code I used a list to illustrate how the process returns the following results:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of players evaluated:  2\n",
      "First set of indicators belongs to: HDEspino\n",
      "First set of indicators contains:  388 indicators.\n",
      "Second set of indicators belongs to: MxChrisxM\n",
      "Second set of indicators contains:  377 indicators.\n"
     ]
    }
   ],
   "source": [
    "print('Number of players evaluated: ',len(indicators_list))\r\n",
    "print('First set of indicators belongs to:', \r\n",
    "      indicators_list[0]['player_username'])\r\n",
    "print('First set of indicators contains: ',\r\n",
    "        len(indicators_list[0]), 'indicators.')\r\n",
    "print('Second set of indicators belongs to:', \r\n",
    "      indicators_list[1]['player_username'])\r\n",
    "print('Second set of indicators contains: ',\r\n",
    "        len(indicators_list[1]), 'indicators.')      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tip: In some cases, each player's indicators lists would contain a different number of indicators. This difference exists because the list stores separate counts for the players' units and buildings according to their race during the match. Remember that each one of the game races has a different number of available units. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exportable functions \r\n",
    "This section defines `inventory_replays`. Users can pass a directory containing multiple `.SC2Replay` files to this function and it will extract all data from these files and store it in the `replays` and `indicators` collection of the projects database (as defined in the `config.json` file) following the logic explained above. \r\n",
    "\r\n",
    "Internally, the function uses three helper functions:\r\n",
    "\r\n",
    "- `set_up_db`: connects to the MongoDB client and loads the working database.\r\n",
    "- `verify_replays_path`: makes sure that the path past by the user is valid.\r\n",
    "- `build_indicators`: runs through the indicator extraction loop and stores the results in the indicators collections.\r\n",
    "\r\n",
    "Of these, I export the `set_up_db`, given that it can be useful in other modules (see for example <<10 - Player Profiler>>)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\r\n",
    "def set_up_db() -> pymongo.database.Database:\r\n",
    "    \"\"\"Loads the database specified in the project's config.json file.\r\n",
    "\r\n",
    "    *Returns*\r\n",
    "        - pymongo.database.Database\r\n",
    "            Python object that allows the user to interact with the\r\n",
    "            database specified in the project's config file. \r\n",
    "    \"\"\"\r\n",
    "    db_settings = load_configurations()\r\n",
    "    mongo_client = pymongo.MongoClient(db_settings.port_address, \r\n",
    "                                    db_settings.port_number)\r\n",
    "    worcking_bd = mongo_client[db_settings.db_name]\r\n",
    "\r\n",
    "    return worcking_bd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is a sample run of the `set_up_db` function to illustrate how it can be used to connect to a mongo database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pymongo.database.Database'>\n"
     ]
    }
   ],
   "source": [
    "worcking_bd = set_up_db()\r\n",
    "print(type(worcking_bd))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exporti\r\n",
    "# Helper function that verifies the path of the where the replays should \r\n",
    "# be located according to the config file. \r\n",
    "def verify_replays_path(rpl_path: Any) -> Path:\r\n",
    "\r\n",
    "    if isinstance(rpl_path, str):\r\n",
    "        path = Path(rpl_path)\r\n",
    "    elif not isinstance(rpl_path, Path):\r\n",
    "        string = 'replay_path must be of type str or Path, not ' \r\n",
    "        raise TypeError((string + str(type(replay_batch))))\r\n",
    "    else:\r\n",
    "        path = rpl_path\r\n",
    "\r\n",
    "    if not path.exists():\r\n",
    "        raise ValueError(f'{path} is not valid location')\r\n",
    "\r\n",
    "    return path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exporti\r\n",
    "# Helper function that extracts the indicators for each matc's players and \r\n",
    "# stores it in the indicators collection. \r\n",
    "def build_indicators(rpl: sc2reader.resources.Replay,\r\n",
    "                    working_db: pymongo.database.Database) -> None:\r\n",
    "    \r\n",
    "    \"\"\"Runs through the indicator extraction loop and stores the results \r\n",
    "    in the indicators collections.\r\n",
    "    \"\"\"\r\n",
    "    simple_functions = [get_player_macro_econ_stats,\r\n",
    "                        get_expan_times,\r\n",
    "                        get_expan_counts,\r\n",
    "                        calc_attack_ratio,\r\n",
    "                        calc_ctrlg_ratio,\r\n",
    "                        count_max_active_groups,\r\n",
    "                        calc_get_ctrl_grp_ratio,\r\n",
    "                        calc_select_ratio,\r\n",
    "                        list_player_upgrades,\r\n",
    "                        calc_spe_abil_ratios,\r\n",
    "                        calc_apms\r\n",
    "                        ]\r\n",
    "\r\n",
    "    double_functions = [count_composition,\r\n",
    "                        count_started]\r\n",
    "    indi_collect = working_db['indicators']\r\n",
    "    for pid in rpl.player.keys():  \r\n",
    "        rpl_indicators = {}\r\n",
    "        for func in simple_functions:\r\n",
    "            rpl_indicators.update(func(rpl, pid))\r\n",
    "\r\n",
    "        v = get_prefered_spec_abil(rpl, pid)\r\n",
    "        rpl_indicators.update(flatten_indicators(v))\r\n",
    "\r\n",
    "        for func in double_functions:\r\n",
    "            for flag in [True, False]:\r\n",
    "                rpl_indicators.update(flatten_indicators(func(rpl, pid, flag)))\r\n",
    "\r\n",
    "        rpl_ind = ({k: v \r\n",
    "                    if (not (isinstance(v, np.int64) \r\n",
    "                        or isinstance(v, np.float64)))\r\n",
    "                    else float(v) for k, v in rpl_indicators.items()})\r\n",
    "        \r\n",
    "        indi_collect.insert_one(rpl_ind)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\r\n",
    "def inventory_replays() -> None:\r\n",
    "    \"\"\"This function builds two collections within the database \r\n",
    "    specified in the config.json file.\r\n",
    "\r\n",
    "    The replay information will be stored in the database specified in \r\n",
    "    cwd/data/config.json in the following collections:\r\n",
    "    - `replays`\r\n",
    "        Stores the metadata of the replays, can be used for indexing and\r\n",
    "        for finding the other replays.\r\n",
    "    - `inicators`\r\n",
    "        Store the indicators for each performance of every player.\r\n",
    "\r\n",
    "    *Args:*\r\n",
    "        - replay_batch\r\n",
    "            Directory address where the replays to process are located.\r\n",
    "\r\n",
    "    *Return:*\r\n",
    "        -None\r\n",
    "\r\n",
    "    \"\"\"\r\n",
    "    project_config = load_configurations()\r\n",
    "    working_db = set_up_db()\r\n",
    "    rpls_collect = working_db['replays']\r\n",
    "    path = verify_replays_path(project_config.replay_path)\r\n",
    "\r\n",
    "    replays = sc2reader.load_replays(str(path))\r\n",
    "    \r\n",
    "    load_count = 0\r\n",
    "    process_count = 0\r\n",
    "    previous = 0\r\n",
    "    ignored = 0\r\n",
    "    \r\n",
    "    print(f'Inventorying replays at: {path} in database {working_db.name}')\r\n",
    "\r\n",
    "    for rpl in replays:\r\n",
    "        process_count += 1\r\n",
    "        if (not (rpl.type == \"1v1\")):\r\n",
    "            ignored += 1\r\n",
    "            # print(f'{rpl.filename}is not 1v1')\r\n",
    "            continue\r\n",
    "        if not rpls_collect.count_documents({'replay_name': rpl.filename}, \r\n",
    "                                            limit = 1):\r\n",
    "            # print(f'Processing {rpl.filename}')\r\n",
    "            rpls_collect.insert_one(asdict(get_replay_info(rpl)))\r\n",
    "            build_indicators(rpl, working_db)\r\n",
    "            load_count += 1\r\n",
    "        else:\r\n",
    "            previous += 1\r\n",
    "            # print(rpl.filename, \"already exists in the replay_info rpls_collect.\")\r\n",
    "        \r\n",
    "    print(f'Load complete.')\r\n",
    "    print(f'{process_count} files processed')\r\n",
    "    print(f'{load_count} files loaded')\r\n",
    "    print(f'{ignored} files ignored')\r\n",
    "    print(f'{previous} files alredy existed')\r\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the function once, I get the following sample results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inventorying replays at: test_replays\\TestProfilerBatch in database TEST_library\n",
      "Load complete.\n",
      "153 files processed\n",
      "149 files loaded\n",
      "4 files ignored\n",
      "0 files alredy existed\n"
     ]
    }
   ],
   "source": [
    "# \r\n",
    "mongo_client.drop_database('TEST_library')\r\n",
    "inventory_replays()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['replays', 'indicators']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collections = [col for col in worcking_bd.list_collection_names()]\r\n",
    "collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "replays has 149 records.\n",
      "indicators has 298 records.\n"
     ]
    }
   ],
   "source": [
    "for col in collections:\r\n",
    "    print(f'{col} has {worcking_bd[col].estimated_document_count()} records.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 00_Comp_model.ipynb.\n",
      "Converted 01_01_ingest_and_clustering.ipynb.\n",
      "Converted 01_summarise_rpl.ipynb.\n",
      "Converted 02_handle_tracker_events.ipynb.\n",
      "Converted 03_macro_econ_parser.ipynb.\n",
      "Converted 04_build_parser.ipynb.\n",
      "Converted 05_handle_command_events.ipynb.\n",
      "Converted 06_selection_parser.ipynb.\n",
      "Converted 07_ingest.ipynb.\n",
      "Converted 08_profiler.ipynb.\n",
      "Converted index.ipynb.\n"
     ]
    }
   ],
   "source": [
    "#hide\r\n",
    "from nbdev.export import notebook2script\r\n",
    "notebook2script()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('sctraining_env': conda)",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}