notebooks/schema_parsing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Schema Parsing Examples\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Path\n",
    "\n",
    "This notebook requires the `mdbrtools` package to be in the Python path.\n",
    "This code block is only required if you cloned the github repository and did not install `mdbrtools` via pip.\n",
    "\n",
    "Alternatively, you can install `mdbrtools` as a development dependency with `pip install -e .`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.append(\"..\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Schema Parsing Example\n",
    "\n",
    "The code block below loads a a few example documents from the JSON file and parses their schema.\n",
    "\n",
    "The `schema` object contains comprehensive information about the fields, types and values.\n",
    "\n",
    "Note that `dict(schema)` returns a short summary of the schema as Python dictionary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parsing schema: 100%|██████████| 10/10 [00:00<00:00, 45294.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'address': [{'counter': 3, 'type': 'document'}],\n",
      " 'address.city': [{'counter': 3, 'type': 'str'}],\n",
      " 'address.street': [{'counter': 3, 'type': 'str'}],\n",
      " 'address.zipcode': [{'counter': 2, 'type': 'str'}],\n",
      " 'age': [{'counter': 7, 'type': 'int'}],\n",
      " 'email': [{'counter': 8, 'type': 'str'}],\n",
      " 'hobbies': [{'counter': 3, 'type': 'array'}],\n",
      " 'hobbies.[]': [{'counter': 5, 'type': 'str'}],\n",
      " 'id': [{'counter': 10, 'type': 'int'}],\n",
      " 'name': [{'counter': 10, 'type': 'str'}],\n",
      " 'phone': [{'counter': 3, 'type': 'str'}],\n",
      " 'preferences': [{'counter': 3, 'type': 'document'}],\n",
      " 'preferences.newsletter': [{'counter': 3, 'type': 'bool'}],\n",
      " 'preferences.notifications': [{'counter': 2, 'type': 'array'}],\n",
      " 'preferences.notifications.[]': [{'counter': 3, 'type': 'str'}]}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from mdbrtools.schema import parse_schema\n",
    "import json\n",
    "from pprint import pprint\n",
    "\n",
    "# load example data\n",
    "with open(\"./example_docs.json\") as f:\n",
    "    docs = json.load(f)\n",
    "\n",
    "schema = parse_schema(docs)\n",
    "pprint(dict(schema))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Navigating the `schema` object\n",
    "\n",
    "At the high level, a `Schema` is a nested tree structure which contains `Field` objects, and `Field` objects contain `Type` objects.\n",
    "\n",
    "Subclasses of `Type` are\n",
    "\n",
    "- `Array`\n",
    "- `Document`\n",
    "- `PrimitiveType`\n",
    "\n",
    "`PrimitiveType` represents leaves in this tree representing data types such as int, str, float, etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parsing schema: 100%|██████████| 3/3 [00:00<00:00, 66576.25it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Types of things:\n",
      " - found 1 of type str: ['string_thing']\n",
      " - found 1 of type int: [123]\n",
      " - found 1 of type bool: [False]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from mdbrtools.schema import parse_schema\n",
    "\n",
    "docs = [\n",
    "    {\"things\": \"string_thing\"},\n",
    "    {\"things\": 123},\n",
    "    {\"things\": False},\n",
    "]\n",
    "\n",
    "schema = parse_schema(docs)\n",
    "\n",
    "# access types of `things`\n",
    "things_types = schema[\"things\"].types\n",
    "\n",
    "print(\"\\nTypes of things:\")\n",
    "for type_name, type in things_types.items():\n",
    "    print(f\" - found {type.count} of type {type_name}: {type.values}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing Primitive Values\n",
    "\n",
    "To retrieve a set of primitive values for a particular field path, you could navigate to the primitive value object\n",
    "as shown above. But there is a convenience helper method: `schema.get_prim_values()`.\n",
    "\n",
    "To align with MongoDB's query language semantics, `get_prim_values()` automatically _dives_ one level into an array structure. This can be disabled by setting `dive_into_arrays=False` as additional argument.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parsing schema: 100%|██████████| 1/1 [00:00<00:00, 12520.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Schema:\n",
      "{'_id': [{'counter': 1, 'type': 'ObjectId'}],\n",
      " 'a0': [{'counter': 1, 'type': 'array'}],\n",
      " 'a0.[]': [{'counter': 1, 'type': 'document'}],\n",
      " 'a0.[].a1': [{'counter': 1, 'type': 'array'}],\n",
      " 'a0.[].a1.[]': [{'counter': 1, 'type': 'document'}],\n",
      " 'a0.[].a1.[].a2': [{'counter': 1, 'type': 'array'}],\n",
      " 'a0.[].a1.[].a2.[]': [{'counter': 2, 'type': 'document'}],\n",
      " 'a0.[].a1.[].a2.[].number': [{'counter': 2, 'type': 'int'}]}\n",
      "\n",
      "Values for inner number:\n",
      "{1, 2}\n",
      "\n",
      "Values for inner number (dive_into_arrays=False):\n",
      "{1, 2}\n",
      "\n",
      "Values for inner number (navigating directly):\n",
      "[1, 2]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from bson import ObjectId\n",
    "from mdbrtools.schema import parse_schema\n",
    "from pprint import pprint\n",
    "\n",
    "docs = [\n",
    "    {\n",
    "        \"_id\": ObjectId(\"6657c93f0c261ad8866ed948\"),\n",
    "        \"a0\": [{\"a1\": [{\"a2\": [{\"number\": 1}, {\"number\": 2}]}]}],\n",
    "    }\n",
    "]\n",
    "\n",
    "schema = parse_schema(docs)\n",
    "\n",
    "# print basic schema\n",
    "print(\"Schema:\")\n",
    "pprint(dict(schema))\n",
    "\n",
    "# get values for a field path\n",
    "# by default, get_prim_values() automatically dives into one level of array nesting at each level\n",
    "print(\"\\nValues for inner number:\")\n",
    "print(schema.get_prim_values(\"a0.a1.a2.number\"))\n",
    "\n",
    "# same as...\n",
    "print(\"\\nValues for inner number (dive_into_arrays=False):\")\n",
    "print(schema.get_prim_values(\"a0.[].a1.[].a2.[].number\", dive_into_arrays=False))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}