dclist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The DC Recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataCamp is a great website for learning skills for data science. They have a lot of courses, mostly in R and Python, around all kinds of DS techniques, everything from manipulating time series data, to data viz courses, to different kinds of machine learning etc.\n",
    "\n",
    "The courses are laid out in 'tracks', which group courses together under a more general topic. These can be 'Skill Tracks' or 'Career Tracks', and you can select any one track on the DC website to guide your learning. A side effect of this grouping is that some courses appear in multiple tracks. On completion of a track, you get a certificate, which can not only be shared on LinkedIn, but also makes you feel warm and fuzzy inside...\n",
    "\n",
    "Ahem.. \n",
    "\n",
    "The issue I have identified is exemplified by DataCamp's 'Career Tracks', which are longer than the 'Skill Tracks' and cover a broader range of topics. By following a longer Career Track, due to the overlap with the shorter Skill Tracks, en route you may inadvertently complete the requisite material for multiple Skill Tracks (or be very close!). However, by not being enrolled on those Skill Tracks, you have no idea this is the case, and so you don't get those flashy DC certificate along the way despite you having done the required work - outrageous!\n",
    "\n",
    "The goal is clear: work out a way of recommending courses based on how close they get you to finishing a track, so that you can rack up those treasured Track certificates as you learn.\n",
    "\n",
    "The recommender should include some key ideas, such as:\n",
    "- Course length (some courses are longer than others)\n",
    "- How many tracks a course appears in (if it appears in more tracks, it brings you closer to multiple certificates)\n",
    "- Some measure of how close a course gets you to completing a track (the closer, the better!)\n",
    "\n",
    "Whilst DataCamp doesn't provide the above information directly, it is possible to infer by going through the various course pages on the website. It is also important to say that some tracks are organised into a logical progession of knowledge, which is particularly important with the more fundamental courses - any recommendation of a particular course may simply direct you to the appropriate track to follow in the original order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The DataCamp website contains over 50 tracks, which each contain multiple courses. It would take a long time to manually collect the data for this, so this should be scraped from the website where possible. All processing of the data should be as automated as possible. It would be great to get to a point where I could simply run a script and get up-to-date recommendations.\n",
    "\n",
    "Whilst the traffic provided by scraping the website course pages is negligible for a company like DC, this should be kept to a minimum whilst testing and coding out of principle. Minimising repeated scraping of the same data will minimise traffic and save me time, so data should be saved / pickled along the way.\n",
    "\n",
    "An additional thing to note: I keep all my DC course certificates in a Github repo. This was initially set up as a DC profile duplicate due to some public access issue on the DC website. That issue has since been resolved, so now I just use that repo as certificate storage, and is something that can be scraped to track my DC progress.\n",
    "\n",
    "Right, enough talking, time to code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "import undetected_chromedriver as uc\n",
    "from bs4 import BeautifulSoup\n",
    "from selenium.webdriver.chrome.options import Options\n",
    "from typing import Tuple, List\n",
    "\n",
    "# import requests\n",
    "# from selenium import webdriver\n",
    "# from selenium.webdriver.chrome.service import Service\n",
    "# from webdriver_manager.chrome import ChromeDriverManager"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping a track"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataCamp is already proving slightly troublesome. I could not just access the website's pages as HTML using `requests` as they use JavaScript rendering in the browser to return the required content.\n",
    "\n",
    "Annoying, but not uncommon - Selenium is a fab package that renders a webpage for you, so problem solved? Alas, the default driver `webdriver` from Selenium doesn't work to access the page either, as the page request is redirected through Cloudflare, which seems to block bot access presumably for DDOS protection (I'm not a evil bot, honest!).\n",
    "\n",
    "The solution? The `undetected_chromedriver` patch reconfigures a few settings and so does not trigger CloudFlare's bot-vision, and works a treat!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set webpage to open in headless browser\n",
    "options = Options()\n",
    "options.headless = True\n",
    "options.add_argument(\"--window-size=1920,1200\")\n",
    "# driver = webdriver.Chrome(options=options, service=Service(ChromeDriverManager().install()))\n",
    "# driver = webdriver.Chrome(options=options)\n",
    "driver = uc.Chrome(options=options)\n",
    "\n",
    "# Set a test webpage to open\n",
    "URL = \"https://www.datacamp.com/tracks/time-series-with-python\"\n",
    "\n",
    "# Render page\n",
    "driver.get(URL)\n",
    "# Store the markup\n",
    "page_source = driver.page_source\n",
    "driver.quit()\n",
    "# Parse with Beautiful Soup\n",
    "soup = BeautifulSoup(page_source, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BeautifulSoup is a well-known library for parsing html code. Exploring the rendered html, I hit Ctrl+F to find the data I want and identify the tags/class names used for the titles, descriptions, and times for each course. Lets see what Beautiful Soup can do..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['Manipulating Time Series Data in Python',\n",
       "  'Time Series Analysis in Python',\n",
       "  'Visualizing Time Series Data in Python',\n",
       "  'ARIMA Models in Python',\n",
       "  'Machine Learning for Time Series Data in Python'],\n",
       " [\"In this course you'll learn the basics of working with time series data.\",\n",
       "  \"In this course you'll learn the basics of analyzing time series data.\",\n",
       "  'Visualize seasonality, trends and other patterns in your time series data. ',\n",
       "  'Learn about ARIMA models in Python and become an expert in time series analysis.',\n",
       "  'This course focuses on feature engineering and machine learning for time series data. '],\n",
       " ['4 hours', '4 hours', '4 hours', '4 hours', '4 hours'])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "course_name = soup.find_all('strong', class_ = 'css-1dbp6pz-TrackContentCard')\n",
    "course_desc = soup.find_all('p', class_ = 'css-r9ojyg-TrackContentCard')\n",
    "course_time = soup.find_all('p', class_ = 'css-1jr04uj-TrackContentCard')\n",
    "\n",
    "course_name = [x.text for x in course_name]\n",
    "course_desc = [x.text for x in course_desc]\n",
    "course_time = [x.text for x in course_time]\n",
    "\n",
    "course_name, course_desc, course_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful! This lets me pull just the info I need in a neat way. Let's get this into a function so I can just plug in a URL and get what I want. It makes sense to split this into 2 functions so I can also just retrieve the HTML of other pages if required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_html(dc_url) -> BeautifulSoup:\n",
    "    \"\"\"\n",
    "    Get the HTML of a page\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    dc_url : str\n",
    "        URL of a page\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    BeautifulSoup\n",
    "        BeautifulSoup object of the page\n",
    "    \"\"\"\n",
    "    # Set webpage to open in headless browser\n",
    "    options = Options()\n",
    "    options.headless = True\n",
    "    options.add_argument(\"--window-size=1920,1200\")\n",
    "    # New browser instance started each time function is called to avoid errors from server limits - don't abuse this!\n",
    "    driver = uc.Chrome(options=options)\n",
    "    \n",
    "    # Render page and save page source\n",
    "    driver.get(dc_url)\n",
    "    page_source = driver.page_source\n",
    "    driver.quit()\n",
    "    \n",
    "    # Parse with Beautiful Soup\n",
    "    soup = BeautifulSoup(page_source, \"html.parser\")\n",
    "    return soup\n",
    "\n",
    "\n",
    "def get_courses(dc_url) -> list:\n",
    "    \"\"\"\n",
    "    Get list of courses from tracks on datacamp.com\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    dc_url : str\n",
    "        URL of datacamp.com track page\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    list\n",
    "        List of courses\n",
    "    \"\"\"\n",
    "    soup = get_html(dc_url)\n",
    "    # Get the courses info\n",
    "    course_names = soup.find_all('strong', class_ = 'css-1dbp6pz-TrackContentCard')\n",
    "    course_names = [x.text for x in course_names]\n",
    "    course_descs = soup.find_all('p', class_ = 'css-r9ojyg-TrackContentCard')\n",
    "    course_descs = [x.text.strip() for x in course_descs]   # Strip whitespace from description\n",
    "    course_times = soup.find_all('p', class_ = 'css-1jr04uj-TrackContentCard')\n",
    "    course_times = [x.text for x in course_times]\n",
    "    \n",
    "    # Combine into a list of tuples\n",
    "    courses = list(zip(course_names, course_descs, course_times))\n",
    "    return courses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's give it a spin on a random track page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Introduction to Deep Learning in Python',\n",
       "  'Learn the fundamentals of neural networks and how to build deep learning models using Keras 2.0.',\n",
       "  '4 hours'),\n",
       " ('Introduction to TensorFlow in Python',\n",
       "  'Learn the fundamentals of neural networks and how to build deep learning models using TensorFlow.',\n",
       "  '4 hours'),\n",
       " ('Introduction to Deep Learning with PyTorch',\n",
       "  'Learn to create deep learning models with the PyTorch library.',\n",
       "  '4 hours'),\n",
       " ('Introduction to Deep Learning with Keras',\n",
       "  'Learn to start developing deep learning models with Keras.',\n",
       "  '4 hours'),\n",
       " ('Advanced Deep Learning with Keras',\n",
       "  'Build multiple-input and multiple-output deep learning models using Keras.',\n",
       "  '4 hours')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_track = get_courses(\"https://www.datacamp.com/tracks/deep-learning-in-python\")\n",
    "test_track"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, the function works fine.\n",
    "\n",
    "Next up, I need to get a list of the tracks and their web addresses. I could compile this manually as there are far fewer tracks than courses, but lets see if this can be scraped also."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping the tracks lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('R Programming', 'R', 'https://www.datacamp.com/tracks/r-programming'),\n",
       " ('Importing & Cleaning Data',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/importing-cleaning-data-with-r'),\n",
       " ('Data Visualization',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/data-visualization-with-r'),\n",
       " ('Data Manipulation',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/data-manipulation-with-r')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the html of the skill tracks page\n",
    "st_url = \"https://www.datacamp.com/tracks/skill\"\n",
    "st_soup = get_html(st_url)\n",
    "\n",
    "# Get the names of the skill tracks\n",
    "# track_names = st_soup.find_all('h3', class_ = 'css-so6h6r-TrackCard')     # The class is not consistent between reloads, use heading 3 only instead\n",
    "track_names = st_soup.find_all('h3')[:53]\n",
    "track_names = [x.text.strip() for x in track_names]\n",
    "\n",
    "# Get the technologies used in the skill tracks\n",
    "track_techs = st_soup.find_all('title')[1:54]\n",
    "track_techs = [x.text.strip() for x in track_techs]\n",
    "\n",
    "# To get the web addresses, using the class tags doesn't work, as some of the weblinks have different html classes\n",
    "# Instead, first find all hrefs from the page where the reference begins with '/tracks/'\n",
    "track_urls = st_soup.find_all('a', href = lambda x: x and x.startswith('/tracks/'))\n",
    "track_urls = [x['href'] for x in track_urls]\n",
    "# The urls are relative, so need to add the base url\n",
    "track_urls = [\"https://www.datacamp.com\" + x for x in track_urls]\n",
    "# Remove links not pointing to a skill track\n",
    "track_urls = track_urls[8:-2]\n",
    "track_urls\n",
    "\n",
    "# Finally, combine the names, technologies, and urls into a list of tuples\n",
    "skill_tracks = list(zip(track_names, track_techs, track_urls))\n",
    "# Show the first few skill tracks\n",
    "skill_tracks[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This worked, but required some literal indexing to remove irrelevant data. Not the greatest solution, but functional for now. If in future more tracks get added to DataCamp, this indexing may need re-examining, particularly if the layout of the page changes such that the number of tags to ignore from the beginning and end changes.\n",
    "Let's see if we can get the career tracks as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Data Analyst', 'R', 'https://www.datacamp.com/tracks/data-analyst-with-r'),\n",
       " ('Data Scientist',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/data-scientist-with-r'),\n",
       " ('Data Analyst',\n",
       "  'Python',\n",
       "  'https://www.datacamp.com/tracks/data-analyst-with-python'),\n",
       " ('Data Scientist',\n",
       "  'Python',\n",
       "  'https://www.datacamp.com/tracks/data-scientist-with-python')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the html of the career tracks page\n",
    "ct_url = \"https://www.datacamp.com/tracks/career\"\n",
    "ct_soup = get_html(ct_url)\n",
    "\n",
    "# Get the names of the career tracks\n",
    "track_names = ct_soup.find_all('h3', class_ = 'css-so6h6r-TrackCard')\n",
    "track_names = [x.text.strip() for x in track_names]\n",
    "\n",
    "# Get the technologies of the career tracks\n",
    "track_techs = ct_soup.find_all('title')[1:15]\n",
    "track_techs = [x.text.strip() for x in track_techs]\n",
    "\n",
    "# To get the web addresses, using the class tags doesn't work, as some of the weblinks have different html classes\n",
    "# Instead, first find all hrefs from the page where the reference begins with '/tracks/'\n",
    "track_urls = ct_soup.find_all('a', href = lambda x: x and x.startswith('/tracks/'))\n",
    "track_urls = [x['href'] for x in track_urls]\n",
    "# The urls are relative, so need to add the base url\n",
    "track_urls = [\"https://www.datacamp.com\" + x for x in track_urls]\n",
    "# Remove links not pointing to a career track\n",
    "track_urls = track_urls[8:-2]\n",
    "\n",
    "# Finally, combine the names, technologies and urls into a list of tuples\n",
    "career_tracks = list(zip(track_names, track_techs, track_urls))\n",
    "# Show the first few career tracks\n",
    "career_tracks[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine the skill and career tracks into one list\n",
    "all_tracks = skill_tracks + career_tracks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once scraped, pickle to avoid re-running code. As I come back to this, I can then just read in the pickle and avoid scraping again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to a text file\n",
    "with open('all_tracks.txt', 'w') as f:\n",
    "    for track in all_tracks:\n",
    "        # Seperate each element in the tuple with ', '\n",
    "        f.write(', '.join(track) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['R Programming', 'R', 'https://www.datacamp.com/tracks/r-programming\\n'],\n",
       " ['Importing & Cleaning Data',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/importing-cleaning-data-with-r\\n'],\n",
       " ['Data Visualization',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/data-visualization-with-r\\n'],\n",
       " ['Data Manipulation',\n",
       "  'R',\n",
       "  'https://www.datacamp.com/tracks/data-manipulation-with-r\\n']]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the text file back in\n",
    "with open('all_tracks.txt', 'r') as f:\n",
    "    all_tracks = f.readlines()\n",
    "    all_tracks = [x.split(', ') for x in all_tracks]\n",
    "# Check the first few lines\n",
    "all_tracks[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrape all courses for all tracks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right, with a list of tracks, techs, and their web addresses, plus a method to scrape the course info from a track webpage, we should be able to get the course list for every track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_track_generator(file_object):\n",
    "    \"\"\"\n",
    "    Generator function to read a text file line by line\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        file_object : file\n",
    "            File object to read\n",
    "    \"\"\"\n",
    "    \n",
    "    # Loop indefinitely until end of the file\n",
    "    while True:\n",
    "        # Read a track name and web address from the file\n",
    "        data = file_object.readline()\n",
    "        # Break if this is the end of the file\n",
    "        if not data:\n",
    "            break\n",
    "\n",
    "        # Change data object to list\n",
    "        data = data.split(\",\")\n",
    "        # Remove newline character and spaces\n",
    "        data = [x.strip() for x in data]\n",
    "\n",
    "        # Yield the line of data\n",
    "        yield data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let the scraping commence!\n",
    "\n",
    "As a side note: had to do a little bit of QA, just to make sure the courses read in correctly. If any tracks are empty, then that dictionary entry is deleted and we re-scrape it. In practice, this happened seemingly at random to just a couple of courses, and may be from datacamp sometimes changing the class names of objects when generating the page. This is slightly strange behaviour, and indicates that I shouldn't rely on HTML class names in the future, but for now this is a minor issue and easily managed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_tracks = dict()\n",
    "# Loop with check variable of `read_tracks` to check if any tracks haven't read correctly\n",
    "read_tracks = 1\n",
    "while read_tracks > 0:\n",
    "\n",
    "    # Open a connection to the file\n",
    "    with open('all_tracks.txt') as f:\n",
    "\n",
    "        # Create a generator object for the file\n",
    "        gen_file = read_track_generator(f)\n",
    "\n",
    "        # Loop through the generator object\n",
    "        while True:\n",
    "            try:\n",
    "                # Read the next track's details\n",
    "                this_track = next(gen_file)\n",
    "                # Break if this is the end of the file\n",
    "                if not this_track:\n",
    "                    break\n",
    "            except:\n",
    "                break\n",
    "\n",
    "            # Add the technology to the key name\n",
    "            this_track[0] = this_track[0] + \", \" + this_track[1]\n",
    "            \n",
    "            # Check if track key already exists in the dictionary\n",
    "            if this_track[0] in complete_tracks:\n",
    "                continue        # Skip iteration if key already exists, allows for resuming a broken loop without re-executing on beginning of file\n",
    "                # # If it already exists, create a new key by adding the last 3 words of the url to the key\n",
    "                # this_track[0] = this_track[0] + \" - \" + \" \".join(this_track[2].split(\"-\")[-2:])\n",
    "\n",
    "            # Get the courses for this track\n",
    "            this_courses = get_courses(this_track[2])\n",
    "            # Convert this_courses to a dictionary with the first element as the key and the second and third elements as the value\n",
    "            this_courses = {x[0]: (x[1], x[2]) for x in this_courses}\n",
    "            \n",
    "            # Add the track to the dictionary\n",
    "            complete_tracks[this_track[0]] = this_courses\n",
    "            # Print progress\n",
    "            print(this_track[0])\n",
    "    read_tracks = 0\n",
    "\n",
    "    # Delete tracks with no courses read\n",
    "    for track in complete_tracks.copy():\n",
    "        if len(complete_tracks[track]) == 0:\n",
    "            print('>>', track)\n",
    "            del complete_tracks[track]\n",
    "            read_tracks += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to a pickle file\n",
    "with open('complete_tracks.pickle', 'wb') as f:\n",
    "    pickle.dump(complete_tracks, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scraping success! On to processing the data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import complete_tracks from the pickle file\n",
    "with open('complete_tracks.pickle', 'rb') as f:\n",
    "    complete_tracks = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'd like to have some strategy for completing these courses. I want to get through as many tracks as I can and as quickly as possible. My focus is currently on Python, so I want to get those done first, as well as some Power BI courses. However, it is good to know all of my options, so I shouldn't exclude the other technologies. It may intially be useful to just get an idea of how the courses are split between the technologies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a list of technologies from the dictionary, where the term after the comma is the technology\n",
    "technologies = {x.split(\", \")[1] for x in complete_tracks.keys()}\n",
    "technologies\n",
    "\n",
    "# Show all courses for a given technology\n",
    "def show_courses(tech):\n",
    "    \"\"\"\n",
    "    Prints all courses names for a given technology\n",
    "    \n",
    "        Parameters\n",
    "        ----------\n",
    "        tech : str\n",
    "            Technology to search for\n",
    "    \"\"\"\n",
    "    print(\"\\nCourses for \" + tech + \":\")\n",
    "    # Loop through the dictionary\n",
    "    for key in complete_tracks:\n",
    "        # Check if the technology is in the key\n",
    "        if tech in key:\n",
    "            # Print the key and the courses\n",
    "            print('\\t', key)\n",
    "            for course in complete_tracks[key]:\n",
    "                print('\\t'*2, course)\n",
    "            print()\n",
    "\n",
    "# Show all courses by track and technology (output hidden here due to length)\n",
    "for tech in technologies:\n",
    "    show_courses(tech)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Useful dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is clear that this data is in a clear hierarchy, where tracks are arranged by technology, courses are sorted into tracks (sometimes in multiple tracks), and the courses themselves have different durations and descriptions.\n",
    "It will be useful to split this data into several simple objects representing key-value relationships for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'R Programming, R': 'R',\n",
       " 'Importing & Cleaning Data, R': 'R',\n",
       " 'Data Visualization, R': 'R',\n",
       " 'Data Manipulation, R': 'R'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# # Get a dictionary of tracks and technologies, where the term after the comma is the technology\n",
    "tracks_tech = dict()\n",
    "for key in complete_tracks:\n",
    "    # Get the technology\n",
    "    tech = key.split(\",\")[1].strip()\n",
    "    # Add the technology to the dictionary\n",
    "    tracks_tech[key] = tech\n",
    "# Show the first few tracks and their technologies\n",
    "{k: tracks_tech[k] for k in list(tracks_tech)[:4]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Intermediate Regression in R': 5,\n",
       " 'Data Manipulation with dplyr ': 4,\n",
       " 'Introduction to Statistics in R': 4,\n",
       " 'Data Manipulation with pandas': 4}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For each course, get the number of tracks it is in\n",
    "course_counts = dict()\n",
    "for key in complete_tracks:\n",
    "    for course in complete_tracks[key]:\n",
    "        if course in course_counts:\n",
    "            course_counts[course] += 1\n",
    "        else:\n",
    "            course_counts[course] = 1\n",
    "# Sort the courses by the number of tracks they are in, and convert to a dictionary with the course as the key and the number of tracks as the value\n",
    "course_counts = {x: course_counts[x] for x in sorted(course_counts, key = course_counts.get, reverse = True)}\n",
    "# Show the first few courses and their counts\n",
    "{k: course_counts[k] for k in list(course_counts)[:4]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Introduction to R': ['R Programming, R',\n",
       "  'Data Analyst, R',\n",
       "  'Data Scientist, R'],\n",
       " 'Intermediate R': ['R Programming, R', 'Data Scientist, R'],\n",
       " 'Writing Efficient R Code': ['R Programming, R',\n",
       "  'Big Data, R',\n",
       "  'R Programmer, R'],\n",
       " 'Introduction to Writing Functions in R': ['R Programming, R',\n",
       "  'Data Scientist, R',\n",
       "  'R Programmer, R']}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dictionary of courses and the tracks they are in\n",
    "course_tracks = dict()\n",
    "for key in complete_tracks:\n",
    "    for course in complete_tracks[key]:\n",
    "        if course in course_tracks:\n",
    "            course_tracks[course].append(key)\n",
    "        else:\n",
    "            course_tracks[course] = [key]\n",
    "# Show the first few courses and their tracks\n",
    "{k: course_tracks[k] for k in list(course_tracks)[:4]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Data Literacy Fundamentals, Theory': 10,\n",
       " 'Data Visualization, R': 12,\n",
       " 'Unsupervised Machine Learning, R': 12,\n",
       " 'Image Processing, Python': 12}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert durations of each course to an integer\n",
    "for key in complete_tracks:\n",
    "    for course in complete_tracks[key]:\n",
    "        course_info = complete_tracks[key][course]\n",
    "        # Convert tuple to list\n",
    "        course_info = list(course_info)\n",
    "        # If the duration is a string, convert to an integer\n",
    "        if isinstance(course_info[1], str):\n",
    "            course_info[1] = int(course_info[1].split(\" \")[0])\n",
    "        # Update the dictionary\n",
    "        complete_tracks[key][course] = course_info\n",
    "complete_tracks\n",
    "\n",
    "# Find the total duration of each track\n",
    "track_durations = dict()\n",
    "for key in complete_tracks:\n",
    "    # Get the total duration of the track\n",
    "    total_duration = sum([x[1] for x in complete_tracks[key].values()])\n",
    "    # Add the total duration to the dictionary\n",
    "    track_durations[key] = total_duration\n",
    "# Sort the tracks by the total duration descending\n",
    "track_durations = {x: track_durations[x] for x in sorted(track_durations, key = track_durations.get, reverse = False)}\n",
    "# Show the first few tracks and their total duration\n",
    "{k: track_durations[k] for k in list(track_durations)[:4]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get current progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is key to have a list of completed courses, to allow the recommendations to adapt to my progress. I could create this manually, but I would prefer to scrape it to limit manual updating. Completed courses are indicated on the 'Courses' page when logged in to DataCamp. However, to avoid logging in to DataCamp through the `chromedriver`, which can get fiddly, I can instead access the Github repo where I store all of my DataCamp certificates by course name. This is doubly beneficial as it contains certificates from multiple DataCamp accounts I have held over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Completed courses: ['ARIMAModelsinR', 'CaseStudiesManipulatingTimeSeriesDatainR', 'Data Manipulation with pandas', 'DataManipulationwithdplyr']\n",
      "\n",
      "Total number of completed courses: 24\n"
     ]
    }
   ],
   "source": [
    "# Get the completed courses as shown on Github\n",
    "dc_done_url = \"https://github.com/faisaljina/datacamp-profile/tree/main/Certificates\"\n",
    "dc_done_soup = get_html(dc_done_url)\n",
    "dc_done_list = [x.text for x in dc_done_soup.find_all('span', class_ = 'css-truncate css-truncate-target d-block width-fit')]\n",
    "# Remove the file extension from the file names\n",
    "dc_done_list = [x.split(\".\")[0] for x in dc_done_list]\n",
    "# Exclude the files that end in 'Track'\n",
    "dc_done_list = [x for x in dc_done_list if not x.endswith(\"Track\")]\n",
    "\n",
    "# Show the first few completed courses and the total number\n",
    "print(\"Completed courses:\", dc_done_list[:4])\n",
    "print(\"Total number of completed courses:\", len(dc_done_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Introduction to R for Finance', 'Introduction to Statistics in R', 'Financial Analytics in Spreadsheets', 'Preprocessing for Machine Learning in Python'}\n",
      "Total number of courses: 289\n"
     ]
    }
   ],
   "source": [
    "# Get list of all possible DataCamp courses\n",
    "all_courses = []\n",
    "for key in complete_tracks:\n",
    "    for course in complete_tracks[key]:\n",
    "        all_courses.append(course)\n",
    "all_courses = set(all_courses)\n",
    "# Show the first few courses and the total number\n",
    "print({k for k in list(all_courses)[:4]})\n",
    "print(\"Total number of courses:\", len(all_courses))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List is complete\n",
      "Completed: 24\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['ARIMA Models in R',\n",
       " 'Case Studies: Manipulating Time Series Data in R',\n",
       " 'Data Manipulation with dplyr ',\n",
       " 'Data Manipulation with pandas']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For each course, check if it is in the scraped list of completed courses\n",
    "completed_courses = []\n",
    "for course in all_courses:\n",
    "    # Remove colons from the course name\n",
    "    course_cleaned = course.replace(\":\", \"\")\n",
    "    # Split course name into list of words\n",
    "    course_words = course_cleaned.split(\" \")\n",
    "    # Check if all words in the course name are in any of the dc_done_list\n",
    "    for comp_course in dc_done_list:\n",
    "        # Remove all spaces from comp_course if present\n",
    "        comp_course = comp_course.replace(\" \", \"\")\n",
    "        # Remove the file extension from comp_course if present\n",
    "        comp_course = comp_course.split(\".\")[0]\n",
    "        # Check if all words in the course name are in the comp_course\n",
    "        if all(word in comp_course for word in course_words):\n",
    "            # Check if removing the matching words leaves no characters - this filters out similar courses with matching stems\n",
    "            if len(comp_course.replace(\"\".join(course_words), \"\")) == 0:\n",
    "                completed_courses.append(course)\n",
    "                break\n",
    "# Sort alphabetically\n",
    "completed_courses = sorted(completed_courses)\n",
    "# Check the list is complete\n",
    "if len(completed_courses) == len(dc_done_list):\n",
    "    print(\"List is complete\")\n",
    "    print(f'Completed: {len(completed_courses)}')\n",
    "else:\n",
    "    print(\"List is not complete\")\n",
    "    # Show scraped and completed courses lengths\n",
    "    print(f'Scraped: {len(dc_done_list)}')\n",
    "    print(f'Matched: {len(completed_courses)}')\n",
    "\n",
    "# Show the first few completed courses as matched with the scraped list\n",
    "completed_courses[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Data Visualization, R': 8,\n",
       " 'Data Literacy Fundamentals, Theory': 10,\n",
       " 'R Programming, R': 12,\n",
       " 'Data Manipulation, Python': 12}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get dictionary of all tracks of courses minus the completed courses\n",
    "tracks_minus_completed = dict()\n",
    "for key in complete_tracks:\n",
    "    # Get the list of courses for this track\n",
    "    courses = complete_tracks[key]\n",
    "    # Remove the completed courses from the list\n",
    "    courses = {x: courses[x] for x in courses if x not in completed_courses}\n",
    "    # Add the courses to the dictionary\n",
    "    tracks_minus_completed[key] = courses\n",
    "tracks_minus_completed['Python Fundamentals, Python']\n",
    "\n",
    "# Get dictionary of track remaining durations\n",
    "track_remaining_durations = dict()\n",
    "for key in tracks_minus_completed:\n",
    "    # Get the total duration of the track\n",
    "    total_duration = sum([x[1] for x in tracks_minus_completed[key].values()])\n",
    "    # Add the total duration to the dictionary\n",
    "    track_remaining_durations[key] = total_duration\n",
    "# Sort the tracks by the total duration descending\n",
    "track_remaining_durations = {x: track_remaining_durations[x] for x in sorted(track_remaining_durations, key = track_remaining_durations.get, reverse = False)}\n",
    "# If the total duration is 0, remove the track from durations and tracks_minus_completed\n",
    "for key in track_remaining_durations.copy():\n",
    "    if track_remaining_durations[key] == 0:\n",
    "        del track_remaining_durations[key]\n",
    "        del tracks_minus_completed[key]\n",
    "# Show the first few tracks and their total duration\n",
    "{k: track_remaining_durations[k] for k in list(track_remaining_durations)[:4]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Recommender is born"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have all the data in all the forms we need. Now we just need to write some appropriate functions to get us the recommender we have always wanted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Joining Data with pandas', 4), ('Analyzing Police Activity with pandas', 4), ('Introduction to Databases in Python', 4), ('Image Processing in Python', 4)]\n",
      "Total number of courses: 17\n"
     ]
    }
   ],
   "source": [
    "# Function taking the technology name, and returning the track names and remaining durations\n",
    "def get_tracks_by_tech(tech, k_tracks=3):\n",
    "    # Get the list of tracks for this technology, where the tracks are the keys in the dictionary\n",
    "    tracks = [x for x in tracks_minus_completed if tech in x]\n",
    "    # Get the remaining durations for the tracks\n",
    "    durations = [track_remaining_durations[x] for x in tracks]\n",
    "    # Zip the tracks and durations together\n",
    "    tracks_durations = list(zip(tracks, durations))\n",
    "    # Sort the tracks by the remaining duration\n",
    "    tracks_durations = sorted(tracks_durations, key = lambda x: x[1])\n",
    "    # Return the k-smallest remaining durations and the tracks\n",
    "    return tracks_durations[:k_tracks]\n",
    "\n",
    "# Function taking the technology, finding the tracks and remaining durations and returning the courses within those tracks\n",
    "def get_courses_by_tech(tech, k_tracks=3)->list:\n",
    "    \"\"\"\n",
    "    Get the courses for a technology\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    tech : str\n",
    "        The technology name\n",
    "        k_tracks : int, optional\n",
    "            The number of tracks to return courses from. The default is 3.\n",
    "            \n",
    "            Returns\n",
    "            -------\n",
    "            list\n",
    "                The list of courses as a list of tuples of the form (course, duration)\n",
    "                \n",
    "                Examples\n",
    "                --------\n",
    "                get_courses_by_tech(\"Python\")\n",
    "                get_courses_by_tech(\"Python\", k_tracks=2)\n",
    "    \"\"\"\n",
    "    # Get the tracks and remaining durations for this technology\n",
    "    tracks_durations = get_tracks_by_tech(tech, k_tracks)\n",
    "    # Get the courses and durations for the selected tracks\n",
    "    courses_durations = []\n",
    "    for track, duration in tracks_durations:\n",
    "        # Get the courses for this track\n",
    "        course = tracks_minus_completed[track]\n",
    "        # Get the remaining durations for the course\n",
    "        duration = [course[x][1] for x in course]\n",
    "        # Zip the course and duration together\n",
    "        course_duration = list(zip(course, duration))\n",
    "        # Add the course and duration to the list\n",
    "        courses_durations.append(course_duration)\n",
    "    # Flatten the list of lists of courses and durations\n",
    "    courses_durations = [item for sublist in courses_durations for item in sublist]\n",
    "    return courses_durations\n",
    "\n",
    "# Show the first few courses for Python and the total number of courses found\n",
    "print(get_courses_by_tech(\"Python\", k_tracks=5)[:4])\n",
    "print(\"Total number of courses:\", len(get_courses_by_tech(\"Python\", k_tracks=5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have a function that find the course names and durations for the k-shortest remaining courses for any given technology.\n",
    "We want to do something similar for a given course name, so that we can find the single shortest track doing that course would contribute towards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Intermediate Spreadsheets, Spreadsheet', 12)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Function taking a course name and finding which tracks it is in, and the remaining duration of the shortest remaining track\n",
    "def get_track_by_course(course) -> Tuple[str, int]:\n",
    "    \"\"\"\n",
    "    Get the track name and remaining duration of the shortest remaining track for a course\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    course : str\n",
    "        The course name\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    track : str\n",
    "        The track name\n",
    "    duration : int\n",
    "        The remaining duration of the shortest remaining track\n",
    "    \"\"\"\n",
    "    # Get the list of tracks for this course\n",
    "    tracks = course_tracks[course]\n",
    "    # Get the remaining durations for the tracks\n",
    "    durations = [track_remaining_durations[x] for x in tracks]\n",
    "    # Zip the tracks and durations together\n",
    "    tracks_durations = list(zip(tracks, durations))\n",
    "    # If there is only one track, return the track and remaining duration\n",
    "    if len(tracks_durations) == 1:\n",
    "        return tracks_durations[0]\n",
    "    # Sort the tracks by the remaining duration\n",
    "    tracks_durations = sorted(tracks_durations, key = lambda x: x[1])\n",
    "    # Check if the remaining duration of the two shortest remaining tracks are the same\n",
    "    if tracks_durations[0][1] == tracks_durations[1][1]:\n",
    "        # If so, return both tracks\n",
    "        return tracks_durations[0], tracks_durations[1]\n",
    "    else:\n",
    "        # If not, return the shortest remaining track\n",
    "        return tracks_durations[0]\n",
    "\n",
    "# Check a course to find the track and remaining duration\n",
    "get_track_by_course('Introduction to Statistics in Spreadsheets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last couple of things now, just want to double-check I can find the number of tracks a course appears in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of tracks: 3\n"
     ]
    }
   ],
   "source": [
    "# Check no. of tracks a course is in\n",
    "test_course = 'Joining Data with pandas'\n",
    "print(\"Number of tracks:\", course_counts[test_course])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, and just as some added sugar, I'd like to have a dictionary of courses and their descriptions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Introduction to R': 'Master the basics of data analysis by manipulating common data structures such as vectors, matrices, and data frames.',\n",
       " 'Intermediate R': 'Continue your journey to becoming an R ninja by learning about conditional statements, loops, and vector functions.',\n",
       " 'Writing Efficient R Code': 'Learn to write faster R code, discover benchmarking and profiling, and unlock the secrets of parallel programming.',\n",
       " 'Introduction to Writing Functions in R': 'Use cluster analysis to glean insights into cryptocurrency gambling behavior.'}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dictionary of all courses and their descriptions\n",
    "course_descriptions = dict()\n",
    "for track in complete_tracks:\n",
    "    for course, description in complete_tracks[track].items():\n",
    "        course_descriptions[course] = description[0]\n",
    "\n",
    "# Show the first few courses and their descriptions\n",
    "{k: course_descriptions[k] for k in list(course_descriptions.keys())[:4]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Put it all together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dataframe is a suitable output for my recommendations as they display nicely in VSCode. Let's put all the above functions together and see what we get."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Choose a technology from:\n",
      "['Theory', 'Spreadsheet', 'Power BI', 'SQL', 'R', 'Tableau', 'Python']\n",
      "(Defaulting to 'Python')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Joining Data with pandas</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Data Manipulation, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to combine data from multiple tables by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Analyzing Police Activity with pandas</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Data Manipulation, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to draw conclusions from limited data us...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Image Processing in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Image Processing, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn powerful techniques for image analysis i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Image Processing with Keras in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Image Processing, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to tune hyperparameters in Python.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Introduction to Databases in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Manipulation, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>In this course, you'll learn the basics of rel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Recurrent Neural Networks for Language Modelin...</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Learning for NLP, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Use RNNs to classify text sentiment, generate ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Machine Translation in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Learning for NLP, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Are you curious about the inner workings of th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Natural Language Generation in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Learning for NLP, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Imitate Shakespear, translate language and aut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Biomedical Image Analysis in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Image Processing, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn the fundamentals of exploring, manipulat...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Course  Course Length  \\\n",
       "0                           Joining Data with pandas              4   \n",
       "1              Analyzing Police Activity with pandas              4   \n",
       "2                         Image Processing in Python              4   \n",
       "3              Image Processing with Keras in Python              4   \n",
       "4                Introduction to Databases in Python              4   \n",
       "5  Recurrent Neural Networks for Language Modelin...              4   \n",
       "6                      Machine Translation in Python              4   \n",
       "7              Natural Language Generation in Python              4   \n",
       "8                Biomedical Image Analysis in Python              4   \n",
       "\n",
       "   Track Duplication                 Shortest Track  Track Time Remaining  \\\n",
       "0                  3      Data Manipulation, Python                    12   \n",
       "1                  2      Data Manipulation, Python                    12   \n",
       "2                  2       Image Processing, Python                    12   \n",
       "3                  2       Image Processing, Python                    12   \n",
       "4                  1      Data Manipulation, Python                    12   \n",
       "5                  1  Deep Learning for NLP, Python                    12   \n",
       "6                  1  Deep Learning for NLP, Python                    12   \n",
       "7                  1  Deep Learning for NLP, Python                    12   \n",
       "8                  1       Image Processing, Python                    12   \n",
       "\n",
       "                                  Course Description  \n",
       "0  Learn to combine data from multiple tables by ...  \n",
       "1  Learn to draw conclusions from limited data us...  \n",
       "2  Learn powerful techniques for image analysis i...  \n",
       "3           Learn to tune hyperparameters in Python.  \n",
       "4  In this course, you'll learn the basics of rel...  \n",
       "5  Use RNNs to classify text sentiment, generate ...  \n",
       "6  Are you curious about the inner workings of th...  \n",
       "7  Imitate Shakespear, translate language and aut...  \n",
       "8  Learn the fundamentals of exploring, manipulat...  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Function taking a technology and performing the above functions to get the courses, tracks and remaining durations\n",
    "def get_recommendations(tech = None, k_tracks = 5, row_limit = 10)-> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Take a technology and get recommended courses and info\n",
    "\n",
    "    Args:\n",
    "        tech (str): Technology name\n",
    "        k_tracks (int): Number of tracks to return\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Dataframe of recommended courses and associated information\n",
    "    \"\"\"\n",
    "\n",
    "    if not tech:\n",
    "        print(f\"Choose a technology from:\\n{[x for x in technologies]}\\n(Defaulting to 'Python')\")\n",
    "        tech = \"Python\"\n",
    "    # Get the courses for this technology\n",
    "    courses = get_courses_by_tech(tech, k_tracks)\n",
    "    # Get the tracks and track durations for each course\n",
    "    tracks_durations = []\n",
    "    for course in courses:\n",
    "        track = get_track_by_course(course[0])\n",
    "        tracks_durations.append(track)\n",
    "    # Find the number of tracks for each course\n",
    "    track_course_counts = []\n",
    "    for course in courses:\n",
    "        if course[0] in course_counts:\n",
    "            track_course_counts.append(course_counts[course[0]])\n",
    "        else:\n",
    "            print(\"No count of course in tracks: \", course[0])\n",
    "            track_course_counts.append(0)\n",
    "    # Zip the courses, track durations data and track course counts together\n",
    "    courses_all_info = list(zip(courses, tracks_durations, track_course_counts))\n",
    "    # Flatten the course info\n",
    "    courses_all_info = [item for sublist in courses_all_info for item in sublist]\n",
    "    # Reshape the course info\n",
    "    course_all_info_reshaped = []\n",
    "    for i in range(len(courses_all_info)):\n",
    "        if i % 3 == 0:\n",
    "            course_all_info_reshaped.append(list(courses_all_info[i]) + list(courses_all_info[i+1]) + [courses_all_info[i+2]])\n",
    "    # Put the course info into a dataframe\n",
    "    course_all_info_df = pd.DataFrame(course_all_info_reshaped, columns = ['Course', 'Course Length', 'Shortest Track', 'Track Time Remaining', 'Track Duplication'])\n",
    "    # Remove any duplicated lines\n",
    "    course_all_info_df = course_all_info_df.drop_duplicates(subset = ['Course'])\n",
    "    # Sort the dataframe by the track duration remaining ascending, then by the course Length ascending, then by the duplication in tracks descending, then by shortest track ascending\n",
    "    course_all_info_df = course_all_info_df.sort_values(by = ['Track Time Remaining', 'Course Length', 'Track Duplication', 'Shortest Track'], ascending = [True, True, False, True])\n",
    "    # Rearrange the columns to course, course length, duplication in tracks, track, track duration remaining\n",
    "    course_all_info_df = course_all_info_df[['Course', 'Course Length', 'Track Duplication', 'Shortest Track', 'Track Time Remaining']]\n",
    "    # Add a column for the course description\n",
    "    course_all_info_df['Course Description'] = course_all_info_df['Course'].map(course_descriptions)\n",
    "    # Reset the index\n",
    "    course_all_info_df = course_all_info_df.reset_index(drop = True)\n",
    "    # Limit the number of rows to the row_limit\n",
    "    course_all_info_df = course_all_info_df.head(row_limit)\n",
    "    # Return the dataframe\n",
    "    return course_all_info_df\n",
    "    \n",
    "\n",
    "get_recommendations(k_tracks=3, row_limit=10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has worked excellently. The key to this was the sorting of the data to suit my preference, and I think this ordering expresses that well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Recommender System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run this to get your recommendations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Increase limit of string output in the dataframe to get the full course description\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Joining Data with pandas</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Data Manipulation, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to combine data from multiple tables by joining data together using pandas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Analyzing Police Activity with pandas</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Data Manipulation, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to draw conclusions from limited data using Python and statistics. This course covers everything from random sampling to stratified and cluster sampling.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Image Processing in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Image Processing, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn powerful techniques for image analysis in Python using deep learning and convolutional neural networks in Keras.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Image Processing with Keras in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Image Processing, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to tune hyperparameters in Python.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Introduction to Databases in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Manipulation, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>In this course, you'll learn the basics of relational databases and how to interact with them.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Recurrent Neural Networks for Language Modeling in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Learning for NLP, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Use RNNs to classify text sentiment, generate sentences, and translate text between languages.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Machine Translation in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Learning for NLP, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Are you curious about the inner workings of the models that are behind products like Google Translate?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Natural Language Generation in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Learning for NLP, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Imitate Shakespear, translate language and autocomplete sentences using Deep Learning in Python.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Biomedical Image Analysis in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Image Processing, Python</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn the fundamentals of exploring, manipulating, and measuring biomedical image data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Intermediate Importing Data in Python</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Importing &amp; Cleaning Data, Python</td>\n",
       "      <td>13</td>\n",
       "      <td>Learn to diagnose and treat dirty data and develop the skills needed to transform your raw data into accurate insights!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Introduction to Importing Data in Python</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>Importing &amp; Cleaning Data, Python</td>\n",
       "      <td>13</td>\n",
       "      <td>Improve your Python data importing skills and learn to work with web and API data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Cleaning Data in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Importing &amp; Cleaning Data, Python</td>\n",
       "      <td>13</td>\n",
       "      <td>Learn how to work with dates and times in Python.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                       Course  Course Length  \\\n",
       "0                                    Joining Data with pandas              4   \n",
       "1                       Analyzing Police Activity with pandas              4   \n",
       "2                                  Image Processing in Python              4   \n",
       "3                       Image Processing with Keras in Python              4   \n",
       "4                         Introduction to Databases in Python              4   \n",
       "5   Recurrent Neural Networks for Language Modeling in Python              4   \n",
       "6                               Machine Translation in Python              4   \n",
       "7                       Natural Language Generation in Python              4   \n",
       "8                         Biomedical Image Analysis in Python              4   \n",
       "9                       Intermediate Importing Data in Python              2   \n",
       "10                   Introduction to Importing Data in Python              3   \n",
       "11                                    Cleaning Data in Python              4   \n",
       "\n",
       "    Track Duplication                     Shortest Track  \\\n",
       "0                   3          Data Manipulation, Python   \n",
       "1                   2          Data Manipulation, Python   \n",
       "2                   2           Image Processing, Python   \n",
       "3                   2           Image Processing, Python   \n",
       "4                   1          Data Manipulation, Python   \n",
       "5                   1      Deep Learning for NLP, Python   \n",
       "6                   1      Deep Learning for NLP, Python   \n",
       "7                   1      Deep Learning for NLP, Python   \n",
       "8                   1           Image Processing, Python   \n",
       "9                   2  Importing & Cleaning Data, Python   \n",
       "10                  2  Importing & Cleaning Data, Python   \n",
       "11                  2  Importing & Cleaning Data, Python   \n",
       "\n",
       "    Track Time Remaining  \\\n",
       "0                     12   \n",
       "1                     12   \n",
       "2                     12   \n",
       "3                     12   \n",
       "4                     12   \n",
       "5                     12   \n",
       "6                     12   \n",
       "7                     12   \n",
       "8                     12   \n",
       "9                     13   \n",
       "10                    13   \n",
       "11                    13   \n",
       "\n",
       "                                                                                                                                                 Course Description  \n",
       "0                                                                                 Learn to combine data from multiple tables by joining data together using pandas.  \n",
       "1   Learn to draw conclusions from limited data using Python and statistics. This course covers everything from random sampling to stratified and cluster sampling.  \n",
       "2                                            Learn powerful techniques for image analysis in Python using deep learning and convolutional neural networks in Keras.  \n",
       "3                                                                                                                          Learn to tune hyperparameters in Python.  \n",
       "4                                                                    In this course, you'll learn the basics of relational databases and how to interact with them.  \n",
       "5                                                                    Use RNNs to classify text sentiment, generate sentences, and translate text between languages.  \n",
       "6                                                            Are you curious about the inner workings of the models that are behind products like Google Translate?  \n",
       "7                                                                  Imitate Shakespear, translate language and autocomplete sentences using Deep Learning in Python.  \n",
       "8                                                                           Learn the fundamentals of exploring, manipulating, and measuring biomedical image data.  \n",
       "9                                           Learn to diagnose and treat dirty data and develop the skills needed to transform your raw data into accurate insights!  \n",
       "10                                                                               Improve your Python data importing skills and learn to work with web and API data.  \n",
       "11                                                                                                                Learn how to work with dates and times in Python.  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Power BI:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Introduction to DAX in Power BI</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Power BI Fundamentals, Power BI</td>\n",
       "      <td>17</td>\n",
       "      <td>Enhance your Power BI knowledge, by learning the fundamentals of Data Analysis Expressions (DAX) such as calculated columns, tables, and measures.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Introduction to Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>Power BI Fundamentals, Power BI</td>\n",
       "      <td>17</td>\n",
       "      <td>Gain a 360° overview of how to explore and use Power BI to build impactful reports.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Data Visualization in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>Power BI Fundamentals, Power BI</td>\n",
       "      <td>17</td>\n",
       "      <td>Power BI is a powerful data visualization tool that can be used in reports and dashboards.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Case Study: Analyzing Job Market Data in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>Power BI Fundamentals, Power BI</td>\n",
       "      <td>17</td>\n",
       "      <td>Help a fictional company in this interactive Power BI case study. You’ll use Power Query, DAX, and dashboards to identify the most in-demand data jobs!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Data Preparation in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>Power BI Fundamentals, Power BI</td>\n",
       "      <td>17</td>\n",
       "      <td>In this interactive Power BI course, you’ll learn how to use Power Query Editor to transform and shape your data to be ready for analysis.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Data Modeling in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>Power BI Fundamentals, Power BI</td>\n",
       "      <td>17</td>\n",
       "      <td>Learn the key concepts of data modeling on Power BI.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Data Connections in Power BI</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Analyst, Power BI</td>\n",
       "      <td>51</td>\n",
       "      <td>Discover the different ways you can enhance your Power BI data importing skills.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Deploying and Maintaining Assets in Power BI</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Analyst, Power BI</td>\n",
       "      <td>51</td>\n",
       "      <td>Learn how to deploy and maintain assets in Power BI. You’ll get to grips with the Power BI Service interface and key elements in it like workspaces.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Case Study: Analyzing Customer Churn in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Analyst, Power BI</td>\n",
       "      <td>51</td>\n",
       "      <td>You will investigate a dataset from a fictitious company called Databel in Power BI, and need to figure out why customers are churning.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Data Transformation in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Analyst, Power BI</td>\n",
       "      <td>51</td>\n",
       "      <td>You’ll learn how to (un)pivot, transpose, append and join tables. Gain power with custom columns, M language, and the Advanced Editor.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Intermediate Data Modeling in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Analyst, Power BI</td>\n",
       "      <td>51</td>\n",
       "      <td>Master data modeling in Power BI.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>DAX Functions in Power BI</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Analyst, Power BI</td>\n",
       "      <td>51</td>\n",
       "      <td>Data Analysis Expressions (DAX) allow you to take your Power BI skills to the next level by writing custom functions.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Course  Course Length  \\\n",
       "0                     Introduction to DAX in Power BI              2   \n",
       "1                            Introduction to Power BI              3   \n",
       "2                      Data Visualization in Power BI              3   \n",
       "3   Case Study: Analyzing Job Market Data in Power BI              3   \n",
       "4                        Data Preparation in Power BI              3   \n",
       "5                           Data Modeling in Power BI              3   \n",
       "6                        Data Connections in Power BI              2   \n",
       "7        Deploying and Maintaining Assets in Power BI              2   \n",
       "8    Case Study: Analyzing Customer Churn in Power BI              3   \n",
       "9                     Data Transformation in Power BI              3   \n",
       "10             Intermediate Data Modeling in Power BI              3   \n",
       "11                          DAX Functions in Power BI              3   \n",
       "\n",
       "    Track Duplication                   Shortest Track  Track Time Remaining  \\\n",
       "0                   2  Power BI Fundamentals, Power BI                    17   \n",
       "1                   2  Power BI Fundamentals, Power BI                    17   \n",
       "2                   2  Power BI Fundamentals, Power BI                    17   \n",
       "3                   2  Power BI Fundamentals, Power BI                    17   \n",
       "4                   2  Power BI Fundamentals, Power BI                    17   \n",
       "5                   2  Power BI Fundamentals, Power BI                    17   \n",
       "6                   1           Data Analyst, Power BI                    51   \n",
       "7                   1           Data Analyst, Power BI                    51   \n",
       "8                   1           Data Analyst, Power BI                    51   \n",
       "9                   1           Data Analyst, Power BI                    51   \n",
       "10                  1           Data Analyst, Power BI                    51   \n",
       "11                  1           Data Analyst, Power BI                    51   \n",
       "\n",
       "                                                                                                                                         Course Description  \n",
       "0        Enhance your Power BI knowledge, by learning the fundamentals of Data Analysis Expressions (DAX) such as calculated columns, tables, and measures.  \n",
       "1                                                                       Gain a 360° overview of how to explore and use Power BI to build impactful reports.  \n",
       "2                                                                Power BI is a powerful data visualization tool that can be used in reports and dashboards.  \n",
       "3   Help a fictional company in this interactive Power BI case study. You’ll use Power Query, DAX, and dashboards to identify the most in-demand data jobs!  \n",
       "4                In this interactive Power BI course, you’ll learn how to use Power Query Editor to transform and shape your data to be ready for analysis.  \n",
       "5                                                                                                      Learn the key concepts of data modeling on Power BI.  \n",
       "6                                                                          Discover the different ways you can enhance your Power BI data importing skills.  \n",
       "7      Learn how to deploy and maintain assets in Power BI. You’ll get to grips with the Power BI Service interface and key elements in it like workspaces.  \n",
       "8                   You will investigate a dataset from a fictitious company called Databel in Power BI, and need to figure out why customers are churning.  \n",
       "9                    You’ll learn how to (un)pivot, transpose, append and join tables. Gain power with custom columns, M language, and the Advanced Editor.  \n",
       "10                                                                                                                        Master data modeling in Power BI.  \n",
       "11                                    Data Analysis Expressions (DAX) allow you to take your Power BI skills to the next level by writing custom functions.  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "SQL:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Intermediate SQL</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL Fundamentals, SQL</td>\n",
       "      <td>12</td>\n",
       "      <td>Master the complex SQL queries necessary to answer a wide variety of data science questions and prepare robust data sets for analysis in PostgreSQL.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>PostgreSQL Summary Stats and Window Functions</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL Fundamentals, SQL</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to create queries for analytics and data engineering with window functions, the SQL secret weapon!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Functions for Manipulating Data in PostgreSQL</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL Fundamentals, SQL</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn the most important PostgreSQL functions for manipulating, processing, and transforming data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Intermediate SQL Server</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL Server Fundamentals, SQL</td>\n",
       "      <td>13</td>\n",
       "      <td>In this course, you will use T-SQL, the flavor of SQL used in Microsoft's SQL Server for data analysis.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Functions for Manipulating Data in SQL Server</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL Server Fundamentals, SQL</td>\n",
       "      <td>13</td>\n",
       "      <td>Learn the most important functions for manipulating, processing, and transforming data in SQL Server.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Time Series Analysis in SQL Server</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL Server Fundamentals, SQL</td>\n",
       "      <td>13</td>\n",
       "      <td>Explore ways to work with date and time data in SQL Server for time series analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Introduction to Relational Databases in SQL</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>SQL for Database Administrators, SQL</td>\n",
       "      <td>16</td>\n",
       "      <td>Learn how to create one of the most efficient ways of storing data - relational databases!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Database Design</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>SQL for Database Administrators, SQL</td>\n",
       "      <td>16</td>\n",
       "      <td>Learn to design databases in SQL.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Creating PostgreSQL Databases</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>SQL for Database Administrators, SQL</td>\n",
       "      <td>16</td>\n",
       "      <td>This course teaches you the skills and knowledge necessary to create and manage your own PostgreSQL databases.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Improving Query Performance in PostgreSQL</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>SQL for Database Administrators, SQL</td>\n",
       "      <td>16</td>\n",
       "      <td>Learn how to structure your PostgreSQL queries to run in a fraction of the time.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Exploratory Data Analysis in SQL</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL for Business Analysts, SQL</td>\n",
       "      <td>20</td>\n",
       "      <td>Learn how to explore what's available in a database: the tables, relationships between them, and data stored in them.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Data-Driven Decision Making in SQL</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>SQL for Business Analysts, SQL</td>\n",
       "      <td>20</td>\n",
       "      <td>Learn how to analyze a SQL table and report insights to management.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Course  Course Length  \\\n",
       "0                                Intermediate SQL              4   \n",
       "1   PostgreSQL Summary Stats and Window Functions              4   \n",
       "2   Functions for Manipulating Data in PostgreSQL              4   \n",
       "3                         Intermediate SQL Server              4   \n",
       "4   Functions for Manipulating Data in SQL Server              4   \n",
       "5              Time Series Analysis in SQL Server              5   \n",
       "6     Introduction to Relational Databases in SQL              4   \n",
       "7                                 Database Design              4   \n",
       "8                   Creating PostgreSQL Databases              4   \n",
       "9       Improving Query Performance in PostgreSQL              4   \n",
       "10               Exploratory Data Analysis in SQL              4   \n",
       "11             Data-Driven Decision Making in SQL              4   \n",
       "\n",
       "    Track Duplication                        Shortest Track  \\\n",
       "0                   2                 SQL Fundamentals, SQL   \n",
       "1                   2                 SQL Fundamentals, SQL   \n",
       "2                   2                 SQL Fundamentals, SQL   \n",
       "3                   2          SQL Server Fundamentals, SQL   \n",
       "4                   2          SQL Server Fundamentals, SQL   \n",
       "5                   2          SQL Server Fundamentals, SQL   \n",
       "6                   4  SQL for Database Administrators, SQL   \n",
       "7                   4  SQL for Database Administrators, SQL   \n",
       "8                   1  SQL for Database Administrators, SQL   \n",
       "9                   1  SQL for Database Administrators, SQL   \n",
       "10                  2        SQL for Business Analysts, SQL   \n",
       "11                  2        SQL for Business Analysts, SQL   \n",
       "\n",
       "    Track Time Remaining  \\\n",
       "0                     12   \n",
       "1                     12   \n",
       "2                     12   \n",
       "3                     13   \n",
       "4                     13   \n",
       "5                     13   \n",
       "6                     16   \n",
       "7                     16   \n",
       "8                     16   \n",
       "9                     16   \n",
       "10                    20   \n",
       "11                    20   \n",
       "\n",
       "                                                                                                                                      Course Description  \n",
       "0   Master the complex SQL queries necessary to answer a wide variety of data science questions and prepare robust data sets for analysis in PostgreSQL.  \n",
       "1                                           Learn how to create queries for analytics and data engineering with window functions, the SQL secret weapon!  \n",
       "2                                                     Learn the most important PostgreSQL functions for manipulating, processing, and transforming data.  \n",
       "3                                                In this course, you will use T-SQL, the flavor of SQL used in Microsoft's SQL Server for data analysis.  \n",
       "4                                                  Learn the most important functions for manipulating, processing, and transforming data in SQL Server.  \n",
       "5                                                                    Explore ways to work with date and time data in SQL Server for time series analysis  \n",
       "6                                                             Learn how to create one of the most efficient ways of storing data - relational databases!  \n",
       "7                                                                                                                      Learn to design databases in SQL.  \n",
       "8                                         This course teaches you the skills and knowledge necessary to create and manage your own PostgreSQL databases.  \n",
       "9                                                                       Learn how to structure your PostgreSQL queries to run in a fraction of the time.  \n",
       "10                                 Learn how to explore what's available in a database: the tables, relationships between them, and data stored in them.  \n",
       "11                                                                                   Learn how to analyze a SQL table and report insights to management.  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Spreadsheet:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Introduction to Statistics in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Intermediate Spreadsheets, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to leverage statistical techniques using spreadsheets to more effectively work with and extract insights from your data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Financial Analytics in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Finance Fundamentals, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to build a graphical dashboard with spreadsheets to track the performance of financial securities.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Financial Modeling in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Finance Fundamentals, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn basic business modeling including cash flows, investments, annuities, loan amortization, and more using Sheets.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Loan Amortization in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Finance Fundamentals, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to build an amortization dashboard in spreadsheets with financial and conditional formulas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Error and Uncertainty in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Intermediate Spreadsheets, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to distinguish real differences from random noise, and explore psychological crutches we use that interfere with our rational decision making.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Marketing Analytics in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Intermediate Spreadsheets, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to ensure clean data entry and build dynamic dashboards to display your marketing data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Data Analysis in Spreadsheets</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Spreadsheet Fundamentals, Spreadsheet</td>\n",
       "      <td>15</td>\n",
       "      <td>Learn how to analyze data with spreadsheets using functions such as SUM(), AVERAGE(), and VLOOKUP().</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Intermediate Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Spreadsheet Fundamentals, Spreadsheet</td>\n",
       "      <td>15</td>\n",
       "      <td>Expand your spreadsheets vocabulary by diving deeper into data types, including numeric data, logical data, and missing data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pivot Tables in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Spreadsheet Fundamentals, Spreadsheet</td>\n",
       "      <td>15</td>\n",
       "      <td>Explore the world of Pivot Tables within Google Sheets, and learn how to quickly organize thousands of data points with just a few clicks of the mouse.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Data Visualization in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Spreadsheet Fundamentals, Spreadsheet</td>\n",
       "      <td>15</td>\n",
       "      <td>Learn the fundamentals of data visualization using spreadsheets.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       Course  Course Length  \\\n",
       "0  Introduction to Statistics in Spreadsheets              4   \n",
       "1         Financial Analytics in Spreadsheets              4   \n",
       "2          Financial Modeling in Spreadsheets              4   \n",
       "3           Loan Amortization in Spreadsheets              4   \n",
       "4       Error and Uncertainty in Spreadsheets              4   \n",
       "5         Marketing Analytics in Spreadsheets              4   \n",
       "6               Data Analysis in Spreadsheets              3   \n",
       "7                   Intermediate Spreadsheets              4   \n",
       "8                Pivot Tables in Spreadsheets              4   \n",
       "9          Data Visualization in Spreadsheets              4   \n",
       "\n",
       "   Track Duplication                          Shortest Track  \\\n",
       "0                  2  Intermediate Spreadsheets, Spreadsheet   \n",
       "1                  1       Finance Fundamentals, Spreadsheet   \n",
       "2                  1       Finance Fundamentals, Spreadsheet   \n",
       "3                  1       Finance Fundamentals, Spreadsheet   \n",
       "4                  1  Intermediate Spreadsheets, Spreadsheet   \n",
       "5                  1  Intermediate Spreadsheets, Spreadsheet   \n",
       "6                  1   Spreadsheet Fundamentals, Spreadsheet   \n",
       "7                  1   Spreadsheet Fundamentals, Spreadsheet   \n",
       "8                  1   Spreadsheet Fundamentals, Spreadsheet   \n",
       "9                  1   Spreadsheet Fundamentals, Spreadsheet   \n",
       "\n",
       "   Track Time Remaining  \\\n",
       "0                    12   \n",
       "1                    12   \n",
       "2                    12   \n",
       "3                    12   \n",
       "4                    12   \n",
       "5                    12   \n",
       "6                    15   \n",
       "7                    15   \n",
       "8                    15   \n",
       "9                    15   \n",
       "\n",
       "                                                                                                                                        Course Description  \n",
       "0                       Learn how to leverage statistical techniques using spreadsheets to more effectively work with and extract insights from your data.  \n",
       "1                                             Learn how to build a graphical dashboard with spreadsheets to track the performance of financial securities.  \n",
       "2                                    Learn basic business modeling including cash flows, investments, annuities, loan amortization, and more using Sheets.  \n",
       "3                                                    Learn how to build an amortization dashboard in spreadsheets with financial and conditional formulas.  \n",
       "4     Learn to distinguish real differences from random noise, and explore psychological crutches we use that interfere with our rational decision making.  \n",
       "5                                                        Learn how to ensure clean data entry and build dynamic dashboards to display your marketing data.  \n",
       "6                                                     Learn how to analyze data with spreadsheets using functions such as SUM(), AVERAGE(), and VLOOKUP().  \n",
       "7                            Expand your spreadsheets vocabulary by diving deeper into data types, including numeric data, logical data, and missing data.  \n",
       "8  Explore the world of Pivot Tables within Google Sheets, and learn how to quickly organize thousands of data points with just a few clicks of the mouse.  \n",
       "9                                                                                         Learn the fundamentals of data visualization using spreadsheets.  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Theory:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Data Visualization for Everyone</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Data Literacy Fundamentals, Theory</td>\n",
       "      <td>10</td>\n",
       "      <td>An introduction to data visualization with no coding involved.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Data Engineering for Everyone</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Data Literacy Fundamentals, Theory</td>\n",
       "      <td>10</td>\n",
       "      <td>Discover how data engineers lay the groundwork that makes data science possible. No coding involved!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Data Science for Everyone</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Literacy Fundamentals, Theory</td>\n",
       "      <td>10</td>\n",
       "      <td>An introduction to data science with no coding involved.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Machine Learning for Everyone</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Literacy Fundamentals, Theory</td>\n",
       "      <td>10</td>\n",
       "      <td>An introduction to machine learning with no coding involved.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cloud Computing for Everyone</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Literacy Fundamentals, Theory</td>\n",
       "      <td>10</td>\n",
       "      <td>A non-coding introduction to the world of cloud computing.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Introduction to Statistics in Spreadsheets</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Intermediate Spreadsheets, Spreadsheet</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to leverage statistical techniques using spreadsheets to more effectively work with and extract insights from your data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Data Science for Business</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Skills for Business, Theory</td>\n",
       "      <td>20</td>\n",
       "      <td>Learn about data science and how can you use it to strengthen your organization.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Machine Learning for Business</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Skills for Business, Theory</td>\n",
       "      <td>20</td>\n",
       "      <td>Understand the fundamentals of Machine Learning and how it's applied in the business world.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Data-Driven Decision Making for Business</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Skills for Business, Theory</td>\n",
       "      <td>20</td>\n",
       "      <td>Discover how to make better business decisions by applying practical data frameworks—no coding required.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Marketing Analytics for Business</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Skills for Business, Theory</td>\n",
       "      <td>20</td>\n",
       "      <td>Discover how Marketing Analysts use data to understand customers and drive business growth.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Introduction to Data Science in Python</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Data Skills for Business, Theory</td>\n",
       "      <td>20</td>\n",
       "      <td>Dive into data science using Python and learn how to effectively analyze and visualize your data. No coding experience or skills needed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>AI Fundamentals</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Skills for Business, Theory</td>\n",
       "      <td>20</td>\n",
       "      <td>Learn the fundamentals of AI. No programming experience required!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Course  Course Length  \\\n",
       "0              Data Visualization for Everyone              2   \n",
       "1                Data Engineering for Everyone              2   \n",
       "2                    Data Science for Everyone              2   \n",
       "3                Machine Learning for Everyone              2   \n",
       "4                 Cloud Computing for Everyone              2   \n",
       "5   Introduction to Statistics in Spreadsheets              4   \n",
       "6                    Data Science for Business              2   \n",
       "7                Machine Learning for Business              2   \n",
       "8     Data-Driven Decision Making for Business              2   \n",
       "9             Marketing Analytics for Business              2   \n",
       "10      Introduction to Data Science in Python              4   \n",
       "11                             AI Fundamentals              4   \n",
       "\n",
       "    Track Duplication                          Shortest Track  \\\n",
       "0                   2      Data Literacy Fundamentals, Theory   \n",
       "1                   2      Data Literacy Fundamentals, Theory   \n",
       "2                   1      Data Literacy Fundamentals, Theory   \n",
       "3                   1      Data Literacy Fundamentals, Theory   \n",
       "4                   1      Data Literacy Fundamentals, Theory   \n",
       "5                   2  Intermediate Spreadsheets, Spreadsheet   \n",
       "6                   1        Data Skills for Business, Theory   \n",
       "7                   1        Data Skills for Business, Theory   \n",
       "8                   1        Data Skills for Business, Theory   \n",
       "9                   1        Data Skills for Business, Theory   \n",
       "10                  3        Data Skills for Business, Theory   \n",
       "11                  1        Data Skills for Business, Theory   \n",
       "\n",
       "    Track Time Remaining  \\\n",
       "0                     10   \n",
       "1                     10   \n",
       "2                     10   \n",
       "3                     10   \n",
       "4                     10   \n",
       "5                     12   \n",
       "6                     20   \n",
       "7                     20   \n",
       "8                     20   \n",
       "9                     20   \n",
       "10                    20   \n",
       "11                    20   \n",
       "\n",
       "                                                                                                                          Course Description  \n",
       "0                                                                             An introduction to data visualization with no coding involved.  \n",
       "1                                       Discover how data engineers lay the groundwork that makes data science possible. No coding involved!  \n",
       "2                                                                                   An introduction to data science with no coding involved.  \n",
       "3                                                                               An introduction to machine learning with no coding involved.  \n",
       "4                                                                                 A non-coding introduction to the world of cloud computing.  \n",
       "5         Learn how to leverage statistical techniques using spreadsheets to more effectively work with and extract insights from your data.  \n",
       "6                                                           Learn about data science and how can you use it to strengthen your organization.  \n",
       "7                                                Understand the fundamentals of Machine Learning and how it's applied in the business world.  \n",
       "8                                   Discover how to make better business decisions by applying practical data frameworks—no coding required.  \n",
       "9                                                Discover how Marketing Analysts use data to understand customers and drive business growth.  \n",
       "10  Dive into data science using Python and learn how to effectively analyze and visualize your data. No coding experience or skills needed.  \n",
       "11                                                                         Learn the fundamentals of AI. No programming experience required!  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Tableau:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Case Study: Analyzing Customer Churn in Tableau</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Tableau Fundamentals, Tableau</td>\n",
       "      <td>25</td>\n",
       "      <td>You will investigate a dataset from a fictitious company called Databel in Tableau, and need to figure out why customers are churning.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Creating Dashboards in Tableau</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Tableau Fundamentals, Tableau</td>\n",
       "      <td>25</td>\n",
       "      <td>Dashboards are a must-have in a data-driven world. Increase your impact on business performance with Tableau dashboards.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Connecting Data in Tableau</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Tableau Fundamentals, Tableau</td>\n",
       "      <td>25</td>\n",
       "      <td>Learn to connect Tableau to different data sources and prepare the data for a smooth analysis.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Introduction to Tableau</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>Tableau Fundamentals, Tableau</td>\n",
       "      <td>25</td>\n",
       "      <td>Get started with Tableau, a widely used business intelligence (BI) and analytics software to explore, visualize, and securely share data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Analyzing Data in Tableau</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>Tableau Fundamentals, Tableau</td>\n",
       "      <td>25</td>\n",
       "      <td>Take your Tableau skills up a notch with advanced analytics and visualizations.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            Course  Course Length  \\\n",
       "0  Case Study: Analyzing Customer Churn in Tableau              3   \n",
       "1                   Creating Dashboards in Tableau              4   \n",
       "2                       Connecting Data in Tableau              4   \n",
       "3                          Introduction to Tableau              6   \n",
       "4                        Analyzing Data in Tableau              8   \n",
       "\n",
       "   Track Duplication                 Shortest Track  Track Time Remaining  \\\n",
       "0                  1  Tableau Fundamentals, Tableau                    25   \n",
       "1                  1  Tableau Fundamentals, Tableau                    25   \n",
       "2                  1  Tableau Fundamentals, Tableau                    25   \n",
       "3                  1  Tableau Fundamentals, Tableau                    25   \n",
       "4                  1  Tableau Fundamentals, Tableau                    25   \n",
       "\n",
       "                                                                                                                          Course Description  \n",
       "0     You will investigate a dataset from a fictitious company called Databel in Tableau, and need to figure out why customers are churning.  \n",
       "1                   Dashboards are a must-have in a data-driven world. Increase your impact on business performance with Tableau dashboards.  \n",
       "2                                             Learn to connect Tableau to different data sources and prepare the data for a smooth analysis.  \n",
       "3  Get started with Tableau, a widely used business intelligence (BI) and analytics software to explore, visualize, and securely share data.  \n",
       "4                                                            Take your Tableau skills up a notch with advanced analytics and visualizations.  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "R:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Course</th>\n",
       "      <th>Course Length</th>\n",
       "      <th>Track Duplication</th>\n",
       "      <th>Shortest Track</th>\n",
       "      <th>Track Time Remaining</th>\n",
       "      <th>Course Description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Intermediate Data Visualization with ggplot2</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Data Visualization, R</td>\n",
       "      <td>8</td>\n",
       "      <td>Learn to use facets, coordinate systems and statistics in ggplot2 to create meaningful explanatory plots.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Visualization Best Practices in R</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Data Visualization, R</td>\n",
       "      <td>8</td>\n",
       "      <td>Learn to  effectively convey your data with an overview of common charts, alternative visualization types, and perception-driven style enhancements.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Introduction to Statistics in R</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>Data Analyst, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Grow your statistical skills and learn how to collect, analyze, and draw accurate conclusions from data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Joining Data with dplyr</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Data Analyst, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to combine data across multiple tables to answer more complex questions with dplyr.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Data Manipulation with R</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Data Analyst, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn how to efficiently collect and download data from any website using R.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Writing Efficient R Code</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>R Programming, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Learn to write faster R code, discover benchmarking and profiling, and unlock the secrets of parallel programming.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Introduction to Writing Functions in R</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>R Programming, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Use cluster analysis to glean insights into cryptocurrency gambling behavior.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Unsupervised Learning in R</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Unsupervised Machine Learning, R</td>\n",
       "      <td>12</td>\n",
       "      <td>This course provides an intro to clustering and dimensionality reduction in R from a machine learning perspective.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Cluster Analysis in R</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Unsupervised Machine Learning, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Develop a strong intuition for how hierarchical and k-means clustering work and learn how to apply them to extract insights from your data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Factor Analysis in R</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Unsupervised Machine Learning, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Explore latent variables, such as personality using exploratory and confirmatory factor analyses.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Object-Oriented Programming with S3 and R6 in R</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>R Programming, R</td>\n",
       "      <td>12</td>\n",
       "      <td>Manage the complexity in your code using object-oriented programming with the S3 and R6 systems.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Course  Course Length  \\\n",
       "0      Intermediate Data Visualization with ggplot2              4   \n",
       "1                 Visualization Best Practices in R              4   \n",
       "2                   Introduction to Statistics in R              4   \n",
       "3                           Joining Data with dplyr              4   \n",
       "4                          Data Manipulation with R              4   \n",
       "5                          Writing Efficient R Code              4   \n",
       "6            Introduction to Writing Functions in R              4   \n",
       "7                        Unsupervised Learning in R              4   \n",
       "8                             Cluster Analysis in R              4   \n",
       "9                              Factor Analysis in R              4   \n",
       "10  Object-Oriented Programming with S3 and R6 in R              4   \n",
       "\n",
       "    Track Duplication                    Shortest Track  Track Time Remaining  \\\n",
       "0                   2             Data Visualization, R                     8   \n",
       "1                   1             Data Visualization, R                     8   \n",
       "2                   4                   Data Analyst, R                    12   \n",
       "3                   3                   Data Analyst, R                    12   \n",
       "4                   3                   Data Analyst, R                    12   \n",
       "5                   3                  R Programming, R                    12   \n",
       "6                   3                  R Programming, R                    12   \n",
       "7                   3  Unsupervised Machine Learning, R                    12   \n",
       "8                   2  Unsupervised Machine Learning, R                    12   \n",
       "9                   2  Unsupervised Machine Learning, R                    12   \n",
       "10                  1                  R Programming, R                    12   \n",
       "\n",
       "                                                                                                                                      Course Description  \n",
       "0                                              Learn to use facets, coordinate systems and statistics in ggplot2 to create meaningful explanatory plots.  \n",
       "1   Learn to  effectively convey your data with an overview of common charts, alternative visualization types, and perception-driven style enhancements.  \n",
       "2                                               Grow your statistical skills and learn how to collect, analyze, and draw accurate conclusions from data.  \n",
       "3                                                              Learn to combine data across multiple tables to answer more complex questions with dplyr.  \n",
       "4                                                                           Learn how to efficiently collect and download data from any website using R.  \n",
       "5                                     Learn to write faster R code, discover benchmarking and profiling, and unlock the secrets of parallel programming.  \n",
       "6                                                                          Use cluster analysis to glean insights into cryptocurrency gambling behavior.  \n",
       "7                                     This course provides an intro to clustering and dimensionality reduction in R from a machine learning perspective.  \n",
       "8            Develop a strong intuition for how hierarchical and k-means clustering work and learn how to apply them to extract insights from your data.  \n",
       "9                                                      Explore latent variables, such as personality using exploratory and confirmatory factor analyses.  \n",
       "10                                                      Manage the complexity in your code using object-oriented programming with the S3 and R6 systems.  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Reorder technologies to preferred order\n",
    "tech_list_ordered = ['Python', 'Power BI', 'SQL', 'Spreadsheet', 'Theory', 'Tableau', 'R']\n",
    "# Add any extracted technologies not in the ordered list to the end of the list\n",
    "tech_list_ordered += [x for x in technologies if x not in tech_list_ordered]\n",
    "\n",
    "# Get recommendations for each technology\n",
    "for tech in tech_list_ordered:\n",
    "    print(f\"{tech}:\")\n",
    "    display(get_recommendations(tech, k_tracks=4, row_limit=12))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DC Recommender shows a good amount of useful information about suitable courses and tracks. Regarding practical use, it may just be used to point me in the direction of appropriate tracks, but even that is useful and can help as a motivator. This notebook has been written such that certain chunks of code can be commented out to avoid unecessary scraping/processing, and commented back in to allow refreshing the cached data.\n",
    "\n",
    "All-in-all, I believe this is a useful script to direct my learning focus, through use of web scraping, data engineering, and production of a bespoke recommendation engine. However, if it seems for a moment that this may have been just a big project to organise doing some courses rather than actually doing them, well I'll let you know my progress!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 64-bit (windows store)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "bf0f26a8c897a5d8040c09a348bc21d597657cf5ce6c851648692949987d84e0"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}