Phase 2 - Crawling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "75761ea8",
   "metadata": {},
   "source": [
    "# Data Science Project | Common Vulnerability Exposures\n",
    "## Phase 2 - Crawling\n",
    "\n",
    "CVE, short for Common Vulnerabilities and Exposures, is a list of publicly disclosed computer security flaws. This project analyzes the different fluctuations in the Cyber Security world.\n",
    "\n",
    "For this project, we used [cvedetails.com](https://www.cvedetails.com/), which provides an easy-to-use web interface to CVE vulnerability data. You can browse for vendors, products and versions and view CVE entries, and vulnerabilities related to them. You can view statistics about vendors, products and versions of products. CVE details are displayed on a single, easy-to-use page, see a sample here.\n",
    "\n",
    "CVE vulnerability data are taken from National Vulnerability Database (NVD) XML feeds provided by the National Institute of Standards and Technology. Additional data from several sources like exploits from www.exploit-db.com, vendor statements and additional vendor-supplied data, and Metasploit modules are also published in addition to NVD CVE data.\n",
    "\n",
    "Inside crawling() function we are:\n",
    "\n",
    "1 – Creating a .CSV file in order to write all the data acquired from cvedetails.com.\n",
    "\n",
    "2 – Editing the .CSV file with the right fitted columns names.\n",
    "\n",
    "3 – Using 10 different URLs, which are relevant for us to acquire valuable data from the site.\n",
    "\n",
    "4 – Inside the root loop, we are iterating every URL and creating a BeautifulSoup “soup” in order to acquire the relevant data.\n",
    "\n",
    "5 – Inside the soup, we scraped the data into variables and typed it into the .CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6813933",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import all libraries\n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "from csv import writervaluable "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2aca3e8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def crawling():\n",
    "    # Open new CSV file\n",
    "    with open('cve_DF.csv', 'w', encoding='utf8', newline='') as f:\n",
    "        thewriter = writer(f)\n",
    "        \n",
    "        # Type all columns names\n",
    "        header = ['CVE_ID', 'Vulnerability Type', 'Publish Date', 'Update Date', 'Score', 'Gained Access Level', 'Access',\n",
    "                  'Complexity', 'Authentication', 'Confidentiality', 'Integrity', 'Availability']\n",
    "        thewriter.writerow(header)\n",
    "        \n",
    "        # Dictionary contains all the relevant URLs\n",
    "        custom_URLs = {\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0.99&year=0&month=0&cweid=0&order=1&trc=791&sha=8cc1914b56963dbe370dba9383f2100c35cf3354\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=1&cvssscoremax=1.99&year=0&month=0&cweid=0&order=1&trc=1172&sha=2ee333910b0146938c917d67aa60489f458567df\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=2&cvssscoremax=2.99&year=0&month=0&cweid=0&order=1&trc=8041&sha=309000079e74eb94dd19bab74ae6ee2280672c3b\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=3&cvssscoremax=3.99&year=0&month=0&cweid=0&order=1&trc=8854&sha=a5f82bbf2e3b4c1a44673f277621d24e63e478cb\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=4&cvssscoremax=4.99&year=0&month=0&cweid=0&order=1&trc=41489&sha=24a76f217bca516d9ddb84350e69616cb8f78973\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=5&cvssscoremax=5.99&year=0&month=0&cweid=0&order=1&trc=33144&sha=487699e41035ffbb827ae68be83ace9ccd82c221\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=6&cvssscoremax=6.99&year=0&month=0&cweid=0&order=1&trc=26093&sha=74e0bf73b5c24af1d8fa0497960c60660de4f638\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=7&cvssscoremax=7.99&year=0&month=0&cweid=0&order=1&trc=34956&sha=8d1dce4336dc15b67abd26e84cfd7dee885ac426\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=8&cvssscoremax=8.99&year=0&month=0&cweid=0&order=1&trc=870&sha=47211ec39e8a5bfc696c510450016af4d6c6f60d\",\n",
    "            \"https://www.cvedetails.com/vulnerability-list.php?vendor_id=0&product_id=0&version_id=0&page=\": \"&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=9&cvssscoremax=10&year=0&month=0&cweid=0&order=1&trc=19533&sha=c560d509f935c26128bfb13d2f2dadfcea62215b\"\n",
    "        }\n",
    "        \n",
    "        # Iterate troughout all the soups\n",
    "        i = 1\n",
    "        for key, value in custom_URLs.items():\n",
    "            custom_URL = key + \"1\" + value\n",
    "            page = requests.get(custom_URL)\n",
    "            soup = BeautifulSoup(page.content, 'html.parser')\n",
    "            paging = soup.find_all('div', class_='paging')[1]\n",
    "            numOfInd = [b.string for b in paging.findAll('b')]\n",
    "            numOfIndex = int(numOfInd[0])\n",
    "            count = 0\n",
    "\n",
    "            i = 1\n",
    "            while count <= numOfIndex:\n",
    "                custom_URL = key+str(i)+value\n",
    "                page = requests.get(custom_URL)\n",
    "                soup = BeautifulSoup(page.content, 'html.parser')\n",
    "                all_Results = soup.find_all('table', class_=\"searchresults sortable\")\n",
    "                for result in all_Results:\n",
    "                    all_CVEs = result.find_all('tr', class_=\"srrowns\")\n",
    "\n",
    "            # Find all the relevant values\n",
    "                for cve in all_CVEs:\n",
    "                    cve_ID = cve.find_all('td')[1].text\n",
    "                    vulnerability_Type = cve.find_all('td')[4].text\n",
    "                    publish_Date = cve.find_all('td')[5].text\n",
    "                    update_Date = cve.find_all('td')[6].text\n",
    "                    score = cve.find_all('td')[7].text\n",
    "                    gained_Access_Level = cve.find_all('td')[8].text\n",
    "                    access = cve.find_all('td')[9].text\n",
    "                    complexity = cve.find_all('td')[10].text\n",
    "                    authentication = cve.find_all('td')[11].text\n",
    "                    confidentiality = cve.find_all('td')[12].text\n",
    "                    integrity = cve.find_all('td')[13].text\n",
    "                    availability = cve.find_all('td')[14].text\n",
    "                    \n",
    "                    # Type the values into the CSV\n",
    "                    info = [cve_ID, vulnerability_Type, publish_Date, update_Date, score, gained_Access_Level, access, complexity, authentication, confidentiality, integrity, availability]\n",
    "                    thewriter.writerow(info)\n",
    "                    count = count + 1\n",
    "                    if count > numOfIndex:\n",
    "                        break\n",
    "                i = i + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dc276f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute function\n",
    "crawling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}