Project 3 - Airbnb_copy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 3: Airbnb\n",
    "**This is the third of three mandatory projects to be handed in as part of the assessment for the course 02807 Computational Tools for Data Science at Technical University of Denmark, autumn 2019.**\n",
    "\n",
    "#### Practical info\n",
    "- **The project is to be done in groups of at most 3 students**\n",
    "- **Each group has to hand in _one_ Jupyter notebook (this notebook) with their solution**\n",
    "- **The hand-in of the notebook is due 2019-12-05, 23:59 on DTU Inside**\n",
    "\n",
    "#### Your solution\n",
    "- **Your solution should be in Python/PySpark**\n",
    "- **For each question you may use as many cells for your solution as you like**\n",
    "- **You should not remove the problem statements**\n",
    "- **Your notebook should be runnable, i.e., clicking [>>] in Jupyter should generate the result that you want to be assessed**\n",
    "- **You are not expected to use machine learning to solve any of the exercises**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "[Airbnb](http://airbnb.com) is an online marketplace for arranging or offering lodgings. In this project you will use Spark to analyze data obtained from the Airbnb website. The purpose of the analysis is to extract information about trends and patterns from the data.\n",
    "\n",
    "The project has two parts.\n",
    "\n",
    "### Part 1: Loading, describing and preparing the data\n",
    "There's quite a lot of data. Make sure that you can load and correctly parse the data, and that you understand what the dataset contains. You should also prepare the data for the analysis in part two. This means cleaning it and staging it so that subsequent queries are fast.\n",
    "\n",
    "### Par 2: Analysis\n",
    "In this part your goal is to learn about trends and usage patterns from the data. You should give solutions to the tasks defined in this notebook, and you should use Spark to do the data processing. You may use other libraries like for instance Pandas and matplotlib for visualisation.\n",
    "\n",
    "## Guidelines\n",
    "- Processing data should be done using Spark. Once data has been reduced to aggregate form, you may use collect to extract it into Python for visualisation.\n",
    "- Your solutions will be evaluated by correctness, code quality and interpretability of the output. This means that you have to write clean and efficient Spark code that will generate sensible execution plans, and that the tables and visualisations that you produce are meaningful and easy to read.\n",
    "- You may add more cells for your solutions, but you should not modify the notebook otherwise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Spark session and define imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import *\n",
    "from pyspark.sql import functions as f\n",
    "from pyspark.sql.types import *\n",
    "\n",
    "spark = SparkSession.builder.appName(\"SparkIntro\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Loading, describing and preparing the data\n",
    "The data comes in two files. Start by downloading the files and putting them in your `data/` folder.\n",
    "\n",
    "- [Listings](https://files.dtu.dk/u/siPzAasj8w2gI_ME/listings.csv?l) (5 GB)\n",
    "- [Reviews](https://files.dtu.dk/u/k3oaPYp6GjKBeho4/reviews.csv?l) (9.5 GB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the data\n",
    "The data has multiline rows (rows that span multiple lines in the file). To correctly parse these you should use the `multiline` option and set the `escape` character to be `\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "import csv\n",
    "import pandas as pd\n",
    "import pyspark.sql.functions as f\n",
    "from functools import reduce\n",
    "from pyspark.sql.types import *\n",
    "from pyspark.sql.functions import substring, length, col, expr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Slicing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "entries = []\n",
    "with open('/home/jovyan/work/Project3/listings.csv', 'r') as f:\n",
    "    mycsv = csv.reader(f)\n",
    "    for row in itertools.islice(mycsv, 10000):\n",
    "        entries.append(row)\n",
    "\n",
    "columns_ = entries[0]\n",
    "df = pd.DataFrame(entries,columns=columns_)\n",
    "df = df.iloc[1:-1,:]\n",
    "df.head()\n",
    "#df.to_csv(\"listings_sub_10000.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Reading into spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.option(\"delimiter\",\",\").option(\"multiline\",True).option(\"header\", True).option(\"escape\",'\"').option(\"inferSchema\",True).csv('listings_sub_10000.csv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.select(\"listing_url\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe the data\n",
    "List the features (schema) and sizes of the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"rows: \", df.count(), \"columns: \", len(df.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Removing all rows which has null\n",
    "df.where(reduce(lambda x, y: x & y,  (f.col(x).isNotNull() for x in df.columns))).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Removing all rows which has null\n",
    "df.dropna(how='any').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the data for analysis\n",
    "You should prepare two dataframes to be used in the analysis part of the project. You should not be concerned with cleaning the data. There's a lot of it, so it will be sufficient to drop rows that have bad values. You may want to go back and refine this step at a later point when doing the analysis.\n",
    "\n",
    "You may also want to consider if you can stage your data so that subsequent processing is more efficient (this is not strictly necessary for Spark to run, but you may be able to decrease the time you sit around waiting for Spark to finish things)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Analysis\n",
    "Use Spark and your favorite tool for data visualization to solve the following tasks.\n",
    "\n",
    "## The basics\n",
    "Compute and show a dataframe with the number of listings and neighbourhoods per city."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the table above, you should choose a city that you want to continue your analysis for. The city should have mulitple neighbourhoods with listings in them.\n",
    "\n",
    "Compute and visualize the number of listings of different property types per neighbourhood in your city."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prices\n",
    "Compute the minimum, maximum and average listing price in your city. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city = \"Stockholm\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_s = df.filter(f.col(\"city\")==city)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_s.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df_s.price = df_s.price.cast('float')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import substring, length, col, expr\n",
    "df_s = df_s.withColumn(\"price\",expr(\"substring(price, 2, length(price))\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  pyspark.sql.functions import regexp_replace,col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_s = df_s.withColumn(\"price\", regexp_replace(f.col(\"price\"), \",\", \"\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df_s.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df_s.select(df_s.price.cast(\"float\").alias('price_new')).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *\n",
    "df_d = df_s.withColumn(\"price\",df_s[\"price\"].cast(DoubleType()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df_d.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Basic statistics for\", city)\n",
    "df_d.select(f.avg('price').alias(\"Averge price\"), f.max('price').alias(\"Max price\"), f.min('price').alias(\"Min price\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute and visualize the distribution of listing prices in your city."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy import stats\n",
    "sns.set(rc={'figure.figsize':(16,6)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_city_p = df_d.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sns.distplot(df_city_p.price, kde=False, rug=False, bins=150);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value of a listing is its rating divided by its price.\n",
    "\n",
    "Compute and show a dataframe with the 3 highest valued listings in each neighbourhood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import lit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_d = df_d.withColumn(\"review_scores_rating\",df_s[\"review_scores_rating\"].cast(DoubleType()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_d = df_d.withColumn(\"value_listing\",lit(f.col(\"review_scores_rating\")/f.col(\"price\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_value = df_d.select(f.col(\"city\"),f.col(\"neighbourhood\"),(f.col(\"value_listing\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df_value.orderBy(df_value.value_listing.desc()).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_by_value = Window.partitionBy('neighbourhood').orderBy(f.desc('value_listing'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranked_by_value = df_d.withColumn('value_listing_rank', f.rank().over(sorted_by_value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranked_df = ranked_by_value.filter(f.col('value_listing_rank') <= 3).drop('value_listing_rank').orderBy('neighbourhood', f.desc('value_listing'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ranked_df.select(f.col(\"neighbourhood\"),f.col(\"value_listing\"),f.col(\"city\"),f.col(\"name\")).show(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trends\n",
    "Now we want to analyze the \"popularity\" of your city. The data does not contain the number of bookings per listing, but we have a large number of reviews, and we will assume that this is a good indicator of activity on listings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute and visualize the popularity (i.e., number of reviews) of your city over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute and visualize the popularity of neighbourhoods over time. If there are many neighbourhoods in your city, you should select a few interesting ones for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute and visualize the popularity of your city by season. For example, visualize the popularity of your city per month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reviews\n",
    "In this part you should determine which words used in reviews that are the most positive. \n",
    "\n",
    "The individual reviews do not have a rating of the listing, so we will assume that each review gave the average rating to the listing, i.e., the one on the listing.\n",
    "\n",
    "You should assign a positivity weight to each word seen in reviews and list the words with the highest weight. It is up to you to decide what the weight should be. For example, it can be a function of the rating on the listing on which it occurs, the number of reviews it occurs in, and the number of unique listings for which it was used to review.\n",
    "\n",
    "Depending on your choice of weight function, you may also want to do some filtering of words. For example, remove words that only occur in a few reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "import csv\n",
    "\n",
    "entries = []\n",
    "with open('../Project3/reviews.csv', 'r') as f:\n",
    "    mycsv = csv.reader(f)\n",
    "    for row in itertools.islice(mycsv, 10000):\n",
    "        entries.append(row)\n",
    "        \n",
    "import pandas as pd\n",
    "columns_ = entries[0]\n",
    "df = pd.DataFrame(entries,columns=columns_)\n",
    "df = df.iloc[1:-1,:]\n",
    "#df.head()\n",
    "#df.to_csv(\"reviews_sub_10000.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-0ed96d398792>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"reviews_sub_10000.csv\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0ml\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"listings_sub_10000.csv\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "r = pd.read_csv(\"reviews_sub_10000.csv\")\n",
    "l = pd.read_csv(\"listings_sub_10000.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the listings we need to:\n",
    "- calculate the average rating for each ID\n",
    "- join the average rating on to the ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfl = spark.read.option(\"delimiter\",\",\").option(\"multiline\",True).option(\"header\", True).option(\"escape\",'\"').option(\"inferSchema\",True).csv('listings_sub_10000.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfr = spark.read.option(\"delimiter\",\",\").option(\"multiline\",True).option(\"header\", True).option(\"escape\",'\"').option(\"inferSchema\",True).csv('reviews_sub_10000.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- _c0: integer (nullable = true)\n",
      " |-- listing_id: integer (nullable = true)\n",
      " |-- id: integer (nullable = true)\n",
      " |-- date: timestamp (nullable = true)\n",
      " |-- reviewer_id: integer (nullable = true)\n",
      " |-- reviewer_name: string (nullable = true)\n",
      " |-- comments: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfr.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import col, lower, regexp_replace, split\n",
    "\n",
    "def clean_text(c):\n",
    "    c = lower(c)\n",
    "    c = regexp_replace(c, \"(https?\\://)\\S+\", \"\")\n",
    "    c = regexp_replace(c, \"[^a-zA-Z0-9\\\\s]\", \"\")\n",
    "    return c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfr = dfr.withColumn(\"text_clean\", clean_text(col(\"comments\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|          text_clean|\n",
      "+--------------------+\n",
      "|prima plek om sto...|\n",
      "|cosy and clean fl...|\n",
      "|the host canceled...|\n",
      "|kims place was ou...|\n",
      "|great spacious ap...|\n",
      "|kim is a very fri...|\n",
      "|the apartment is ...|\n",
      "|nicely appointed ...|\n",
      "|it was a pleasure...|\n",
      "|kims place is sim...|\n",
      "|we absolutely lov...|\n",
      "|amazing place to ...|\n",
      "|very pleasant sta...|\n",
      "|we had a wonderfu...|\n",
      "|great location co...|\n",
      "|lovely flat in a ...|\n",
      "|kims flat is in a...|\n",
      "|i would definitel...|\n",
      "|kims place is per...|\n",
      "|the flat is so co...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfr.select(f.col(\"text_clean\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "#mean_scores = dfl.groupBy(\"id\").agg(f.mean(\"review_scores_rating\").alias(\"average_rating\"))\n",
    "#mean_scores = mean_scores.select(f.col(\"id\").alias(\"id_\"), f.col(\"average_rating\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfl = dfl.select(f.col(\"id\").alias(\"id_\"), f.col(\"review_scores_rating\").alias(\"average_rating\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.conf.set(\"spark.sql.autoBroadcastJoinThreshold\", -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+--------------+\n",
      "|   id_|average_rating|\n",
      "+------+--------------+\n",
      "|145320|            97|\n",
      "|155220|          null|\n",
      "|155685|            94|\n",
      "|164448|            97|\n",
      "|170651|            93|\n",
      "|206221|            98|\n",
      "|220233|          null|\n",
      "|220851|            93|\n",
      "|242188|            97|\n",
      "|259025|            89|\n",
      "|259722|            92|\n",
      "|273906|           100|\n",
      "|274876|            99|\n",
      "|278311|            92|\n",
      "|283446|            97|\n",
      "|299154|            92|\n",
      "|301680|            92|\n",
      "|302496|            99|\n",
      "|313195|            96|\n",
      "|313763|            98|\n",
      "+------+--------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfl.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfr = dfr.join(dfl, (f.col('listing_id') == f.col('id_')), 'inner').drop('id_')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+\n",
      "| _c0|listing_id|      id|               date|reviewer_id|     reviewer_name|            comments|          text_clean|average_rating|\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+\n",
      "|9766|   1271843| 8513212|2013-11-03 00:00:00|    4641577|            Pamela|André was my firs...|andr was my first...|            99|\n",
      "|9767|   1271843| 9043505|2013-12-02 00:00:00|    6890450|              Emma|Andre was the bes...|andre was the bes...|            99|\n",
      "|9768|   1271843| 9533868|2014-01-01 00:00:00|    9496791|   Julie And Chris|André was a fanta...|andr was a fantas...|            99|\n",
      "|9769|   1271843|10049736|2014-01-27 00:00:00|     940929|             Denis|It was a right ch...|it was a right ch...|            99|\n",
      "|9770|   1271843|10324840|2014-02-13 00:00:00|   11454004|             Filip|We stayed in Stoc...|we stayed in stoc...|            99|\n",
      "|9771|   1271843|10396795|2014-02-17 00:00:00|   10505993|             Pablo|Stop looking for ...|stop looking for ...|            99|\n",
      "|9772|   1271843|11969012|2014-04-20 00:00:00|    1865604|              Eric|André pretty much...|andr pretty much ...|            99|\n",
      "|9773|   1271843|33486084|2015-05-28 00:00:00|    2039120|             Lynne|Andre's fiancé Jo...|andres fianc joha...|            99|\n",
      "|9774|   1271843|33756415|2015-05-31 00:00:00|    1116857|              Dave|Andre was away wh...|andre was away wh...|            99|\n",
      "|9775|   1271843|33949154|2015-06-02 00:00:00|    2730754|Richard En Jolande|We zijn hartelijk...|we zijn hartelijk...|            99|\n",
      "|9776|   1271843|34575016|2015-06-09 00:00:00|    3680651|          Katarina|I stayed in Stock...|i stayed in stock...|            99|\n",
      "|9777|   1271843|35240315|2015-06-16 00:00:00|    9964210|            Jasmin|Andre and Johanna...|andre and johanna...|            99|\n",
      "|9778|   1271843|35938455|2015-06-23 00:00:00|     981642|              Leif|Andre and Johanna...|andre and johanna...|            99|\n",
      "|9779|   1271843|36986749|2015-07-03 00:00:00|    3077469|            Arleen|We really enjoyed...|we really enjoyed...|            99|\n",
      "|9780|   1271843|37598608|2015-07-08 00:00:00|    4549906|              Evan|André and Johanna...|andr and johanna ...|            99|\n",
      "|9781|   1271843|41486488|2015-08-07 00:00:00|   25536243|              Ulla|The host was very...|the host was very...|            99|\n",
      "|9782|   1271843|42958488|2015-08-16 00:00:00|   40102284|               Teo|André and Johanna...|andr and johanna ...|            99|\n",
      "|9783|   1271843|44462335|2015-08-26 00:00:00|   28465077|           Carolyn|A friend and I st...|a friend and i st...|            99|\n",
      "|9784|   1271843|46175902|2015-09-08 00:00:00|   10574638|              Alex|Andre and Johanna...|andre and johanna...|            99|\n",
      "|9785|   1271843|48752427|2015-09-28 00:00:00|    5849808|        Seb & Stef|André and Johanna...|andr and johanna ...|            99|\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfr.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Tokenizer, RegexTokenizer\n",
    "\n",
    "tokenizer = Tokenizer(inputCol=\"text_clean\", outputCol=\"words\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfr = tokenizer.transform(dfr)\n",
    "#tokenized.select(\"comments\", \"words\")\\\n",
    "    #.withColumn(\"tokens\", countTokens(col(\"words\"))).show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+--------------------+\n",
      "| _c0|listing_id|      id|               date|reviewer_id|     reviewer_name|            comments|          text_clean|average_rating|               words|\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+--------------------+\n",
      "|9766|   1271843| 8513212|2013-11-03 00:00:00|    4641577|            Pamela|André was my firs...|andr was my first...|            99|[andr, was, my, f...|\n",
      "|9767|   1271843| 9043505|2013-12-02 00:00:00|    6890450|              Emma|Andre was the bes...|andre was the bes...|            99|[andre, was, the,...|\n",
      "|9768|   1271843| 9533868|2014-01-01 00:00:00|    9496791|   Julie And Chris|André was a fanta...|andr was a fantas...|            99|[andr, was, a, fa...|\n",
      "|9769|   1271843|10049736|2014-01-27 00:00:00|     940929|             Denis|It was a right ch...|it was a right ch...|            99|[it, was, a, righ...|\n",
      "|9770|   1271843|10324840|2014-02-13 00:00:00|   11454004|             Filip|We stayed in Stoc...|we stayed in stoc...|            99|[we, stayed, in, ...|\n",
      "|9771|   1271843|10396795|2014-02-17 00:00:00|   10505993|             Pablo|Stop looking for ...|stop looking for ...|            99|[stop, looking, f...|\n",
      "|9772|   1271843|11969012|2014-04-20 00:00:00|    1865604|              Eric|André pretty much...|andr pretty much ...|            99|[andr, pretty, mu...|\n",
      "|9773|   1271843|33486084|2015-05-28 00:00:00|    2039120|             Lynne|Andre's fiancé Jo...|andres fianc joha...|            99|[andres, fianc, j...|\n",
      "|9774|   1271843|33756415|2015-05-31 00:00:00|    1116857|              Dave|Andre was away wh...|andre was away wh...|            99|[andre, was, away...|\n",
      "|9775|   1271843|33949154|2015-06-02 00:00:00|    2730754|Richard En Jolande|We zijn hartelijk...|we zijn hartelijk...|            99|[we, zijn, hartel...|\n",
      "|9776|   1271843|34575016|2015-06-09 00:00:00|    3680651|          Katarina|I stayed in Stock...|i stayed in stock...|            99|[i, stayed, in, s...|\n",
      "|9777|   1271843|35240315|2015-06-16 00:00:00|    9964210|            Jasmin|Andre and Johanna...|andre and johanna...|            99|[andre, and, joha...|\n",
      "|9778|   1271843|35938455|2015-06-23 00:00:00|     981642|              Leif|Andre and Johanna...|andre and johanna...|            99|[andre, and, joha...|\n",
      "|9779|   1271843|36986749|2015-07-03 00:00:00|    3077469|            Arleen|We really enjoyed...|we really enjoyed...|            99|[we, really, enjo...|\n",
      "|9780|   1271843|37598608|2015-07-08 00:00:00|    4549906|              Evan|André and Johanna...|andr and johanna ...|            99|[andr, and, johan...|\n",
      "|9781|   1271843|41486488|2015-08-07 00:00:00|   25536243|              Ulla|The host was very...|the host was very...|            99|[the, host, was, ...|\n",
      "|9782|   1271843|42958488|2015-08-16 00:00:00|   40102284|               Teo|André and Johanna...|andr and johanna ...|            99|[andr, and, johan...|\n",
      "|9783|   1271843|44462335|2015-08-26 00:00:00|   28465077|           Carolyn|A friend and I st...|a friend and i st...|            99|[a, friend, and, ...|\n",
      "|9784|   1271843|46175902|2015-09-08 00:00:00|   10574638|              Alex|Andre and Johanna...|andre and johanna...|            99|[andre, and, joha...|\n",
      "|9785|   1271843|48752427|2015-09-28 00:00:00|    5849808|        Seb & Stef|André and Johanna...|andr and johanna ...|            99|[andr, and, johan...|\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfr.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import StopWordsRemover\n",
    "\n",
    "# Define a list of stop words or use default list\n",
    "remover = StopWordsRemover()\n",
    "stopwords = remover.getStopWords() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify input/output columns\n",
    "remover.setInputCol(\"words\")\n",
    "remover.setOutputCol(\"comments_wo_stopwords\")\n",
    "\n",
    "# Transform existing dataframe with the StopWordsRemover\n",
    "dfr = remover.transform(dfr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+--------------------+---------------------+\n",
      "| _c0|listing_id|      id|               date|reviewer_id|     reviewer_name|            comments|          text_clean|average_rating|               words|comments_wo_stopwords|\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+--------------------+---------------------+\n",
      "|9766|   1271843| 8513212|2013-11-03 00:00:00|    4641577|            Pamela|André was my firs...|andr was my first...|            99|[andr, was, my, f...| [andr, first, hos...|\n",
      "|9767|   1271843| 9043505|2013-12-02 00:00:00|    6890450|              Emma|Andre was the bes...|andre was the bes...|            99|[andre, was, the,...| [andre, best, hos...|\n",
      "|9768|   1271843| 9533868|2014-01-01 00:00:00|    9496791|   Julie And Chris|André was a fanta...|andr was a fantas...|            99|[andr, was, a, fa...| [andr, fantastic,...|\n",
      "|9769|   1271843|10049736|2014-01-27 00:00:00|     940929|             Denis|It was a right ch...|it was a right ch...|            99|[it, was, a, righ...| [right, choice, s...|\n",
      "|9770|   1271843|10324840|2014-02-13 00:00:00|   11454004|             Filip|We stayed in Stoc...|we stayed in stoc...|            99|[we, stayed, in, ...| [stayed, stockhol...|\n",
      "|9771|   1271843|10396795|2014-02-17 00:00:00|   10505993|             Pablo|Stop looking for ...|stop looking for ...|            99|[stop, looking, f...| [stop, looking, p...|\n",
      "|9772|   1271843|11969012|2014-04-20 00:00:00|    1865604|              Eric|André pretty much...|andr pretty much ...|            99|[andr, pretty, mu...| [andr, pretty, mu...|\n",
      "|9773|   1271843|33486084|2015-05-28 00:00:00|    2039120|             Lynne|Andre's fiancé Jo...|andres fianc joha...|            99|[andres, fianc, j...| [andres, fianc, j...|\n",
      "|9774|   1271843|33756415|2015-05-31 00:00:00|    1116857|              Dave|Andre was away wh...|andre was away wh...|            99|[andre, was, away...| [andre, away, sta...|\n",
      "|9775|   1271843|33949154|2015-06-02 00:00:00|    2730754|Richard En Jolande|We zijn hartelijk...|we zijn hartelijk...|            99|[we, zijn, hartel...| [zijn, hartelijk,...|\n",
      "|9776|   1271843|34575016|2015-06-09 00:00:00|    3680651|          Katarina|I stayed in Stock...|i stayed in stock...|            99|[i, stayed, in, s...| [stayed, stockhol...|\n",
      "|9777|   1271843|35240315|2015-06-16 00:00:00|    9964210|            Jasmin|Andre and Johanna...|andre and johanna...|            99|[andre, and, joha...| [andre, johanna, ...|\n",
      "|9778|   1271843|35938455|2015-06-23 00:00:00|     981642|              Leif|Andre and Johanna...|andre and johanna...|            99|[andre, and, joha...| [andre, johanna, ...|\n",
      "|9779|   1271843|36986749|2015-07-03 00:00:00|    3077469|            Arleen|We really enjoyed...|we really enjoyed...|            99|[we, really, enjo...| [really, enjoyed,...|\n",
      "|9780|   1271843|37598608|2015-07-08 00:00:00|    4549906|              Evan|André and Johanna...|andr and johanna ...|            99|[andr, and, johan...| [andr, johanna, g...|\n",
      "|9781|   1271843|41486488|2015-08-07 00:00:00|   25536243|              Ulla|The host was very...|the host was very...|            99|[the, host, was, ...| [host, friendly, ...|\n",
      "|9782|   1271843|42958488|2015-08-16 00:00:00|   40102284|               Teo|André and Johanna...|andr and johanna ...|            99|[andr, and, johan...| [andr, johanna, e...|\n",
      "|9783|   1271843|44462335|2015-08-26 00:00:00|   28465077|           Carolyn|A friend and I st...|a friend and i st...|            99|[a, friend, and, ...| [friend, stayed, ...|\n",
      "|9784|   1271843|46175902|2015-09-08 00:00:00|   10574638|              Alex|Andre and Johanna...|andre and johanna...|            99|[andre, and, joha...| [andre, johanna, ...|\n",
      "|9785|   1271843|48752427|2015-09-28 00:00:00|    5849808|        Seb & Stef|André and Johanna...|andr and johanna ...|            99|[andr, and, johan...| [andr, johanna, p...|\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------+--------------------+--------------+--------------------+---------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfr.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+----------+--------+-------------------+-----------+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|_c0 |listing_id|id      |date               |reviewer_id|reviewer_name     |comments                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |average_rating|words                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |features                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|9766|1271843   |8513212 |2013-11-03 00:00:00|4641577    |Pamela            |André was my first host and i must say it's gonna be difficult to find better. He is  well organised ans helped me to make the best of my journey in Stockholm by giving me the best addresses. He lives in the best area of the town and His place is  spaceous, the bed very very confortable, the kitchen and bathroom perfect, It is really not a problem to stay with him also since the appartment is big And Because he Will make your trip better than you expected, trust me ;)                                                                                                                                                                                                                                                                                                          |99            |[andré, was, my, first, host, and, i, must, say, it's, gonna, be, difficult, to, find, better., he, is, , well, organised, ans, helped, me, to, make, the, best, of, my, journey, in, stockholm, by, giving, me, the, best, addresses., he, lives, in, the, best, area, of, the, town, and, his, place, is, , spaceous,, the, bed, very, very, confortable,, the, kitchen, and, bathroom, perfect,, it, is, really, not, a, problem, to, stay, with, him, also, since, the, appartment, is, big, and, because, he, will, make, your, trip, better, than, you, expected,, trust, me, ;)]                                                                                                                                                                                                                                                                                                                                                  |(36166,[0,1,2,3,4,5,6,7,8,11,12,13,16,17,18,20,28,31,38,45,52,55,58,59,60,68,79,82,93,111,113,127,136,137,146,152,157,194,209,221,235,237,281,291,293,297,362,425,457,509,711,721,765,987,1045,1205,1249,1293,1490,1526,1716,1790,1873,4610,5678,5789,17694,24682],[7.0,4.0,1.0,3.0,4.0,1.0,2.0,2.0,2.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,3.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                    |\n",
      "|9767|1271843   |9043505 |2013-12-02 00:00:00|6890450    |Emma              |Andre was the best host EVER. His flat is in such an amazing location, close to everything. He was really accommodating and showed us where to go, what to do, and got us into the best bars/clubs and restaurants. The flat was perfect for what we needed with a super comfy bed. Anytime we go back to Stockholm now we will definitely be staying with Andre! HIGHLY RECOMMEND.                                                                                                                                                                                                                                                                                                                                                                                                               |99            |[andre, was, the, best, host, ever., his, flat, is, in, such, an, amazing, location,, close, to, everything., he, was, really, accommodating, and, showed, us, where, to, go,, what, to, do,, and, got, us, into, the, best, bars/clubs, and, restaurants., the, flat, was, perfect, for, what, we, needed, with, a, super, comfy, bed., anytime, we, go, back, to, stockholm, now, we, will, definitely, be, staying, with, andre!, highly, recommend.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |(36166,[0,1,2,3,4,5,6,9,10,13,22,28,31,45,47,51,54,58,67,68,77,79,80,90,102,113,149,152,178,186,196,200,210,213,276,310,324,408,481,483,563,597,610,660,878,2204,2270,3017,3429,4050,10244,30082],[3.0,3.0,1.0,4.0,1.0,3.0,1.0,3.0,1.0,2.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "|9768|1271843   |9533868 |2014-01-01 00:00:00|9496791    |Julie And Chris   |André was a fantastic host, he went out of his way to show us around the local spots and made us feel very welcome! We had a great time with him- he made us feel really at home in his house, reccomended great eating and drinking spots which made our new year very special. I hope that we can keep in contact and repay the favour one day, it was a fantastic air b&b experience.                                                                                                                                                                                                                                                                                                                                                                                                          |99            |[andré, was, a, fantastic, host,, he, went, out, of, his, way, to, show, us, around, the, local, spots, and, made, us, feel, very, welcome!, we, had, a, great, time, with, him-, he, made, us, feel, really, at, home, in, his, house,, reccomended, great, eating, and, drinking, spots, which, made, our, new, year, very, special., i, hope, that, we, can, keep, in, contact, and, repay, the, favour, one, day,, it, was, a, fantastic, air, b&b, experience.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |(36166,[0,1,2,3,5,6,7,9,11,12,13,15,16,19,22,23,24,28,45,48,63,68,69,72,97,106,110,114,118,130,145,240,279,286,308,530,566,614,642,753,923,987,1189,1197,1304,1408,1485,1780,2678,3053,3386,4353,10686,21931,28784,29744],[2.0,3.0,3.0,1.0,2.0,2.0,2.0,2.0,1.0,1.0,1.0,2.0,1.0,1.0,3.0,1.0,1.0,1.0,2.0,1.0,1.0,2.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                             |\n",
      "|9769|1271843   |10049736|2014-01-27 00:00:00|940929     |Denis             |It was a right choice to stay at Andre`s apartment. The location is in the most interesting part of Stockholm full of bars, restaurants and small designer shops. There is a metro station in 5 minutes of walk from the flat. \n",
      "Right after booking Andre got in contact and gave us very precise manual how to get and what to do. We got in very smooth. Thanks Andre it was easy to communicate.\n",
      "The apartment is very clean and spacious with fully equipped kitchen and big common room. There are no animals inside. The room has enough space for two persons to stay on several days.\n",
      "Nothing to complain. It a was a good place to stay for our short shopping binge :)\n",
      "\n",
      "Denis\n",
      "                                                                                                          |99            |[it, was, a, right, choice, to, stay, at, andre`s, apartment., the, location, is, in, the, most, interesting, part, of, stockholm, full, of, bars,, restaurants, and, small, designer, shops., there, is, a, metro, station, in, 5, minutes, of, walk, from, the, flat., , right, after, booking, andre, got, in, contact, and, gave, us, very, precise, manual, how, to, get, and, what, to, do., we, got, in, very, smooth., thanks, andre, it, was, easy, to, communicate., the, apartment, is, very, clean, and, spacious, with, fully, equipped, kitchen, and, big, common, room., there, are, no, animals, inside., the, room, has, enough, space, for, two, persons, to, stay, on, several, days., nothing, to, complain., it, a, was, a, good, place, to, stay, for, our, short, shopping, binge, :), , denis]                                                                                                                   |(36166,[0,1,2,3,4,5,6,7,8,9,10,11,13,14,16,18,19,20,22,24,26,30,31,36,39,41,46,49,50,56,64,66,70,73,89,95,105,109,117,124,127,131,133,135,158,164,171,176,186,194,217,226,229,230,233,241,280,309,324,346,361,381,415,495,553,570,614,703,878,899,907,946,952,1080,1379,1394,1494,1594,2616,3462,4804,4849,4870,6661,20482,20979,23866,24553],[5.0,5.0,4.0,7.0,3.0,3.0,4.0,3.0,2.0,1.0,2.0,3.0,1.0,1.0,3.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                         |\n",
      "|9770|1271843   |10324840|2014-02-13 00:00:00|11454004   |Filip             |We stayed in Stockholm for 5 nights. Thanks to André for being such a kind host and trusting us with his spaceous and well-located design apartment. Most of the time, André was not at home, so we had the apartment all for ourselves. We've appreciated the privacy, his warm welcome, his clear communication and good upfront advice! The king-size bed was very soft and comfy. André knows the best places in town. Highly recommended.                                                                                                                                                                                                                                                                                                                                                    |99            |[we, stayed, in, stockholm, for, 5, nights., thanks, to, andré, for, being, such, a, kind, host, and, trusting, us, with, his, spaceous, and, well-located, design, apartment., most, of, the, time,, andré, was, not, at, home,, so, we, had, the, apartment, all, for, ourselves., we've, appreciated, the, privacy,, his, warm, welcome,, his, clear, communication, and, good, upfront, advice!, the, king-size, bed, was, very, soft, and, comfy., andré, knows, the, best, places, in, town., highly, recommended.]                                                                                                                                                                                                                                                                                                                                                                                                                |(36166,[0,1,2,3,5,6,7,9,10,11,13,14,19,22,23,31,40,41,42,58,59,68,80,105,124,129,134,136,152,176,177,215,225,229,310,352,427,464,468,562,782,838,987,1102,1198,1367,1390,1617,1783,2254,2389,3660,5058,6070,7018,12011,17292,23138],[5.0,4.0,1.0,1.0,2.0,2.0,1.0,2.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                           |\n",
      "|9771|1271843   |10396795|2014-02-17 00:00:00|10505993   |Pablo             |Stop looking for a place. You are in the right spot. Great host, very well located, clean, tidy and confortable. What else could you ask for?\n",
      "\n",
      "Thanks for everything                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |99            |[stop, looking, for, a, place., you, are, in, the, right, spot., great, host,, very, well, located,, clean,, tidy, and, confortable., what, else, could, you, ask, for?, , thanks, for, everything]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |(36166,[0,1,2,6,7,8,10,15,17,34,49,55,120,121,122,124,145,164,186,327,418,452,601,810,1167,1506,4226,5103],[1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n",
      "|9772|1271843   |11969012|2014-04-20 00:00:00|1865604    |Eric              |André pretty much sets the bar for hosting.  Couldn't have asked for more with his hospitality and wealth of knowledge of where to go in Stockholm.  What's hot, what's not, and all the insider tips for various neighborhoods proved to be a real help in navigating through all that this great city has to offer.  Even moreso,  André was very generous with his time in getting me situated and comfortable in his place along with a tour of his neighborhood.  If you want a real insider's guide to Stockholm along with a friendly, easygoing host then this is the place.                                                                                                                                                                                                              |99            |[andré, pretty, much, sets, the, bar, for, hosting., , couldn't, have, asked, for, more, with, his, hospitality, and, wealth, of, knowledge, of, where, to, go, in, stockholm., , what's, hot,, what's, not,, and, all, the, insider, tips, for, various, neighborhoods, proved, to, be, a, real, help, in, navigating, through, all, that, this, great, city, has, to, offer., , even, moreso,, , andré, was, very, generous, with, his, time, in, getting, me, situated, and, comfortable, in, his, place, along, with, a, tour, of, his, neighborhood., , if, you, want, a, real, insider's, guide, to, stockholm, along, with, a, friendly,, easygoing, host, then, this, is, the, place.]                                                                                                                                                                                                                                           |(36166,[0,1,2,3,4,5,6,7,8,10,11,13,15,17,20,27,31,42,44,48,58,60,61,62,63,65,68,70,78,79,86,121,140,156,178,196,249,256,289,395,404,416,469,477,480,486,517,617,693,739,877,883,904,987,1025,1488,1727,1993,2844,3024,3118,4398,4708,5285,5520,5748,8190,9050,9498,13988,19967,24493],[3.0,3.0,4.0,4.0,1.0,1.0,4.0,1.0,5.0,3.0,3.0,4.0,1.0,1.0,1.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,2.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                 |\n",
      "|9773|1271843   |33486084|2015-05-28 00:00:00|2039120    |Lynne             |Andre's fiancé Johanna greeted us on arrival at their apartment and she was an absolute delight. She made us feel very welcome and offered much information on the surrounding area together with details on directions for meeting our ship. The apartment was very clean and tidy. The location was close to Gamla Stan and within walking distance of attractions. We were able to do laundry in the shared commercial washing machines and dryers. This was something you cannot always achieve when traveling. We enjoyed our stay at Johanna and Andre's cute apartment and thank you Johanna for your lovely assistance.                                                                                                                                                                   |99            |[andre's, fiancé, johanna, greeted, us, on, arrival, at, their, apartment, and, she, was, an, absolute, delight., she, made, us, feel, very, welcome, and, offered, much, information, on, the, surrounding, area, together, with, details, on, directions, for, meeting, our, ship., the, apartment, was, very, clean, and, tidy., the, location, was, close, to, gamla, stan, and, within, walking, distance, of, attractions., we, were, able, to, do, laundry, in, the, shared, commercial, washing, machines, and, dryers., this, was, something, you, cannot, always, achieve, when, traveling., we, enjoyed, our, stay, at, johanna, and, andre's, cute, apartment, and, thank, you, johanna, for, your, lovely, assistance.]                                                                                                                                                                                                     |(36166,[0,1,3,5,6,7,9,10,11,13,14,17,18,19,22,24,27,30,35,36,43,47,50,51,74,83,97,103,111,115,119,130,137,140,165,193,207,211,232,245,268,295,330,345,393,471,511,624,625,781,852,966,1049,1185,1230,1269,1281,1292,1487,1543,1755,2538,2567,3060,3341,3985,4942,5114,7820,8644,14411,22755],[4.0,7.0,2.0,4.0,1.0,2.0,2.0,2.0,1.0,1.0,3.0,2.0,1.0,2.0,2.0,2.0,1.0,1.0,2.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                          |\n",
      "|9774|1271843   |33756415|2015-05-31 00:00:00|1116857    |Dave              |Andre was away when I stayed but arranged for a friend to give me the key. He sent very clear directions to get to the apartment\n",
      "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |99            |[andre, was, away, when, i, stayed, but, arranged, for, a, friend, to, give, me, the, key., he, sent, very, clear, directions, to, get, to, the, apartment]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |(36166,[0,2,3,5,7,10,12,14,29,45,60,73,83,129,166,450,464,471,503,839,876,878,3584],[2.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |\n",
      "|9775|1271843   |33949154|2015-06-02 00:00:00|2730754    |Richard En Jolande|We zijn hartelijk ontvangen door André en Johanna. We voelden ons zeer welkom in hun smaakvol en schone appartement. Een goed bed en heerljik ontbijt in de knusse keuken. Het appartement bevindt zich in een mooi deel van Stockholm en is goed bereikbaar per Metro. Voor herhaling vatbaar!!!!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |99            |[we, zijn, hartelijk, ontvangen, door, andré, en, johanna., we, voelden, ons, zeer, welkom, in, hun, smaakvol, en, schone, appartement., een, goed, bed, en, heerljik, ontbijt, in, de, knusse, keuken., het, appartement, bevindt, zich, in, een, mooi, deel, van, stockholm, en, is, goed, bereikbaar, per, metro., voor, herhaling, vatbaar!!!!]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |(36166,[4,6,9,31,53,98,136,263,470,474,494,602,640,915,987,1168,1299,1601,2040,2693,3188,3240,3796,6614,7109,7505,8573,8895,12191,12753,13122,13409,16878,22988,23775,27138,28644,29348,29365,34908],[1.0,3.0,2.0,1.0,1.0,4.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "|9776|1271843   |34575016|2015-06-09 00:00:00|3680651    |Katarina          |I stayed in Stockholm for three nights and managed to meet Andre on my last night -he was flexible due to my arrival dates and friendly along with his partner!  I had an awesome stay and the location of the apartment was perfect!  Very close to the metro and cool bars etc. Very hospitable, lovely and chilled out couple...thanks guys :)                                                                                                                                                                                                                                                                                                                                                                                                                                                 |99            |[i, stayed, in, stockholm, for, three, nights, and, managed, to, meet, andre, on, my, last, night, -he, was, flexible, due, to, my, arrival, dates, and, friendly, along, with, his, partner!, , i, had, an, awesome, stay, and, the, location, of, the, apartment, was, perfect!, , very, close, to, the, metro, and, cool, bars, etc., very, hospitable,, lovely, and, chilled, out, couple...thanks, guys, :)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |(36166,[0,1,3,5,6,7,8,10,11,12,13,14,18,23,30,31,36,38,47,51,66,68,74,91,118,129,158,266,340,345,358,373,376,431,435,449,555,557,693,694,729,878,1895,3369,4203,5591,5822,24800,31966,34494],[3.0,5.0,3.0,2.0,1.0,2.0,2.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "|9777|1271843   |35240315|2015-06-16 00:00:00|9964210    |Jasmin            |Andre and Johanna were the perfect hosts! Despite us arriving past 1.00am, it was arranged that Johanna would be waiting for us at the apartment. :) \n",
      "\n",
      "Andre had prepared a document that provided ample information about getting around, and to the apartment. It also has many recommendations of things to do/eat as well! That was really helpful! :)\n",
      "\n",
      "The apartment is clean and very spacious. Its location is quiet and not too densely populated - which makes it nice and relaxing. :) It was easy to get around too with the metro station just a 2 minute walk away from the apartment. \n",
      "\n",
      "I would recommend this listing, as well as Andre and Johanna as hosts. It was lovely having stayed and met with them! Thank you!                                                            |99            |[andre, and, johanna, were, the, perfect, hosts!, despite, us, arriving, past, 1.00am,, it, was, arranged, that, johanna, would, be, waiting, for, us, at, the, apartment., :), , , andre, had, prepared, a, document, that, provided, ample, information, about, getting, around,, and, to, the, apartment., it, also, has, many, recommendations, of, things, to, do/eat, as, well!, that, was, really, helpful!, :), , the, apartment, is, clean, and, very, spacious., its, location, is, quiet, and, not, too, densely, populated, -, which, makes, it, nice, and, relaxing., :), it, was, easy, to, get, around, too, with, the, metro, station, just, a, 2, minute, walk, away, from, the, apartment., , , i, would, recommend, this, listing,, as, well, as, andre, and, johanna, as, hosts., it, was, lovely, having, stayed, and, met, with, them!, thank, you!]                                                               |(36166,[0,1,2,3,4,5,7,8,10,11,12,13,14,16,19,21,22,23,25,26,27,28,30,32,33,43,48,50,52,54,55,56,57,59,64,66,69,70,71,73,74,75,79,89,100,105,110,115,129,153,158,166,180,216,220,232,264,278,385,395,397,411,507,514,616,653,841,860,876,878,1046,1134,1151,1424,1470,1521,1543,2075,2582,2714,3106,3823,4558,6272,11317,20332,34426,35954],[6.0,7.0,2.0,3.0,2.0,4.0,1.0,5.0,1.0,1.0,1.0,2.0,1.0,5.0,1.0,1.0,2.0,1.0,4.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                            |\n",
      "|9778|1271843   |35938455|2015-06-23 00:00:00|981642     |Leif              |Andre and Johanna have a great apartment in a wonderful quiet neighborhood with a nice park, cafes and restaurants, and is very close to the Mariatorget T-bana and a short walk from the central Slussen terminal. Their apartment is really cozy and the bed in their guest room is fantastic. They were amazing hosts to us, giving us great tips about where to go and what to see, including a new art gallery just outside the city, and they were a lot of fun to just sit and chat with. We can't recommend staying here enough!                                                                                                                                                                                                                                                          |99            |[andre, and, johanna, have, a, great, apartment, in, a, wonderful, quiet, neighborhood, with, a, nice, park,, cafes, and, restaurants,, and, is, very, close, to, the, mariatorget, t-bana, and, a, short, walk, from, the, central, slussen, terminal., their, apartment, is, really, cozy, and, the, bed, in, their, guest, room, is, fantastic., they, were, amazing, hosts, to, us,, giving, us, great, tips, about, where, to, go, and, what, to, see,, including, a, new, art, gallery, just, outside, the, city,, and, they, were, a, lot, of, fun, to, just, sit, and, chat, with., we, can't, recommend, staying, here, enough!]                                                                                                                                                                                                                                                                                                |(36166,[0,1,2,3,4,6,7,9,11,13,14,15,21,22,26,28,32,43,46,47,57,64,65,75,90,96,100,116,136,167,169,171,178,186,187,196,202,204,213,249,295,303,443,460,475,543,613,642,654,763,765,802,826,827,878,917,1008,1183,1527,1534,1543,1581,1596,1681,2785,4908,8997,18159],[4.0,8.0,6.0,5.0,3.0,2.0,1.0,1.0,1.0,1.0,2.0,2.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                   |\n",
      "|9779|1271843   |36986749|2015-07-03 00:00:00|3077469    |Arleen            |We really enjoyed our short stay in Andre's and Johanna's apartment. They were warm and welcoming, with good suggestions for things to do and see in Stockholm. The apartment is lovely and comfortable.  It was conveniently close to the metro and within walking distance to Gamla Stan. We especially enjoyed the breakfast foods they provided.  We enjoyed our conversations and hope to see them again.                                                                                                                                                                                                                                                                                                                                                                                    |99            |[we, really, enjoyed, our, short, stay, in, andre's, and, johanna's, apartment., they, were, warm, and, welcoming,, with, good, suggestions, for, things, to, do, and, see, in, stockholm., the, apartment, is, lovely, and, comfortable., , it, was, conveniently, close, to, the, metro, and, within, walking, distance, to, gamla, stan., we, especially, enjoyed, the, breakfast, foods, they, provided., , we, enjoyed, our, conversations, and, hope, to, see, them, again.]                                                                                                                                                                                                                                                                                                                                                                                                                                                       |(36166,[0,1,3,4,5,6,8,9,10,13,14,16,18,24,28,41,43,44,47,66,74,103,105,119,169,171,193,201,207,211,225,244,245,250,267,278,444,473,530,545,619,701,708,1694,1854,2538,3387,6295],[3.0,6.0,4.0,1.0,1.0,2.0,2.0,3.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "|9780|1271843   |37598608|2015-07-08 00:00:00|4549906    |Evan              |André and Johanna were great host. Johanna greeted us when we got there to give us the key and show us around. The apartment was centrally located and perfect for our needs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |99            |[andré, and, johanna, were, great, host., johanna, greeted, us, when, we, got, there, to, give, us, the, key, and, show, us, around., the, apartment, was, centrally, located, and, perfect, for, our, needs.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |(36166,[0,1,3,5,9,10,14,15,22,24,39,43,54,81,83,85,324,450,579,585,769,852,923,987,1163,1543],[2.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |\n",
      "|9781|1271843   |41486488|2015-08-07 00:00:00|25536243   |Ulla              |The host was very friendly on welcoming, answered all our questions, was not in a hurry and and even sent all details in advance via email with maps about nice places in the neighborhood.  \n",
      "\n",
      "The neighborhood was beautiful, full of cafes, restaurants, parks and vintage & design shopping. Easy access to subway too. \n",
      "\n",
      "Apartment had lovely artistic details, and description of apartment was accurate. Thank you so much!                                                                                                                                                                                                                                                                                                                                                                 |99            |[the, host, was, very, friendly, on, welcoming,, answered, all, our, questions,, was, not, in, a, hurry, and, and, even, sent, all, details, in, advance, via, email, with, maps, about, nice, places, in, the, neighborhood., , , , the, neighborhood, was, beautiful,, full, of, cafes,, restaurants,, parks, and, vintage, &, design, shopping., easy, access, to, subway, too., , , apartment, had, lovely, artistic, details,, and, description, of, apartment, was, accurate., thank, you, so, much!]                                                                                                                                                                                                                                                                                                                                                                                                                              |(36166,[0,1,2,3,5,6,7,8,11,13,14,17,21,23,24,36,40,42,56,58,59,74,75,86,91,115,204,205,215,231,247,303,361,445,469,588,599,632,701,707,716,839,910,1035,1131,1185,1198,1346,1535,1788,1991,2120,2222,3737,4382,32583],[3.0,4.0,1.0,1.0,4.0,3.0,1.0,5.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                 |\n",
      "|9782|1271843   |42958488|2015-08-16 00:00:00|40102284   |Teo               |André and Johanna were extremely responsive and accommodating in arranging my booking and arrival. André was actually out of town the weekend I was there, but Johanna was there to greet me. She was incredibly friendly and welcoming, offering refreshments from the kitchen as well as a custom-made local map with her personal recommendations for me. We sat down and chatted, which really helped me to feel relaxed and at home--of course Johanna's calm yet friendly and outgoing manner helped too!\n",
      "\n",
      "The apartment is spacious and very clean. The room is quite beautiful; I would say the photos on here don't really do it justice. Johanna also offered me use of a bicycle during my stay, and even helped me figure out the train schedule for going to the airport when I left!|99            |[andré, and, johanna, were, extremely, responsive, and, accommodating, in, arranging, my, booking, and, arrival., andré, was, actually, out, of, town, the, weekend, i, was, there,, but, johanna, was, there, to, greet, me., she, was, incredibly, friendly, and, welcoming,, offering, refreshments, from, the, kitchen, as, well, as, a, custom-made, local, map, with, her, personal, recommendations, for, me., we, sat, down, and, chatted,, which, really, helped, me, to, feel, relaxed, and, at, home--of, course, johanna's, calm, yet, friendly, and, outgoing, manner, helped, too!, , the, apartment, is, spacious, and, very, clean., the, room, is, quite, beautiful;, i, would, say, the, photos, on, here, don't, really, do, it, justice., johanna, also, offered, me, use, of, a, bicycle, during, my, stay,, and, even, helped, me, figure, out, the, train, schedule, for, going, to, the, airport, when, i, left!]|(36166,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,19,25,26,28,29,33,35,36,37,38,39,43,46,52,55,60,69,83,86,91,118,127,130,159,175,187,193,212,223,233,239,251,281,285,297,308,371,398,408,414,428,467,511,514,532,549,559,701,711,712,731,757,800,815,851,907,987,1038,1145,1203,1536,1543,1586,1728,1786,1904,2373,2596,3502,4361,5387,5725,6295,9044,9929,11782,15618,18682,27115,27536],[7.0,9.0,2.0,3.0,2.0,4.0,1.0,1.0,1.0,1.0,2.0,2.0,3.0,1.0,1.0,1.0,1.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,2.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])|\n",
      "|9783|1271843   |44462335|2015-08-26 00:00:00|28465077   |Carolyn           |A friend and I stayed with Andre and Johanna for several nights and loved our experience. They went above and beyond as accommodating hosts to make sure we had great recommendations for Stockholm, work around our travel schedules, and even met us for a great dinner. They are both extremely personal and hospitable. Their flat is well-decorated, spacious, and clean, and they will make you feel at home. The location is perfect--You really get the best of both worlds with a quiet, hip neighborhood, but are also 2 min walk to the subway and an easy walk to the big sites at Gamla Stan. We are so grateful we chose to stay with Andre and Johanna.                                                                                                                            |99            |[a, friend, and, i, stayed, with, andre, and, johanna, for, several, nights, and, loved, our, experience., they, went, above, and, beyond, as, accommodating, hosts, to, make, sure, we, had, great, recommendations, for, stockholm,, work, around, our, travel, schedules,, and, even, met, us, for, a, great, dinner., they, are, both, extremely, personal, and, hospitable., their, flat, is, well-decorated,, spacious,, and, clean,, and, they, will, make, you, feel, at, home., the, location, is, perfect--you, really, get, the, best, of, both, worlds, with, a, quiet,, hip, neighborhood,, but, are, also, 2, min, walk, to, the, subway, and, an, easy, walk, to, the, big, sites, at, gamla, stan., we, are, so, grateful, we, chose, to, stay, with, andre, and, johanna.]                                                                                                                                              |(36166,[0,1,2,3,4,9,10,11,12,13,15,17,18,19,22,23,24,25,28,29,30,40,49,51,52,56,64,67,73,86,110,113,122,129,130,152,153,169,185,190,194,205,220,221,222,239,245,261,269,286,295,311,358,408,436,443,476,496,503,514,566,570,675,708,744,830,878,913,1082,1145,1409,1543,1871,1890,2648,3025,7109,7784,11653,12869,31526],[4.0,10.0,3.0,4.0,2.0,3.0,3.0,1.0,1.0,3.0,2.0,1.0,1.0,2.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                         |\n",
      "|9784|1271843   |46175902|2015-09-08 00:00:00|10574638   |Alex              |Andre and Johanna were fantastic hosts. Kind, generous, funny and willing to take the time to show me around the city or introduce me to places I would have never found in Stockholm on my own. They live in a wonderful area in Stockholm, and if you're visiting Stockholm for the first time, I highly recommend checking in with Andre and Johanna.                                                                                                                                                                                                                                                                                                                                                                                                                                          |99            |[andre, and, johanna, were, fantastic, hosts., kind,, generous,, funny, and, willing, to, take, the, time, to, show, me, around, the, city, or, introduce, me, to, places, i, would, have, never, found, in, stockholm, on, my, own., they, live, in, a, wonderful, area, in, stockholm,, and, if, you're, visiting, stockholm, for, the, first, time,, i, highly, recommend, checking, in, with, andre, and, johanna.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |(36166,[0,1,2,3,6,10,12,13,31,32,33,36,38,43,60,61,62,63,65,80,96,108,110,111,169,190,209,215,240,298,384,399,484,535,645,653,838,878,902,923,962,988,1543,2455,3209,3424,4701,7109],[3.0,4.0,1.0,3.0,4.0,1.0,2.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "|9785|1271843   |48752427|2015-09-28 00:00:00|5849808    |Seb & Stef        |André and Johanna were perfect hosts. They're house was clean, beautiful and located in a very nice part of Stockholm. We recommend staying with them to anyone.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |99            |[andré, and, johanna, were, perfect, hosts., they're, house, was, clean,, beautiful, and, located, in, a, very, nice, part, of, stockholm., we, recommend, staying, with, them, to, anyone.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |(36166,[1,2,3,5,6,7,9,11,13,21,32,43,44,54,81,90,104,112,122,280,545,653,987,1359,1543,4082],[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "+----+----------+--------+-------------------+-----------+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.feature import CountVectorizer\n",
    "\n",
    "cv = CountVectorizer(inputCol=\"words\", outputCol=\"features\")\n",
    "\n",
    "model = cv.fit(tokenized)\n",
    "\n",
    "result = model.transform(tokenized)\n",
    "result.show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- _c0: integer (nullable = true)\n",
      " |-- listing_id: integer (nullable = true)\n",
      " |-- id: integer (nullable = true)\n",
      " |-- date: timestamp (nullable = true)\n",
      " |-- reviewer_id: integer (nullable = true)\n",
      " |-- reviewer_name: string (nullable = true)\n",
      " |-- comments: string (nullable = true)\n",
      " |-- average_rating: string (nullable = true)\n",
      " |-- words: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### -----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "daa = spark.createDataFrame([([1, 2], {\"key\": \"value\"})], [\"l\", \"d\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+--------------+\n",
      "|     l|             d|\n",
      "+------+--------------+\n",
      "|[1, 2]|[key -> value]|\n",
      "+------+--------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "daa.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- l: array (nullable = true)\n",
      " |    |-- element: long (containsNull = true)\n",
      " |-- d: map (nullable = true)\n",
      " |    |-- key: string\n",
      " |    |-- value: string (valueContainsNull = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "daa.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+\n",
      "|l[0]|\n",
      "+----+\n",
      "|   1|\n",
      "+----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "daa.select(daa.l.getItem(0)).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### -----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import col,when\n",
    "orderitems.withColumn(\"listing_id\",\n",
    "          when(col(\"order_item_subtotal\") != (col(\"order_item_product_price\") * col(\"order_item_quantity\")),\"N\")\n",
    "          .otherwise(\"Y\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "orderitems.withColumn(\"valid\",\n",
    "          when(col(\"order_item_subtotal\") != (col(\"order_item_product_price\") * col(\"order_item_quantity\")),\"N\")\n",
    "          .otherwise(\"Y\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import col,when"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "ename": "AnalysisException",
     "evalue": "\"cannot resolve '(`features` * CAST(`average_rating` AS DOUBLE))' due to data type mismatch: differing types in '(`features` * CAST(`average_rating` AS DOUBLE))' (struct<type:tinyint,size:int,indices:array<int>,values:array<double>> and double).;;\\n'Project [_c0#214, CASE WHEN NOT (reviewer_id#218 = (features#329 * cast(average_rating#232 as double))) THEN N ELSE Y END AS listing_id#444, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, words#313, features#329]\\n+- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, words#313, UDF(words#313) AS features#329]\\n   +- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, UDF(comments#220) AS words#313]\\n      +- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232]\\n         +- Join Inner, (listing_id#215 = cast(id_#231 as int))\\n            :- Relation[_c0#214,listing_id#215,id#216,date#217,reviewer_id#218,reviewer_name#219,comments#220] csv\\n            +- Project [id#1 AS id_#231, review_scores_rating#87 AS average_rating#232]\\n               +- Relation[_c0#0,id#1,listing_url#2,scrape_id#3,last_scraped#4,name#5,summary#6,space#7,description#8,experiences_offered#9,neighborhood_overview#10,notes#11,transit#12,access#13,interaction#14,house_rules#15,thumbnail_url#16,medium_url#17,picture_url#18,xl_picture_url#19,host_id#20,host_url#21,host_name#22,host_since#23,... 83 more fields] csv\\n\"",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     64\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m    327\u001b[0m                     \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 328\u001b[0;31m                     format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m    329\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o248.withColumn.\n: org.apache.spark.sql.AnalysisException: cannot resolve '(`features` * CAST(`average_rating` AS DOUBLE))' due to data type mismatch: differing types in '(`features` * CAST(`average_rating` AS DOUBLE))' (struct<type:tinyint,size:int,indices:array<int>,values:array<double>> and double).;;\n'Project [_c0#214, CASE WHEN NOT (reviewer_id#218 = (features#329 * cast(average_rating#232 as double))) THEN N ELSE Y END AS listing_id#444, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, words#313, features#329]\n+- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, words#313, UDF(words#313) AS features#329]\n   +- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, UDF(comments#220) AS words#313]\n      +- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232]\n         +- Join Inner, (listing_id#215 = cast(id_#231 as int))\n            :- Relation[_c0#214,listing_id#215,id#216,date#217,reviewer_id#218,reviewer_name#219,comments#220] csv\n            +- Project [id#1 AS id_#231, review_scores_rating#87 AS average_rating#232]\n               +- Relation[_c0#0,id#1,listing_url#2,scrape_id#3,last_scraped#4,name#5,summary#6,space#7,description#8,experiences_offered#9,neighborhood_overview#10,notes#11,transit#12,access#13,interaction#14,house_rules#15,thumbnail_url#16,medium_url#17,picture_url#18,xl_picture_url#19,host_id#20,host_url#21,host_name#22,host_since#23,... 83 more fields] csv\n\n\tat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\n\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:115)\n\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:107)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$catalyst$trees$TreeNode$$mapChild$2(TreeNode.scala:304)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$13.apply(TreeNode.scala:354)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.immutable.List.map(List.scala:296)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:354)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:275)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.immutable.List.map(List.scala:296)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:93)\n\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:107)\n\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)\n\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)\n\tat org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)\n\tat org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)\n\tat org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)\n\tat org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:3406)\n\tat org.apache.spark.sql.Dataset.select(Dataset.scala:1334)\n\tat org.apache.spark.sql.Dataset.withColumns(Dataset.scala:2252)\n\tat org.apache.spark.sql.Dataset.withColumn(Dataset.scala:2219)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mAnalysisException\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-45-8e52be6f9e79>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m result.withColumn(\"listing_id\",when(col(\"reviewer_id\") != (col(\"features\") * col(\"average_rating\")),\"N\")\\\n\u001b[0;32m----> 2\u001b[0;31m                   .otherwise(\"Y\")).show()\n\u001b[0m",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mwithColumn\u001b[0;34m(self, colName, col)\u001b[0m\n\u001b[1;32m   1987\u001b[0m         \"\"\"\n\u001b[1;32m   1988\u001b[0m         \u001b[0;32massert\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mColumn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"col should be Column\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1989\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwithColumn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolName\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msql_ctx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1990\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1991\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mignore_unicode_prefix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1255\u001b[0m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1256\u001b[0m         return_value = get_return_value(\n\u001b[0;32m-> 1257\u001b[0;31m             answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m   1258\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1259\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     67\u001b[0m                                              e.java_exception.getStackTrace()))\n\u001b[1;32m     68\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.AnalysisException: '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.catalyst.analysis'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAnalysisException\u001b[0m: \"cannot resolve '(`features` * CAST(`average_rating` AS DOUBLE))' due to data type mismatch: differing types in '(`features` * CAST(`average_rating` AS DOUBLE))' (struct<type:tinyint,size:int,indices:array<int>,values:array<double>> and double).;;\\n'Project [_c0#214, CASE WHEN NOT (reviewer_id#218 = (features#329 * cast(average_rating#232 as double))) THEN N ELSE Y END AS listing_id#444, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, words#313, features#329]\\n+- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, words#313, UDF(words#313) AS features#329]\\n   +- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232, UDF(comments#220) AS words#313]\\n      +- Project [_c0#214, listing_id#215, id#216, date#217, reviewer_id#218, reviewer_name#219, comments#220, average_rating#232]\\n         +- Join Inner, (listing_id#215 = cast(id_#231 as int))\\n            :- Relation[_c0#214,listing_id#215,id#216,date#217,reviewer_id#218,reviewer_name#219,comments#220] csv\\n            +- Project [id#1 AS id_#231, review_scores_rating#87 AS average_rating#232]\\n               +- Relation[_c0#0,id#1,listing_url#2,scrape_id#3,last_scraped#4,name#5,summary#6,space#7,description#8,experiences_offered#9,neighborhood_overview#10,notes#11,transit#12,access#13,interaction#14,house_rules#15,thumbnail_url#16,medium_url#17,picture_url#18,xl_picture_url#19,host_id#20,host_url#21,host_name#22,host_since#23,... 83 more fields] csv\\n\""
     ]
    }
   ],
   "source": [
    "result.withColumn(\"listing_id\",when(col(\"reviewer_id\") != (col(\"features\") * col(\"average_rating\")),\"N\")\\\n",
    "                  .otherwise(\"Y\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import udf\n",
    "from pyspark.sql.types import FloatType"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "firstelement=udf(lambda x, y: x*y,FloatType())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def col_mul(col1,col2):\n",
    "    return col1*col2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul = udf(col_mul)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "withColumn() missing 1 required positional argument: 'col'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-39-3fd77e323e47>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwithColumn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"features\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'average_rating'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: withColumn() missing 1 required positional argument: 'col'"
     ]
    }
   ],
   "source": [
    "res = result.withColumn(mul(f.col(\"features\"),f.col('average_rating')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "ename": "Py4JJavaError",
     "evalue": "An error occurred while calling o541.showString.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 33.0 failed 1 times, most recent failure: Lost task 0.0 in stage 33.0 (TID 627, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 377, in main\n    process()\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 372, in process\n    serializer.dump_stream(func(split_index, iterator), outfile)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 345, in dump_stream\n    self.serializer.dump_stream(self._batched(iterator), stream)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 141, in dump_stream\n    for obj in iterator:\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 334, in _batched\n    for item in iterator:\n  File \"<string>\", line 1, in <lambda>\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 85, in <lambda>\n    return lambda *a: f(*a)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/util.py\", line 99, in wrapper\n    return f(*args, **kwargs)\n  File \"<ipython-input-32-280f59bb0cab>\", line 1, in <lambda>\nTypeError: can't multiply sequence by non-int of type 'SparseVector'\n\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:121)\n\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)\n\tat org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:365)\n\tat org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3383)\n\tat org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2544)\n\tat org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2544)\n\tat org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)\n\tat org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)\n\tat org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)\n\tat org.apache.spark.sql.Dataset.head(Dataset.scala:2544)\n\tat org.apache.spark.sql.Dataset.take(Dataset.scala:2758)\n\tat org.apache.spark.sql.Dataset.getRows(Dataset.scala:254)\n\tat org.apache.spark.sql.Dataset.showString(Dataset.scala:291)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 377, in main\n    process()\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 372, in process\n    serializer.dump_stream(func(split_index, iterator), outfile)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 345, in dump_stream\n    self.serializer.dump_stream(self._batched(iterator), stream)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 141, in dump_stream\n    for obj in iterator:\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 334, in _batched\n    for item in iterator:\n  File \"<string>\", line 1, in <lambda>\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 85, in <lambda>\n    return lambda *a: f(*a)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/util.py\", line 99, in wrapper\n    return f(*args, **kwargs)\n  File \"<ipython-input-32-280f59bb0cab>\", line 1, in <lambda>\nTypeError: can't multiply sequence by non-int of type 'SparseVector'\n\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:121)\n\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\t... 1 more\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-33-f2feb65ad480>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfirstelement\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'features'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'average_rating'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mshow\u001b[0;34m(self, n, truncate, vertical)\u001b[0m\n\u001b[1;32m    376\u001b[0m         \"\"\"\n\u001b[1;32m    377\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtruncate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtruncate\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 378\u001b[0;31m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvertical\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    379\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    380\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtruncate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvertical\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1255\u001b[0m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1256\u001b[0m         return_value = get_return_value(\n\u001b[0;32m-> 1257\u001b[0;31m             answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m   1258\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1259\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     61\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdeco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     64\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m             \u001b[0ms\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjava_exception\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m    326\u001b[0m                 raise Py4JJavaError(\n\u001b[1;32m    327\u001b[0m                     \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 328\u001b[0;31m                     format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m    329\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    330\u001b[0m                 raise Py4JError(\n",
      "\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o541.showString.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 33.0 failed 1 times, most recent failure: Lost task 0.0 in stage 33.0 (TID 627, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 377, in main\n    process()\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 372, in process\n    serializer.dump_stream(func(split_index, iterator), outfile)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 345, in dump_stream\n    self.serializer.dump_stream(self._batched(iterator), stream)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 141, in dump_stream\n    for obj in iterator:\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 334, in _batched\n    for item in iterator:\n  File \"<string>\", line 1, in <lambda>\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 85, in <lambda>\n    return lambda *a: f(*a)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/util.py\", line 99, in wrapper\n    return f(*args, **kwargs)\n  File \"<ipython-input-32-280f59bb0cab>\", line 1, in <lambda>\nTypeError: can't multiply sequence by non-int of type 'SparseVector'\n\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:121)\n\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)\n\tat org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:365)\n\tat org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3383)\n\tat org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2544)\n\tat org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2544)\n\tat org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)\n\tat org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)\n\tat org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)\n\tat org.apache.spark.sql.Dataset.head(Dataset.scala:2544)\n\tat org.apache.spark.sql.Dataset.take(Dataset.scala:2758)\n\tat org.apache.spark.sql.Dataset.getRows(Dataset.scala:254)\n\tat org.apache.spark.sql.Dataset.showString(Dataset.scala:291)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 377, in main\n    process()\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 372, in process\n    serializer.dump_stream(func(split_index, iterator), outfile)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 345, in dump_stream\n    self.serializer.dump_stream(self._batched(iterator), stream)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 141, in dump_stream\n    for obj in iterator:\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 334, in _batched\n    for item in iterator:\n  File \"<string>\", line 1, in <lambda>\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 85, in <lambda>\n    return lambda *a: f(*a)\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/util.py\", line 99, in wrapper\n    return f(*args, **kwargs)\n  File \"<ipython-input-32-280f59bb0cab>\", line 1, in <lambda>\nTypeError: can't multiply sequence by non-int of type 'SparseVector'\n\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)\n\tat org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:288)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:121)\n\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\t... 1 more\n"
     ]
    }
   ],
   "source": [
    "result.select(firstelement('features','average_rating')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "ename": "AnalysisException",
     "evalue": "\"Can't extract value from features#329: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;\"",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     64\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m    327\u001b[0m                     \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 328\u001b[0;31m                     format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m    329\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o248.select.\n: org.apache.spark.sql.AnalysisException: Can't extract value from features#329: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;\n\tat org.apache.spark.sql.catalyst.expressions.ExtractValue$.apply(complexTypeExtractors.scala:73)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve(Analyzer.scala:897)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve$2.apply(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve$2.apply(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$35.apply(Analyzer.scala:958)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$35.apply(Analyzer.scala:958)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.AbstractTraversable.map(Traversable.scala:104)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:958)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:901)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:89)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86)\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:901)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:758)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)\n\tat scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)\n\tat scala.collection.immutable.List.foldLeft(List.scala:84)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:127)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:121)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:106)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)\n\tat org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)\n\tat org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)\n\tat org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)\n\tat org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:3406)\n\tat org.apache.spark.sql.Dataset.select(Dataset.scala:1334)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mAnalysisException\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-27-9cbbc36d0ba6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mselect\u001b[0;34m(self, *cols)\u001b[0m\n\u001b[1;32m   1318\u001b[0m         \u001b[0;34m[\u001b[0m\u001b[0mRow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34mu'Alice'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mage\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m12\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mRow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34mu'Bob'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mage\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m15\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1319\u001b[0m         \"\"\"\n\u001b[0;32m-> 1320\u001b[0;31m         \u001b[0mjdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jcols\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mcols\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1321\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjdf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msql_ctx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1322\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1255\u001b[0m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1256\u001b[0m         return_value = get_return_value(\n\u001b[0;32m-> 1257\u001b[0;31m             answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m   1258\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1259\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     67\u001b[0m                                              e.java_exception.getStackTrace()))\n\u001b[1;32m     68\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.AnalysisException: '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.catalyst.analysis'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAnalysisException\u001b[0m: \"Can't extract value from features#329: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;\""
     ]
    }
   ],
   "source": [
    "result.select(result.features[0]).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = result.select(f.col(\"features\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,2,3,5...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,2,3,5...|\n",
      "|(36166,[0,1,2,6,7...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,3,5,6...|\n",
      "|(36166,[0,2,3,5,7...|\n",
      "|(36166,[4,6,9,31,...|\n",
      "|(36166,[0,1,3,5,6...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,3,4,5...|\n",
      "|(36166,[0,1,3,5,9...|\n",
      "|(36166,[0,1,2,3,5...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,2,3,4...|\n",
      "|(36166,[0,1,2,3,6...|\n",
      "|(36166,[1,2,3,5,6...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "w.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#w = tokenized.select(f.col(\"words\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.mllib.linalg.distributed import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.mllib.linalg.distributed import *\n",
    "\n",
    "def as_block_matrix(rdd, rowsPerBlock=1024, colsPerBlock=1024):\n",
    "    return IndexedRowMatrix(\n",
    "        rdd.zipWithIndex().map(lambda xi: IndexedRow(xi[1], xi[0]))\n",
    "    ).toBlockMatrix(rowsPerBlock, colsPerBlock)\n",
    "\n",
    "#as_block_matrix(rows_1).multiply(as_block_matrix(rows_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-84-c6b4ffdb9a0d>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-84-c6b4ffdb9a0d>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    list(w.collect()[0][0]).*2\u001b[0m\n\u001b[0m                            ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "[w*result.select(f.col(\"\")) list(w.collect()[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "ename": "AnalysisException",
     "evalue": "\"Can't extract value from features#1542: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;\"",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     64\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m    327\u001b[0m                     \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 328\u001b[0;31m                     format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m    329\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o513.withColumn.\n: org.apache.spark.sql.AnalysisException: Can't extract value from features#1542: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;\n\tat org.apache.spark.sql.catalyst.expressions.ExtractValue$.apply(complexTypeExtractors.scala:73)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve(Analyzer.scala:897)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve$2.apply(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve$2.apply(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve$2.apply(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve$2.apply(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$resolve(Analyzer.scala:898)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$35.apply(Analyzer.scala:958)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$35.apply(Analyzer.scala:958)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.immutable.List.map(List.scala:296)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:958)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:901)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:89)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86)\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:901)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:758)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)\n\tat scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)\n\tat scala.collection.immutable.List.foldLeft(List.scala:84)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:127)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:121)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:106)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)\n\tat org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)\n\tat org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)\n\tat org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)\n\tat org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)\n\tat org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:3406)\n\tat org.apache.spark.sql.Dataset.select(Dataset.scala:1334)\n\tat org.apache.spark.sql.Dataset.withColumns(Dataset.scala:2252)\n\tat org.apache.spark.sql.Dataset.withColumn(Dataset.scala:2219)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mAnalysisException\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-73-a5ab8c245062>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwithColumn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"words\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"average_rating\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"features\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mwithColumn\u001b[0;34m(self, colName, col)\u001b[0m\n\u001b[1;32m   1987\u001b[0m         \"\"\"\n\u001b[1;32m   1988\u001b[0m         \u001b[0;32massert\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mColumn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"col should be Column\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1989\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwithColumn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolName\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msql_ctx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1990\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1991\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mignore_unicode_prefix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1255\u001b[0m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1256\u001b[0m         return_value = get_return_value(\n\u001b[0;32m-> 1257\u001b[0;31m             answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m   1258\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1259\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m     67\u001b[0m                                              e.java_exception.getStackTrace()))\n\u001b[1;32m     68\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.AnalysisException: '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'org.apache.spark.sql.catalyst.analysis'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mAnalysisException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m': '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstackTrace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAnalysisException\u001b[0m: \"Can't extract value from features#1542: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;\""
     ]
    }
   ],
   "source": [
    "result.withColumn(\"words\",f.col(\"average_rating\")*f.col(\"features\")[0]).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_title(name):\n",
    "    name_parts = name.split(',')name.split(',')\n",
    "    title_and_first = name_parts[1].split('.')\n",
    "    return title_and_first[0]\n",
    "\n",
    "extract_title_udf = f.udf(extract_title)\n",
    "\n",
    "df.withColumn('Title', extract_title_udf(f.col('Name'))).select('Name', 'Title').show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple way to assess how positive each word is:\n",
    "- for every review: tokenize sentence\n",
    "- create a bag of words --> a document-term matrix\n",
    "- create a dictionary, containing every word from the reviews and create an aggregated score on each word. The aggreated word review could be as simple as:\n",
    "    - adding all the average reviews and dividing with the total number of reviews it has been a part of (to account for non-frequent words) \n",
    "    - creating a much simpler score, where bad reviews= -1, middle= 0, high= 1. These simple metrics can then be added up and normalised again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfl.groupBy('country').agg(f.sum('review_scores_rating')).filter(f.col('country') == 'Sweden').explain()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfl.filter(f.col('country') == 'Sweden').groupBy('country').agg(f.sum('review_scores_rating')).explain()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}