Skip to content

Latest commit

 

History

History

airbnb

README.md

AirBnB sentiment analysis of the comments from multiple cities

Login to the IBM Watson Studio, first you need to activate watson services on your account to access the download links below. After downloading the Airbnb dataset, upload it to a COS bucket. The dataset consists of information -like review, reviewer info, coordinates- of the reviews from Airbnb.

  1. Amsterdam
  2. Antwerp Belgium
  3. Athens Europe
  4. Austin
  5. Barcelona
  6. Berlin
  7. Boston
  8. Brussels
  9. Chicago
  10. Dublin
  11. London
  12. Los Angeles
  13. Madrid
  14. Palma Mallorca Spain
  15. Melbourne
  16. Montreal
  17. Nashville
  18. New Orleans
  19. New York City
  20. Oakland
  21. Paris
  22. Portland
  23. San Diego
  24. City of San Francisco
  25. Santa Cruz
  26. Seattle
  27. Sydney
  28. Toronto
  29. Trento
  30. Vancouver
  31. Venice Italy
  32. Vienna Austria.
  33. Washington D.C.

How to run the application

First, you need to install required packages.

import io
import base64
import time
import shutil
import csv
import lithops
import regex
import re
import matplotlib.pyplot as plt
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer

Then you should set the BUCKET variable as the name of the bucket which you uploaded the dataset.

 BUCKET = ['<YOUR_BUCKET_NAME>']

There are 2 major functions in this example:

  • analyze_comments: It is used as the map function in map reduce paradigm. The function parses the dataset and classifies the reviews using nltk by their polarity scores and groups them by being positive, negative or neutral.

  • create_map: This method functions as the reduce function in this scenario. It reduces all the intermediate data grouped by sentiments and draws a map displaying the results in different colors accordingly.

Configuration

IBM Cloud

This is the original code of the experiments presented in Serverless Data Analytics in the IBM Cloud. You need to configure lithops with your own IBM account keys. You can also see more options about the configuration here.

config = {'lithops': {'backend': 'ibm_cf', 'storage': 'ibm_cos'},
          'ibm': {'iam_api_key': '<IAM_API_KEY>'},# If your namespace is IAM based (To reach cloud functions API without cf api key)
          'ibm_cf':  {'endpoint': '<CLOUD_FUNCTIONS_ENDPOINT>',
                      'namespace': '<NAME_OF_YOUR_NAMESPACE>',
                      'namespace_id': '<GUID_OF_YOUR_NAMESPACE>'# If your namespace is IAM based
                      #'api_key': 'YOUR_API_KEY' #If your namespace is foundary based
                     },
          'ibm_cos': {'storage_bucket': '<YOUR_COS_BUCKET_NAME>',
                      'region': '<BUCKET_REGION>',
                      'api_key': '<YOUR_API_KEY>'}}

AWS

We also provide a small execution demo in AWS. Input data is publicly available at lithops-applications-data, users should upload it to their custom bucket with import_dataset_aws.sh (incurs into "Requester pays" billing).

To run the AWS example, simply run the map_sentiment_analysis.ipynb notebook following the AWS-specific steps.