Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes for getting newsdiffs working on OSX #39

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 31 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,29 +18,33 @@ Requirements
You need to have installed on your local machine
* Git
* Python 2.6 or later
* Django and other Python libraries
* Django (~1.8) and other Python libraries

On a Debian- or Ubuntu-based system, it may suffice (untested) to run
$ sudo apt-get install git-core python-django python-django-south python-simplejson
```
$ sudo apt-get install git-core python-django python-django-south python-simplejson
```

On Mac OS, the easiest way may be to install pip:
http://www.pip-installer.org/en/latest/installing.html
and then
$ pip install Django

```
$ pip install Django==1.8
```

Initial setup
-------------

$ python website/manage.py syncdb && python website/manage.py migrate
$ mkdir articles

```
$ python website/manage.py syncdb && python website/manage.py migrate && mkdir articles
```

Running NewsDiffs Locally
-------------------------

Do the initial setup above. Then to start the webserver for testing:
$ python website/manage.py runserver
```
$ python website/manage.py runserver
```

and visit http://localhost:8000/

Expand All @@ -51,19 +55,23 @@ Running the scraper
Do the initial setup above. You will also need additional Python
libraries; on a Debian- or Ubuntu-based system, it may suffice
(untested) to run
$ sudo apt-get install python-bs4 python-beautifulsoup
```
$ sudo apt-get install python-bs4 python-beautifulsoup
```

on a Mac, you will want something like

$ pip install beautifulsoup4
$ pip install beautifulsoup
$ pip install html5lib

```
$ pip install beautifulsoup4
$ pip install beautifulsoup
$ pip install html5lib
```
Note that we need two versions of BeautifulSoup, both 3.2 and 4.0;
some websites are parsed correctly in only one version.

Then run
$ python website/manage.py scraper
```
$ python website/manage.py scraper
```

This will populate the articles repository with a list of current news
articles. This is a snapshot at a single time, so the website will
Expand All @@ -77,9 +85,9 @@ overwritten each run) and errors to /tmp/newsdiffs/logging_errs (which
is cumulative).

To run the scraper every hour, run something like:

$ while true; do python website/manage.py scraper; sleep 60m; done

```
$while true; do python website/manage.py scraper; sleep 60m; done
```
or make a cron job.

Adding new sites to the scraper
Expand All @@ -92,13 +100,13 @@ parsers/__init__.py . You need to
subclass of BaseParser (in parsers/baseparser.py). Model it off
the other parsers in that directory. You can test the parser
with by running, e.g.,

```
$ python parsers/test_parser.py bbc.BBCParser

```
which will output a list of URLs to track, and

```
$ python parsers/test_parser.py bbc.BBCParser http://www.bbc.co.uk/news/uk-21649494

```
which will output the text that NewsDiffs would store.

(2) Add the parser to 'parsers' in parsers/__init__.py
Expand Down
2 changes: 1 addition & 1 deletion website/manage.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "website.settings")
sys.path.append(os.path.dirname(os.getcwd()))
sys.path.append(os.getcwd())
from django.core.management import execute_from_command_line

execute_from_command_line(sys.argv)
1 change: 0 additions & 1 deletion website/settings_dev.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,5 @@
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'south',
'frontend',
)
1 change: 0 additions & 1 deletion website/settings_main.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,6 @@
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'south',
'frontend',
)

Expand Down