Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add syntax highlight to python code in README.rst #286

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 39 additions & 13 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ Configuration

2. Enable the Splash middleware by adding it to ``DOWNLOADER_MIDDLEWARES``
in your ``settings.py`` file and changing HttpCompressionMiddleware
priority::
priority:

.. code:: python

DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
Expand All @@ -63,7 +65,9 @@ Configuration
for details.

3. Enable ``SplashDeduplicateArgsMiddleware`` by adding it to
``SPIDER_MIDDLEWARES`` in your ``settings.py``::
``SPIDER_MIDDLEWARES`` in your ``settings.py``:

.. code:: python

SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
Expand All @@ -75,13 +79,17 @@ Configuration
also allows to save network traffic by not sending these duplicate
arguments to Splash server multiple times.

4. Set a custom ``DUPEFILTER_CLASS``::
4. Set a custom ``DUPEFILTER_CLASS``:

.. code:: python

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

5. If you use Scrapy HTTP cache then a custom cache storage backend
is required. scrapy-splash provides a subclass of
``scrapy.contrib.httpcache.FilesystemCacheStorage``::
``scrapy.contrib.httpcache.FilesystemCacheStorage``:

.. code:: python

HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

Expand Down Expand Up @@ -120,7 +128,9 @@ Requests
--------

The easiest way to render requests with Splash is to
use ``scrapy_splash.SplashRequest``::
use ``scrapy_splash.SplashRequest``:

.. code:: python

yield SplashRequest(url, self.parse_result,
args={
Expand All @@ -137,7 +147,9 @@ use ``scrapy_splash.SplashRequest``::
)

Alternatively, you can use regular scrapy.Request and
``'splash'`` Request `meta` key::
``'splash'`` Request `meta` key:

.. code:: python

yield scrapy.Request(url, self.parse_result, meta={
'splash': {
Expand Down Expand Up @@ -361,7 +373,9 @@ all request; any value like '1' or 'foo' is fine.

For scrapy-splash session handling to work you must use ``/execute`` endpoint
and a Lua script which accepts 'cookies' argument and returns 'cookies'
field in the result::
field in the result:

.. code:: python

function main(splash)
splash:init_cookies(splash.args.cookies)
Expand Down Expand Up @@ -389,7 +403,9 @@ to add cookies to the current Splash cookiejar.
Examples
========

Get HTML contents::
Get HTML contents:

.. code:: python

import scrapy
from scrapy_splash import SplashRequest
Expand All @@ -406,7 +422,9 @@ Get HTML contents::
# contains HTML processed by a browser.
# ...

Get HTML contents and a screenshot::
Get HTML contents and a screenshot:

.. code:: python

import json
import base64
Expand Down Expand Up @@ -439,7 +457,9 @@ Get HTML contents and a screenshot::

# ...

Run a simple `Splash Lua Script`_::
Run a simple `Splash Lua Script`_:

.. code:: python

import json
import base64
Expand All @@ -466,7 +486,9 @@ Run a simple `Splash Lua Script`_::

More complex `Splash Lua Script`_ example - get a screenshot of an HTML
element by its CSS selector (it requires Splash 2.1+).
Note how are arguments passed to the script::
Note how are arguments passed to the script:

.. code:: python

import json
import base64
Expand Down Expand Up @@ -527,7 +549,9 @@ Note how are arguments passed to the script::

Use a Lua script to get an HTML response with cookies, headers, body
and method set to correct values; ``lua_source`` argument value is cached
on Splash server and is not sent with each request (it requires Splash 2.1+)::
on Splash server and is not sent with each request (it requires Splash 2.1+):

.. code:: python

import scrapy
from scrapy_splash import SplashRequest
Expand Down Expand Up @@ -596,7 +620,9 @@ Why not use the Splash HTTP API directly?

The obvious alternative to scrapy-splash would be to send requests directly
to the Splash `HTTP API`_. Take a look at the example below and make
sure to read the observations after it::
sure to read the observations after it:

.. code:: python

import json

Expand Down