Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix doc examples on callback_for() #75

Merged
merged 1 commit into from
Jun 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions docs/intro/basic-tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -200,23 +200,35 @@ returning the result of the ``to_item`` method call. We could use

.. tip::

:func:`~.callback_for` also supports `async generators`. So having the
:func:`~.callback_for` also supports `async generators`. So if we have the
following:

.. code-block:: python

class BookPage(web_poet.ItemWebPage):
async def to_item(self):
return await do_something_async()
class BooksSpider(scrapy.Spider):
name = 'books'
start_urls = ['http://books.toscrape.com/']

callback_for(BookPage)
def parse(self, response):
links = response.css('.image_container a')
yield from response.follow_all(links, self.parse_book)

would result in:
async def parse_book(self, response: DummyResponse, page: BookPage):
yield await page.to_item()

It could be turned into:

.. code-block:: python

async def parse_book(self, response: DummyResponse, page: BookPage):
yield await page.to_item()
class BooksSpider(scrapy.Spider):
name = 'books'
start_urls = ['http://books.toscrape.com/']

def parse(self, response):
links = response.css('.image_container a')
yield from response.follow_all(links, self.parse_book)

parse_book = callback_for(BookPage)

This is useful when the Page Objects uses additional requests, which rely
heavily on ``async/await`` syntax. More info on this in this tutorial
Expand Down
31 changes: 22 additions & 9 deletions scrapy_poet/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,24 +68,37 @@ def parse(self, response):

parse_book = callback_for(BookPage)

This also produces an async generator callable if the Page Objects's
It also supports producing an async generator callable if the Page Objects's
``to_item()`` method is a coroutine which uses the ``async/await`` syntax.
So having the following:

So if we have the following:

.. code-block:: python

class BookPage(web_poet.ItemWebPage):
async def to_item(self):
return await do_something_async()
class BooksSpider(scrapy.Spider):
name = 'books'
start_urls = ['http://books.toscrape.com/']

callback_for(BookPage)
def parse(self, response):
links = response.css('.image_container a')
yield from response.follow_all(links, self.parse_book)

would result in:
async def parse_book(self, response: DummyResponse, page: BookPage):
yield await page.to_item()

It could be turned into:

.. code-block:: python

async def parse_book(self, response: DummyResponse, page: BookPage):
yield await page.to_item()
class BooksSpider(scrapy.Spider):
name = 'books'
start_urls = ['http://books.toscrape.com/']

def parse(self, response):
links = response.css('.image_container a')
yield from response.follow_all(links, self.parse_book)

parse_book = callback_for(BookPage)

The generated callback could be used as a spider instance method or passed
as an inline/anonymous argument. Make sure to define it as a spider
Expand Down