crawling support?

[matthewmueller]

I love this idea! I was wondering if there's crawling support (basically go to the links of all links on the page), or if there's any plans to add it?

[GromNaN]

It seams to be the purpose of the visit method. But I can't figure how to use it.

[renato]

Two years later, if anyone wants to use visit, you use it within an a selector, and you can scrape this visited content too. The only issue is that GraphQL isn't recursive and you can't create a fragment that would result in a loop/recursion (like following a pagination).

A simple example that visits all td.title > a websites and return their <title>.

{
  page(url:"http://news.ycombinator.com") {
    items: query(selector:"tr.athing") {
      rank: text(selector:"td span.rank")
      title: text(selector:"td.title a")
      sitebit: text(selector:"span.comhead a")
      content: query(selector:"td.title > a") {
        visit {
          title
        }
      }
    }
  }
}