How to deal with SPAs when downloading websites using the library?

[jonathanvinicius]

I'm using the website-scraper library to download websites, and it works great for static websites. However, I am experiencing difficulties when trying to download websites that use Single Page Applications (SPAs). These pages do not download correctly, likely because the content is dynamically loaded via JavaScript after the home page is loaded.

I would like to know what would be the best approach to dealing with SPAs when using the website-scraper. Are there any specific settings or plugins that can be used to ensure that all content is downloaded, including dynamic content generated by JavaScript?

Any suggestions or experience with this type of problem would be greatly appreciated!

[aivus]

Hello @jonathanvinicius

As it's mentioned in README, you can use website-scraper-puppeteer for dynamic content scraping.

[jonathanvinicius]

I will provide you with a code snippet where I have to scan the entire page of a website. Note that only the home page is downloaded, I need the others to also be downloaded, such as the reception page, for example. Could you help me with this? @aivus

try {
  await scrape({
    urls: ["https://noivos.casar.com/jonathan-e-ana-2024-12-24"],
    directory: "./site-baixado",
    plugins: [ 
      new PuppeteerPlugin({
        launchOptions: { headless: "new" }, /* optional */
        gotoOptions: { waitUntil: "networkidle0" }, /* optional */
        scrollToBottom: { timeout: 10000, viewportN: 10 }, /* optional */
        blockNavigation: true, /* optional */
      })
    ],
    subdirectories: [
      { directory: "img", extensions: [".jpg", ".png", ".svg", ".webp"] },
      { directory: "js", extensions: [".js"] },
      { directory: "css", extensions: [".css"] },
      { directory: "fonts", extensions: [".woff2", ".woff", ".ttf", ".eot"] },
      { directory: "maps", extensions: [".json"] },
      { directory: "html", extensions: [".html"] },
      { directory: "php", extensions: [".php"] },
      { directory: "txt", extensions: [".txt"] },
      { directory: "gif", extensions: [".gif"] },
      { directory: "svg", extensions: [".svg"] },
      { directory: "icon", extensions: [".ico"] },
      { directory: "other", extensions: [""] },
    ],
  });
} catch (error) {
  console.error("Ocorreu um erro:", error);
}

[s0ph1e]

Hi @jonathanvinicius

To load other pages, please try to set recursive: true option and maxRecursiveDepth as described in readme. Recursive page downloading is disabled by default.

Hope it helps