Code

Headless crawler#310

Code

Merged

devl00p merged 40 commits intowapiti-scanner:masterwapiti-scanner/wapiti:masterfrom devl00p:headless_crawlerdevl00p/wapiti:headless_crawlerCopy head branch name to clipboardSep 20, 2022

+2,227-1,118

Commits

Commits on Aug 23, 2022

• wip: headless crawler

  devl00p

  committed
  2b8683a

  Copy full SHA for 2b8683a

• missing dependency

  devl00p

  committed
  e3073c2

  Copy full SHA for e3073c2

• fix most tests

  devl00p

  committed
  a603c23

  Copy full SHA for a603c23

• fix more tests

  devl00p

  committed
  b9ac5eb

  Copy full SHA for b9ac5eb

• remove our asyncmock as we removed python3.7 support and 3.8 has builtin support

  devl00p

  committed
  2f8ca4d

  Copy full SHA for 2f8ca4d

• fix annoying warning in the mod_xxe test (... not awaited)

  devl00p

  committed
  1ba7112

  Copy full SHA for 1ba7112

• improving stop of the headless crawler

  devl00p

  committed
  25c48d1

  Copy full SHA for 25c48d1

• manage several headless modes, cli option

  devl00p

  committed
  ddc7fcd

  Copy full SHA for ddc7fcd

• Loading cookies inside intercepting explorer.

  Show description for 49182f0
  devl00p

  committed
  49182f0

  Copy full SHA for 49182f0

• fixing beginner level error in CrawlerConfiguration (:sweating:) + style

  devl00p

  committed
  c8d9564

  Copy full SHA for c8d9564

• ignore intercepted CONNECT requests + increase delay before reading the webpage

  devl00p

  committed
  6130465

  Copy full SHA for 6130465

• add --wait option for headless mode + force Request objects in "start urls"

  devl00p

  committed
  2f82d73

  Copy full SHA for 2f82d73

• fix test and style

  devl00p

  committed
  174d779

  Copy full SHA for 174d779

• fix style (again)

  devl00p

  committed
  8cb0ee2

  Copy full SHA for 8cb0ee2

• extract more urls

  devl00p

  committed
  f583709

  Copy full SHA for f583709

• integrate exclusions for headless (had to do it in both crawler and mitm proxy)

  devl00p

  committed
  703762b

  Copy full SHA for 703762b

• fix on urls with fragments

  devl00p

  committed
  55d16b3

  Copy full SHA for 55d16b3

• need two separate exclusion lists in intercepting_explorer.py

  devl00p

  committed
  8fd2404

  Copy full SHA for 8fd2404

• Use link_depth in headless crawler. Real values won't appear in output but it will prevent endless scans

  devl00p

  committed
  305c73f

  Copy full SHA for 305c73f

• brings some limits into the intercepting explorer

  devl00p

  committed
  2001dce

  Copy full SHA for 2001dce

• lock pyasn1 version

  devl00p

  committed
  520ae44

  Copy full SHA for 520ae44

• prevent downloading files from the headless browser by checking the mime type

  devl00p

  committed
  c587aaf

  Copy full SHA for c587aaf

• fix setup.py

  devl00p

  committed
  202a4bc

  Copy full SHA for 202a4bc

• style

  devl00p

  committed
  4c91696

  Copy full SHA for 4c91696

Commits on Aug 24, 2022

• prevent out of scope redirections

  devl00p

  committed
  38c2d04

  Copy full SHA for 38c2d04

Commits on Aug 25, 2022

• click on some buttons

  devl00p

  committed
  a2e20a4

  Copy full SHA for a2e20a4

Commits on Aug 30, 2022

• Use a headless browser to detect technologies a website is using + refactor wappalyzer.py

  devl00p

  committed
  88b168f

  Copy full SHA for 88b168f

Commits on Aug 31, 2022

• fix some tests

  devl00p

  committed
  37065d7

  Copy full SHA for 37065d7

• Fix one wappalizer related test, figured out the "find implied software" feature was broken and the tests were wrong.

  devl00p

  committed
  9808582

  Copy full SHA for 9808582

Commits on Sep 1, 2022

• fix more wappalyzer tests

  devl00p

  committed
  e29af3a

  Copy full SHA for e29af3a

• figured out reason behind last test failures (i18n)

  devl00p

  committed
  a065c45

  Copy full SHA for a065c45

Commits on Sep 2, 2022

• refactoring

  devl00p

  committed
  07ff825

  Copy full SHA for 07ff825

Commits on Sep 5, 2022

• pin aiohttp version / use headless browser in mod_wapp only if --headless is activated / completely hide arsenic & marionette logs

  devl00p

  committed
  d14bcf2

  Copy full SHA for d14bcf2

• fix tests for mod_wapp

  devl00p

  committed
  d0b5760

  Copy full SHA for d0b5760

Commits on Sep 9, 2022

• check geckodriver presence before activating headless mode

  devl00p

  committed
  9a4b36a

  Copy full SHA for 9a4b36a

Commits on Sep 12, 2022

• headless mode: allows Firefox to connect to 127.0.0.1 using the proxy

  devl00p

  committed
  01d9360

  Copy full SHA for 01d9360

Commits on Sep 13, 2022

• headless mode: remove the buttons that are added by firefox when reading a JSON file

  devl00p

  committed
  a2161d9

  Copy full SHA for a2161d9

Commits on Sep 14, 2022

• catch asyncio.TimeoutError due to arsenic issue

  devl00p

  committed
  4f317eb

  Copy full SHA for 4f317eb

Commits on Sep 15, 2022

• remove content-disposition header when set (intercepting mode)

  devl00p

  committed
  772ef37

  Copy full SHA for 772ef37

Commits on Sep 18, 2022

• put back use of HTTP redirection urls + raise usage error is auth-type=post is used without -s

  devl00p

  committed
  dfb8df7

  Copy full SHA for dfb8df7