bug: Detection of alive hosts

[psyray]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

According to this issue #7 and some investigation on my side, I think we have a problem with the detection of alive hosts.

I explain

This piece of code is used to check if an endpoint is alive

rengine-ng/web/reNgine/common_func.py

Lines 338 to 340 in 52b4baa

	# If is_alive is True, select only endpoints that are alive
	if is_alive:
	endpoints = [e for e in endpoints if e.is_alive]

The main problem with this check is that it is used as the base check to launch scan like

• dir_file_fuzz
  

  rengine-ng/web/reNgine/tasks.py

  Lines 1620 to 1628 in 52b4baa

  	# Grab URLs to fuzz
  	urls = get_http_urls(
  	is_alive=True,
  	ignore_files=False,
  	write_filepath=input_path,
  	get_only_default_urls=True,
  	ctx=ctx
  	)
  	logger.warning(urls)

• fetch_url
  

  rengine-ng/web/reNgine/tasks.py

  Lines 1748 to 1754 in 52b4baa

  	urls = get_http_urls(
  	is_alive=enable_http_crawl,
  	write_filepath=input_path,
  	exclude_subdomains=exclude_subdomains,
  	get_only_default_urls=True,
  	ctx=ctx
  	)

• vulnerability_scan
  • Nuclei
    

    rengine-ng/web/reNgine/tasks.py

    Lines 2305 to 2310 in 52b4baa

    	get_http_urls(
    	is_alive=enable_http_crawl,
    	ignore_files=True,
    	write_filepath=input_path,
    	ctx=ctx
    	)

  • Dalfox
    

    rengine-ng/web/reNgine/tasks.py

    Lines 2419 to 2424 in 52b4baa

    	get_http_urls(
    	is_alive=False,
    	ignore_files=False,
    	write_filepath=input_path,
    	ctx=ctx
    	)

  • CRL Fuzz
    

    rengine-ng/web/reNgine/tasks.py

    Lines 2541 to 2546 in 52b4baa

    	get_http_urls(
    	is_alive=False,
    	ignore_files=True,
    	write_filepath=input_path,
    	ctx=ctx
    	)

So the method get_http_urls is mandatory to launch scan of the above type.

The main problem comes from the is_alive method of the Endpoint class in the startScan model

rengine-ng/web/startScan/models.py

Lines 374 to 375 in 52b4baa

	def is_alive(self):
	return self.http_status and (0 < self.http_status < 500) and self.http_status != 404

As you can see, if, in those conditions :

• an URL returns 404 or http status code above 500
• get_only_default_urls option is set to true
• No default URL has been set (the default URL are set only in the target scan, not subdomain scan)
  

  rengine-ng/web/reNgine/tasks.py

  Lines 146 to 152 in 52b4baa

  	endpoint, _ = save_endpoint(
  	http_url,
  	ctx=ctx,
  	crawl=enable_http_crawl,
  	is_default=True,
  	subdomain=subdomain
  	)

No base url is returned, so no scan is launched.

It's problematic because dir_file_fuzz could be launched even if the base endpoint returned 404, and it's the same thing for fetch_url and vulnerability_scan

So we need to rework this part to always send to some tools the base URL, and also correctly set the default URL for root endpoint

Expected Behavior

From the moment we have a subdomain, that have an IP, and give some HTTP response, we must run :

• dir_file_fuzz
• vulnerability_scan
• fetch_url

Steps To Reproduce

Try to launch a scan on a website that have the base URL responding HTTP status code >= 500 or 404

Environment

- reNgine: 2.0.2
- OS: debian
- Python: 3.10

Anything else?

No response