How to configure a proxy?

[jeroengui]

Hi, no matter what I try I keep getting errors when I enable any sort of proxy in the web capture settings.

• When trying http proxy (http://user:pass@host:port) I get:

  The capture *insert uuid* (2024-07-23T21:46:50.538238) has an error: Capture.capture_page: net::ERR_TIMED_OUT at https://example.com/ Call log: navigating to "https://example.com/", waiting until "domcontentloaded"

• I tried changing the protocol from http to socks5 (socks5://user:pass@host:port) using another proxy and in the logs I found:

  2024-07-23 20:06:17,058 playwrightcapture INFO:[*uuid*] Unable to process https://example.com: net::ERR_SOCKS_CONNECTION_FAILED

While troubleshooting I opened up a socks5 connection from my server in the same poetry env as lookyloo:

import aiohttp
from aiohttp_socks import ProxyType, ProxyConnector, ChainProxyConnector


async def fetch(url):
    connector = ProxyConnector.from_url('socks5://user:pass@host:port')
    
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get(url) as response:

            return await response.text()
        

import asyncio

async def main():
    text = await fetch('http://example.com')
    print(text)

asyncio.run(main())

This works fine and prints out the text of example.com.

Anything else I should try or change in my config? I'm not using global proxy, because it doesn't seem to support configuring the port at the moment. I'm also not using an external Lacus Instance.

[Rafiot]

I never tried HTTP proxies (simply because I have none at hand to check), but socks5 for tor is extensively tested and should work.

Just to wrap up and make sure we're on the same page:

• Your first error (net::ERR_TIMED_OUT) seems to imply that the HTTP proxy is unreachable for some reason
• then you try with socks5 where it seems to straight fail to connect, but then you try from the same machine with a direct call and it works.

I have no way to test with the same settings as you, but I used this string as a proxy socks5://127.0.0.1:9050 (tor client running locally), and the following tests worked:

1. Capturing http://check.torproject.org with default settings and socks5://127.0.0.1:9050 in the proxy field

2. Configuring a global proxy this way in config/generic.json:

  "global_proxy": {                      
    "enable": true,                      
    "server": "socks5://127.0.0.1:9050", 
    "username": "",                      
    "password": ""                       
  }                               

And capturing http://check.torproject.org without passing the proxy in the proxy field.

---

So I'm not completely sure what is going wrong. My suspicion is that it might be the username and password, and it might not be supported by the browser when passed like that (socks5://user:pass@host:port). I thought I could try with a tor client, but it doesn't seem possible.

Can you try to configure the global proxy this way?

  "global_proxy": {                      
    "enable": true,                      
    "server": "socks5://host:port", 
    "username": "user",                      
    "password": "pass"                       
  }

[Rafiot]

Update on that: microsoft/playwright#2425 (comment)

In short, socks5 authentication is not supported by playwright, sorry. But if your proxy can also do http(s), it should work.

[jeroengui]

Good to know that socks5 is not supported. I'm not using the tor scanning functionality, so I didn't test that.

I tested out the global http proxy and it works with authentication and a custom port, this satisfies my usecase at the moment.

Somehow inputting the same proxy in the Lookyloo web interface doesn't work (when global proxy is disabled) and gives this error:

The capture *insert uuid* (2024-07-23T21:46:50.538238) has an error: Capture.capture_page: net::ERR_TIMED_OUT at https://example.com/ Call log: navigating to "https://example.com/", waiting until "domcontentloaded"

[Rafiot]

Yes, I'm trying that as we speak, and it seems that passing the proxy that way http://user:pass@host:port doesn't work, it has to be passed as a dictionary.

I can patch that on my side and just explode the settings. Nice find!

[Rafiot]

Fixed in this commit: Lookyloo/PlaywrightCapture@c4a699f

http://user:pass@host:port will work as expected.

[jeroengui]

Just updated the playwrightcapture dep to the latest version, now the capture also works when using http://user:pass@host:port in the web interface (global proxy disabled). Thanks for your time and effort!